Nov 02 14:55:28.374455 ip-10-0-135-138 systemd[1]: Starting Kubernetes Kubelet...
Nov 02 14:55:28.746668 ip-10-0-135-138 kubenswrapper[2566]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 02 14:55:28.746668 ip-10-0-135-138 kubenswrapper[2566]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Nov 02 14:55:28.746668 ip-10-0-135-138 kubenswrapper[2566]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 02 14:55:28.746668 ip-10-0-135-138 kubenswrapper[2566]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 02 14:55:28.751660 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.751580    2566 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 14:55:28.751660 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.751659    2566 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 14:55:28.751660 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.751664    2566 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 14:55:28.751753 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.751667    2566 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 14:55:28.751753 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.751670    2566 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 14:55:28.751753 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.751673    2566 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 14:55:28.751753 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.751676    2566 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 14:55:28.751753 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.751678    2566 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 14:55:28.751753 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.751681    2566 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 14:55:28.751753 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.751684    2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 14:55:28.751753 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.751686    2566 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 14:55:28.751753 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.751689    2566 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 14:55:28.751753 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.751692    2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 14:55:28.751753 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.751695    2566 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 14:55:28.751753 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.751698    2566 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 14:55:28.751753 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.751701    2566 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 14:55:28.751753 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.751703    2566 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 14:55:28.751753 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.751706    2566 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 14:55:28.751753 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.751709    2566 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 14:55:28.751753 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.751711    2566 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 14:55:28.751753 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.751714    2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 14:55:28.751753 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.751717    2566 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 14:55:28.752240 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.751719    2566 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 14:55:28.752240 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.751722    2566 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 14:55:28.752240 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.751724    2566 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 14:55:28.752240 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.751727    2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 14:55:28.752240 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.751730    2566 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 14:55:28.752240 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.751733    2566 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 14:55:28.752240 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.751736    2566 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 14:55:28.752240 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.751739    2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 14:55:28.752240 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.751742    2566 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 14:55:28.752240 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.751744    2566 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 14:55:28.752240 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.751747    2566 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 14:55:28.752240 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.751751    2566 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 14:55:28.752240 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.751756    2566 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 14:55:28.752240 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.751759    2566 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 14:55:28.752240 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.751763    2566 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 14:55:28.752240 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.751766    2566 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 14:55:28.752240 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.751770    2566 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 14:55:28.752240 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.751773    2566 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 14:55:28.752240 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.751776    2566 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 14:55:28.752682 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.751779    2566 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 14:55:28.752682 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.751782    2566 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 14:55:28.752682 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.751784    2566 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 14:55:28.752682 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.751787    2566 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 14:55:28.752682 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.751790    2566 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 14:55:28.752682 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.751793    2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 14:55:28.752682 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.751795    2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 14:55:28.752682 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.751798    2566 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 14:55:28.752682 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.751801    2566 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 14:55:28.752682 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.751803    2566 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 14:55:28.752682 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.751806    2566 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 14:55:28.752682 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.751809    2566 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 14:55:28.752682 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.751811    2566 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 14:55:28.752682 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.751814    2566 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 14:55:28.752682 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.751818    2566 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 14:55:28.752682 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.751821    2566 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 14:55:28.752682 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.751824    2566 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 14:55:28.752682 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.751827    2566 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 14:55:28.752682 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.751830    2566 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 14:55:28.753138 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.751833    2566 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 14:55:28.753138 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.751836    2566 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 14:55:28.753138 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.751838    2566 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 14:55:28.753138 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.751841    2566 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 14:55:28.753138 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.751844    2566 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 14:55:28.753138 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.751847    2566 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 14:55:28.753138 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.751849    2566 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 14:55:28.753138 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.751852    2566 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 14:55:28.753138 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.751855    2566 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 14:55:28.753138 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.751858    2566 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 14:55:28.753138 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.751860    2566 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 14:55:28.753138 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.751863    2566 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 14:55:28.753138 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.751866    2566 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 14:55:28.753138 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.751868    2566 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 14:55:28.753138 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.751871    2566 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 14:55:28.753138 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.751874    2566 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 14:55:28.753138 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.751883    2566 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 14:55:28.753138 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.751886    2566 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 14:55:28.753138 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.751889    2566 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 14:55:28.753138 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.751891    2566 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 14:55:28.753615 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.751894    2566 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 14:55:28.753615 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.751897    2566 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 14:55:28.753615 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.751899    2566 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 14:55:28.753615 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.751902    2566 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 14:55:28.753615 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.751905    2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 14:55:28.753615 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.751907    2566 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 14:55:28.753615 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.751910    2566 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 14:55:28.753615 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.751913    2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 14:55:28.753615 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.751916    2566 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 14:55:28.753615 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.751920    2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 14:55:28.753615 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.751922    2566 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 14:55:28.753615 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.751925    2566 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 14:55:28.753615 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.751928    2566 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 14:55:28.753615 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.751931    2566 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 14:55:28.753615 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.751934    2566 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 14:55:28.753615 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.751936    2566 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 14:55:28.753615 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.751939    2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 14:55:28.753615 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.751942    2566 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 14:55:28.753615 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.751944    2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 14:55:28.754066 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.751947    2566 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 14:55:28.754066 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.751949    2566 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 14:55:28.754066 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.751955    2566 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 14:55:28.754066 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.751959    2566 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 14:55:28.754066 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.751962    2566 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 14:55:28.754066 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.751965    2566 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 14:55:28.754066 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.751968    2566 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 14:55:28.754066 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.752779    2566 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 14:55:28.754066 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.752787    2566 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 14:55:28.754066 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.752790    2566 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 14:55:28.754066 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.752794    2566 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 14:55:28.754066 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.752797    2566 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 14:55:28.754066 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.752800    2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 14:55:28.754066 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.752803    2566 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 14:55:28.754066 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.752806    2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 14:55:28.754066 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.752809    2566 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 14:55:28.754066 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.752811    2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 14:55:28.754066 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.752814    2566 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 14:55:28.754560 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.752817    2566 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 14:55:28.754560 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.752820    2566 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 14:55:28.754560 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.752823    2566 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 14:55:28.754560 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.752826    2566 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 14:55:28.754560 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.752829    2566 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 14:55:28.754560 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.752832    2566 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 14:55:28.754560 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.752834    2566 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 14:55:28.754560 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.752837    2566 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 14:55:28.754560 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.752840    2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 14:55:28.754560 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.752843    2566 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 14:55:28.754560 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.752845    2566 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 14:55:28.754560 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.752848    2566 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 14:55:28.754560 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.752851    2566 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 14:55:28.754560 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.752853    2566 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 14:55:28.754560 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.752856    2566 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 14:55:28.754560 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.752858    2566 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 14:55:28.754560 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.752861    2566 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 14:55:28.754560 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.752863    2566 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 14:55:28.754560 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.752866    2566 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 14:55:28.755026 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.752869    2566 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 14:55:28.755026 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.752871    2566 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 14:55:28.755026 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.752874    2566 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 14:55:28.755026 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.752877    2566 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 14:55:28.755026 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.752879    2566 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 14:55:28.755026 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.752882    2566 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 14:55:28.755026 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.752885    2566 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 14:55:28.755026 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.752888    2566 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 14:55:28.755026 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.752890    2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 14:55:28.755026 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.752893    2566 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 14:55:28.755026 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.752896    2566 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 14:55:28.755026 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.752898    2566 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 14:55:28.755026 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.752901    2566 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 14:55:28.755026 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.752904    2566 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 14:55:28.755026 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.752907    2566 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 14:55:28.755026 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.752910    2566 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 14:55:28.755026 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.752913    2566 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 14:55:28.755026 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.752916    2566 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 14:55:28.755026 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.752918    2566 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 14:55:28.755026 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.752921    2566 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 14:55:28.755537 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.752924    2566 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 14:55:28.755537 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.752926    2566 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 14:55:28.755537 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.752929    2566 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 14:55:28.755537 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.752931    2566 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 14:55:28.755537 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.752934    2566 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 14:55:28.755537 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.752937    2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 14:55:28.755537 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.752939    2566 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 14:55:28.755537 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.752942    2566 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 14:55:28.755537 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.752944    2566 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 14:55:28.755537 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.752947    2566 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 14:55:28.755537 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.752949    2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 14:55:28.755537 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.752952    2566 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 14:55:28.755537 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.752955    2566 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 14:55:28.755537 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.752958    2566 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 14:55:28.755537 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.752961    2566 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 14:55:28.755537 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.752963    2566 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 14:55:28.755537 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.752966    2566 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 14:55:28.755537 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.752969    2566 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 14:55:28.755537 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.752973    2566 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 14:55:28.755537 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.752978    2566 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 14:55:28.756004 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.752981    2566 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 14:55:28.756004 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.752984    2566 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 14:55:28.756004 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.752986    2566 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 14:55:28.756004 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.752989    2566 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 14:55:28.756004 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.752991    2566 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 14:55:28.756004 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.752995    2566 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 14:55:28.756004 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.752998    2566 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 14:55:28.756004 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.753001    2566 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 14:55:28.756004 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.753003    2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 14:55:28.756004 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.753006    2566 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 14:55:28.756004 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.753009    2566 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 14:55:28.756004 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.753011    2566 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 14:55:28.756004 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.753014    2566 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 14:55:28.756004 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.753017    2566 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 14:55:28.756004 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.753019    2566 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 14:55:28.756004 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.753022    2566 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 14:55:28.756004 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.753026    2566 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 14:55:28.756004 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.753029    2566 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 14:55:28.756004 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.753031    2566 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 14:55:28.756004 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.753033    2566 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 14:55:28.756505 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.753036    2566 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 14:55:28.756505 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.753039    2566 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 14:55:28.756505 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.753041    2566 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 14:55:28.756505 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.753044    2566 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 14:55:28.756505 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.753046    2566 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 14:55:28.756505 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.753049    2566 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 14:55:28.756505 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.753051    2566 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 14:55:28.756505 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.753054    2566 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 14:55:28.756505 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.753057    2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 14:55:28.756505 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.753059    2566 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 14:55:28.756505 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.753062    2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 14:55:28.756505 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.753064    2566 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 14:55:28.756505 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.753067    2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 14:55:28.756505 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.753070    2566 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 14:55:28.756505 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.753072    2566 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 14:55:28.756505 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.753075    2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 14:55:28.756505 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.753613    2566 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 14:55:28.756505 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.753619    2566 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 14:55:28.756505 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.753623    2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 14:55:28.756505 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.753626    2566 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 14:55:28.756978 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.753629    2566 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 14:55:28.756978 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.753632    2566 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 14:55:28.756978 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.753635    2566 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 14:55:28.756978 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.753638    2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 14:55:28.756978 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.753641    2566 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 14:55:28.756978 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.753643    2566 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 14:55:28.756978 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.753646    2566 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 14:55:28.756978 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.753649    2566 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 14:55:28.756978 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.753652    2566 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 14:55:28.756978 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.753654    2566 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 14:55:28.756978 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.753657    2566 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 14:55:28.756978 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.753660    2566 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 14:55:28.756978 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.753662    2566 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 14:55:28.756978 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.753665    2566 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 14:55:28.756978 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.753668    2566 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 14:55:28.756978 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.753671    2566 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 14:55:28.756978 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.753673    2566 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 14:55:28.756978 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.753676    2566 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 14:55:28.756978 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.753678    2566 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 14:55:28.756978 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.753681    2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 14:55:28.757455 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.753683    2566 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 14:55:28.757455 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.753686    2566 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 14:55:28.757455 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.753689    2566 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 14:55:28.757455 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.753692    2566 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 14:55:28.757455 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.753695    2566 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 14:55:28.757455 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.753698    2566 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 14:55:28.757455 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.753701    2566 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 14:55:28.757455 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.753703    2566 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 14:55:28.757455 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.753707    2566 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 14:55:28.757455 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.753710    2566 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 14:55:28.757455 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.753712    2566 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 14:55:28.757455 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.753715    2566 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 14:55:28.757455 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.753718    2566 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 14:55:28.757455 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.753721    2566 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 14:55:28.757455 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.753723    2566 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 14:55:28.757455 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.753726    2566 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 14:55:28.757455 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.753728    2566 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 14:55:28.757455 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.753731    2566 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 14:55:28.757455 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.753734    2566 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 14:55:28.757455 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.753736    2566 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 14:55:28.757913 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.753739    2566 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 14:55:28.757913 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.753742    2566 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 14:55:28.757913 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.753744    2566 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 14:55:28.757913 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.753748    2566 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 14:55:28.757913 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.753750    2566 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 14:55:28.757913 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.753753    2566 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 14:55:28.757913 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.753755    2566 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 14:55:28.757913 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.753758    2566 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 14:55:28.757913 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.753761    2566 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 14:55:28.757913 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.753764    2566 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 14:55:28.757913 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.753766    2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 14:55:28.757913 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.753769    2566 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 14:55:28.757913 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.753771    2566 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 14:55:28.757913 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.753774    2566 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 14:55:28.757913 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.753777    2566 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 14:55:28.757913 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.753780    2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 14:55:28.757913 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.753783    2566 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 14:55:28.757913 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.753790    2566 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 14:55:28.757913 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.753792    2566 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 14:55:28.758372 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.753795    2566 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 14:55:28.758372 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.753797    2566 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 14:55:28.758372 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.753804    2566 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 14:55:28.758372 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.753808    2566 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 14:55:28.758372 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.753811    2566 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 14:55:28.758372 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.753814    2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 14:55:28.758372 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.753816    2566 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 14:55:28.758372 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.753819    2566 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 14:55:28.758372 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.753821    2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 14:55:28.758372 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.753824    2566 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 14:55:28.758372 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.753826    2566 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 14:55:28.758372 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.753829    2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 14:55:28.758372 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.753832    2566 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 14:55:28.758372 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.753834    2566 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 14:55:28.758372 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.753836    2566 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 14:55:28.758372 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.753839    2566 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 14:55:28.758372 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.753842    2566 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 14:55:28.758372 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.753844    2566 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 14:55:28.758372 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.753847    2566 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 14:55:28.758807 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.753849    2566 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 14:55:28.758807 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.753852    2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 14:55:28.758807 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.753855    2566 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 14:55:28.758807 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.753859    2566 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 14:55:28.758807 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.753863    2566 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 14:55:28.758807 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.753866    2566 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 14:55:28.758807 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.753869    2566 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 14:55:28.758807 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.753872    2566 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 14:55:28.758807 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.753875    2566 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 14:55:28.758807 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.753878    2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 14:55:28.758807 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.753880    2566 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 14:55:28.758807 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.753883    2566 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 14:55:28.758807 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.753886    2566 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 14:55:28.758807 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.753889    2566 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 14:55:28.758807 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.753891    2566 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 14:55:28.758807 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.753896    2566 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 14:55:28.758807 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.753899    2566 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 14:55:28.758807 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.753902    2566 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 14:55:28.758807 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.753904    2566 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 14:55:28.759269 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.753907    2566 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 14:55:28.759269 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.753910    2566 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 14:55:28.759269 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.753912    2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 14:55:28.759269 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.753915    2566 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 14:55:28.759269 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.753917    2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 14:55:28.759269 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:28.754869    2566 flags.go:64] FLAG: --address="0.0.0.0"
Nov 02 14:55:28.759269 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:28.754880    2566 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Nov 02 14:55:28.759269 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:28.754887    2566 flags.go:64] FLAG: --anonymous-auth="true"
Nov 02 14:55:28.759269 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:28.754891    2566 flags.go:64] FLAG: --application-metrics-count-limit="100"
Nov 02 14:55:28.759269 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:28.754896    2566 flags.go:64] FLAG: --authentication-token-webhook="false"
Nov 02 14:55:28.759269 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:28.754900    2566 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Nov 02 14:55:28.759269 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:28.754904    2566 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Nov 02 14:55:28.759269 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:28.754909    2566 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Nov 02 14:55:28.759269 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:28.754912    2566 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Nov 02 14:55:28.759269 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:28.754915    2566 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Nov 02 14:55:28.759269 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:28.754918    2566 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Nov 02 14:55:28.759269 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:28.754922    2566 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Nov 02 14:55:28.759269 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:28.754925    2566 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Nov 02 14:55:28.759269 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:28.754928    2566 flags.go:64] FLAG: --cgroup-root=""
Nov 02 14:55:28.759269 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:28.754931    2566 flags.go:64] FLAG: --cgroups-per-qos="true"
Nov 02 14:55:28.759269 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:28.754934    2566 flags.go:64] FLAG: --client-ca-file=""
Nov 02 14:55:28.759269 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:28.754937    2566 flags.go:64] FLAG: --cloud-provider="external"
Nov 02 14:55:28.759269 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:28.754941    2566 flags.go:64] FLAG: --cluster-dns="[]"
Nov 02 14:55:28.759892 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:28.754946    2566 flags.go:64] FLAG: --cluster-domain=""
Nov 02 14:55:28.759892 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:28.754949    2566 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Nov 02 14:55:28.759892 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:28.754952    2566 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Nov 02 14:55:28.759892 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:28.754956    2566 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Nov 02 14:55:28.759892 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:28.754960    2566 flags.go:64] FLAG: --container-log-max-files="5"
Nov 02 14:55:28.759892 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:28.754965    2566 flags.go:64] FLAG: --container-log-max-size="10Mi"
Nov 02 14:55:28.759892 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:28.754969    2566 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Nov 02 14:55:28.759892 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:28.754972    2566 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Nov 02 14:55:28.759892 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:28.754976    2566 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Nov 02 14:55:28.759892 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:28.754979    2566 flags.go:64] FLAG: --contention-profiling="false"
Nov 02 14:55:28.759892 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:28.754982    2566 flags.go:64] FLAG: --cpu-cfs-quota="true"
Nov 02 14:55:28.759892 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:28.754985    2566 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Nov 02 14:55:28.759892 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:28.754988    2566 flags.go:64] FLAG: --cpu-manager-policy="none"
Nov 02 14:55:28.759892 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:28.754992    2566 flags.go:64] FLAG: --cpu-manager-policy-options=""
Nov 02 14:55:28.759892 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:28.754996    2566 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Nov 02 14:55:28.759892 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:28.754999    2566 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Nov 02 14:55:28.759892 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:28.755002    2566 flags.go:64] FLAG: --enable-debugging-handlers="true"
Nov 02 14:55:28.759892 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:28.755005    2566 flags.go:64] FLAG: --enable-load-reader="false"
Nov 02 14:55:28.759892 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:28.755008    2566 flags.go:64] FLAG: --enable-server="true"
Nov 02 14:55:28.759892 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:28.755011    2566 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Nov 02 14:55:28.759892 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:28.755016    2566 flags.go:64] FLAG: --event-burst="100"
Nov 02 14:55:28.759892 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:28.755019    2566 flags.go:64] FLAG: --event-qps="50"
Nov 02 14:55:28.759892 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:28.755023    2566 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Nov 02 14:55:28.759892 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:28.755026    2566 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Nov 02 14:55:28.760555 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:28.755029    2566 flags.go:64] FLAG: --eviction-hard=""
Nov 02 14:55:28.760555 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:28.755034    2566 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Nov 02 14:55:28.760555 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:28.755037    2566 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Nov 02 14:55:28.760555 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:28.755040    2566 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Nov 02 14:55:28.760555 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:28.755043    2566 flags.go:64] FLAG: --eviction-soft=""
Nov 02 14:55:28.760555 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:28.755046    2566 flags.go:64] FLAG: --eviction-soft-grace-period=""
Nov 02 14:55:28.760555 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:28.755049    2566 flags.go:64] FLAG: --exit-on-lock-contention="false"
Nov 02 14:55:28.760555 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:28.755052    2566 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Nov 02 14:55:28.760555 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:28.755055    2566 flags.go:64] FLAG: --experimental-mounter-path=""
Nov 02 14:55:28.760555 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:28.755058    2566 flags.go:64] FLAG: --fail-cgroupv1="true"
Nov 02 14:55:28.760555 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:28.755061    2566 flags.go:64] FLAG: --fail-swap-on="true"
Nov 02 14:55:28.760555 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:28.755064    2566 flags.go:64] FLAG: --feature-gates=""
Nov 02 14:55:28.760555 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:28.755069    2566 flags.go:64] FLAG: --file-check-frequency="20s"
Nov 02 14:55:28.760555 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:28.755073    2566 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Nov 02 14:55:28.760555 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:28.755076    2566 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Nov 02 14:55:28.760555 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:28.755079    2566 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Nov 02 14:55:28.760555 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:28.755083    2566 flags.go:64] FLAG: --healthz-port="10248"
Nov 02 14:55:28.760555 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:28.755086    2566 flags.go:64] FLAG: --help="false"
Nov 02 14:55:28.760555 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:28.755089    2566 flags.go:64] FLAG: --hostname-override="ip-10-0-135-138.ec2.internal"
Nov 02 14:55:28.760555 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:28.755092    2566 flags.go:64] FLAG: --housekeeping-interval="10s"
Nov 02 14:55:28.760555 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:28.755095    2566 flags.go:64] FLAG: --http-check-frequency="20s"
Nov 02 14:55:28.760555 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:28.755098    2566 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Nov 02 14:55:28.760555 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:28.755101    2566 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Nov 02 14:55:28.760555 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:28.755111    2566 flags.go:64] FLAG: --image-gc-high-threshold="85"
Nov 02 14:55:28.761108 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:28.755115    2566 flags.go:64] FLAG: --image-gc-low-threshold="80"
Nov 02 14:55:28.761108 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:28.755118    2566 flags.go:64] FLAG: --image-service-endpoint=""
Nov 02 14:55:28.761108 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:28.755121    2566 flags.go:64] FLAG: --kernel-memcg-notification="false"
Nov 02 14:55:28.761108 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:28.755134    2566 flags.go:64] FLAG: --kube-api-burst="100"
Nov 02 14:55:28.761108 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:28.755139    2566 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Nov 02 14:55:28.761108 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:28.755142    2566 flags.go:64] FLAG: --kube-api-qps="50"
Nov 02 14:55:28.761108 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:28.755145    2566 flags.go:64] FLAG: --kube-reserved=""
Nov 02 14:55:28.761108 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:28.755148    2566 flags.go:64] FLAG: --kube-reserved-cgroup=""
Nov 02 14:55:28.761108 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:28.755151    2566 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Nov 02 14:55:28.761108 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:28.755154    2566 flags.go:64] FLAG: --kubelet-cgroups=""
Nov 02 14:55:28.761108 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:28.755158    2566 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Nov 02 14:55:28.761108 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:28.755161    2566 flags.go:64] FLAG: --lock-file=""
Nov 02 14:55:28.761108 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:28.755164    2566 flags.go:64] FLAG: --log-cadvisor-usage="false"
Nov 02 14:55:28.761108 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:28.755167    2566 flags.go:64] FLAG: --log-flush-frequency="5s"
Nov 02 14:55:28.761108 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:28.755170    2566 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Nov 02 14:55:28.761108 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:28.755176    2566 flags.go:64] FLAG: --log-json-split-stream="false"
Nov 02 14:55:28.761108 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:28.755179    2566 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Nov 02 14:55:28.761108 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:28.755182    2566 flags.go:64] FLAG: --log-text-split-stream="false"
Nov 02 14:55:28.761108 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:28.755185    2566 flags.go:64] FLAG: --logging-format="text"
Nov 02 14:55:28.761108 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:28.755188    2566 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Nov 02 14:55:28.761108 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:28.755192    2566 flags.go:64] FLAG: --make-iptables-util-chains="true"
Nov 02 14:55:28.761108 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:28.755195    2566 flags.go:64] FLAG: --manifest-url=""
Nov 02 14:55:28.761108 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:28.755199    2566 flags.go:64] FLAG: --manifest-url-header=""
Nov 02 14:55:28.761108 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:28.755203    2566 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Nov 02 14:55:28.761108 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:28.755206    2566 flags.go:64] FLAG: --max-open-files="1000000"
Nov 02 14:55:28.761725 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:28.755210    2566 flags.go:64] FLAG: --max-pods="110"
Nov 02 14:55:28.761725 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:28.755213    2566 flags.go:64] FLAG: --maximum-dead-containers="-1"
Nov 02 14:55:28.761725 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:28.755216    2566 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Nov 02 14:55:28.761725 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:28.755219    2566 flags.go:64] FLAG: --memory-manager-policy="None"
Nov 02 14:55:28.761725 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:28.755222    2566 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Nov 02 14:55:28.761725 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:28.755225    2566 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Nov 02 14:55:28.761725 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:28.755228    2566 flags.go:64] FLAG: --node-ip="0.0.0.0"
Nov 02 14:55:28.761725 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:28.755231    2566 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Nov 02 14:55:28.761725 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:28.755238    2566 flags.go:64] FLAG: --node-status-max-images="50"
Nov 02 14:55:28.761725 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:28.755241    2566 flags.go:64] FLAG: --node-status-update-frequency="10s"
Nov 02 14:55:28.761725 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:28.755244    2566 flags.go:64] FLAG: --oom-score-adj="-999"
Nov 02 14:55:28.761725 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:28.755248    2566 flags.go:64] FLAG: --pod-cidr=""
Nov 02 14:55:28.761725 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:28.755251    2566 flags.go:64] FLAG: --pod-manifest-path=""
Nov 02 14:55:28.761725 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:28.755254    2566 flags.go:64] FLAG: --pod-max-pids="-1"
Nov 02 14:55:28.761725 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:28.755258    2566 flags.go:64] FLAG: --pods-per-core="0"
Nov 02 14:55:28.761725 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:28.755261    2566 flags.go:64] FLAG: --port="10250"
Nov 02 14:55:28.761725 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:28.755264    2566 flags.go:64] FLAG: --protect-kernel-defaults="false"
Nov 02 14:55:28.761725 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:28.755267    2566 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0b10dc7a7e78e104b"
Nov 02 14:55:28.761725 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:28.755271    2566 flags.go:64] FLAG: --qos-reserved=""
Nov 02 14:55:28.761725 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:28.755274    2566 flags.go:64] FLAG: --read-only-port="10255"
Nov 02 14:55:28.761725 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:28.755277    2566 flags.go:64] FLAG: --register-node="true"
Nov 02 14:55:28.761725 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:28.755280    2566 flags.go:64] FLAG: --register-with-taints=""
Nov 02 14:55:28.761725 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:28.755284    2566 flags.go:64] FLAG: --registry-burst="10"
Nov 02 14:55:28.761725 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:28.755287    2566 flags.go:64] FLAG: --registry-qps="5"
Nov 02 14:55:28.761725 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:28.755290    2566 flags.go:64] FLAG: --reserved-cpus=""
Nov 02 14:55:28.761725 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:28.755293    2566 flags.go:64] FLAG: --reserved-memory=""
Nov 02 14:55:28.762358 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:28.755297    2566 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Nov 02 14:55:28.762358 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:28.755300    2566 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Nov 02 14:55:28.762358 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:28.755303    2566 flags.go:64] FLAG: --rotate-certificates="false"
Nov 02 14:55:28.762358 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:28.755307    2566 flags.go:64] FLAG: --rotate-server-certificates="false"
Nov 02 14:55:28.762358 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:28.755310    2566 flags.go:64] FLAG: --runonce="false"
Nov 02 14:55:28.762358 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:28.755313    2566 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Nov 02 14:55:28.762358 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:28.755316    2566 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Nov 02 14:55:28.762358 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:28.755319    2566 flags.go:64] FLAG: --seccomp-default="false"
Nov 02 14:55:28.762358 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:28.755322    2566 flags.go:64] FLAG: --serialize-image-pulls="true"
Nov 02 14:55:28.762358 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:28.755325    2566 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Nov 02 14:55:28.762358 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:28.755328    2566 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Nov 02 14:55:28.762358 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:28.755331    2566 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Nov 02 14:55:28.762358 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:28.755334    2566 flags.go:64] FLAG: --storage-driver-password="root"
Nov 02 14:55:28.762358 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:28.755337    2566 flags.go:64] FLAG: --storage-driver-secure="false"
Nov 02 14:55:28.762358 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:28.755340    2566 flags.go:64] FLAG: --storage-driver-table="stats"
Nov 02 14:55:28.762358 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:28.755343    2566 flags.go:64] FLAG: --storage-driver-user="root"
Nov 02 14:55:28.762358 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:28.755346    2566 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Nov 02 14:55:28.762358 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:28.755350    2566 flags.go:64] FLAG: --sync-frequency="1m0s"
Nov 02 14:55:28.762358 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:28.755353    2566 flags.go:64] FLAG: --system-cgroups=""
Nov 02 14:55:28.762358 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:28.755356    2566 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Nov 02 14:55:28.762358 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:28.755361    2566 flags.go:64] FLAG: --system-reserved-cgroup=""
Nov 02 14:55:28.762358 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:28.755364    2566 flags.go:64] FLAG: --tls-cert-file=""
Nov 02 14:55:28.762358 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:28.755367    2566 flags.go:64] FLAG: --tls-cipher-suites="[]"
Nov 02 14:55:28.762358 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:28.755372    2566 flags.go:64] FLAG: --tls-min-version=""
Nov 02 14:55:28.762358 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:28.755375    2566 flags.go:64] FLAG: --tls-private-key-file=""
Nov 02 14:55:28.762949 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:28.755378    2566 flags.go:64] FLAG: --topology-manager-policy="none"
Nov 02 14:55:28.762949 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:28.755381    2566 flags.go:64] FLAG: --topology-manager-policy-options=""
Nov 02 14:55:28.762949 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:28.755384    2566 flags.go:64] FLAG: --topology-manager-scope="container"
Nov 02 14:55:28.762949 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:28.755387    2566 flags.go:64] FLAG: --v="2"
Nov 02 14:55:28.762949 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:28.755391    2566 flags.go:64] FLAG: --version="false"
Nov 02 14:55:28.762949 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:28.755396    2566 flags.go:64] FLAG: --vmodule=""
Nov 02 14:55:28.762949 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:28.755400    2566 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Nov 02 14:55:28.762949 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:28.755403    2566 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Nov 02 14:55:28.762949 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.755540    2566 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 14:55:28.762949 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.755544    2566 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 14:55:28.762949 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.755547    2566 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 14:55:28.762949 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.755551    2566 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 14:55:28.762949 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.755554    2566 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 14:55:28.762949 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.755557    2566 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 14:55:28.762949 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.755559    2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 14:55:28.762949 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.755562    2566 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 14:55:28.762949 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.755565    2566 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 14:55:28.762949 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.755567    2566 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 14:55:28.762949 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.755570    2566 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 14:55:28.762949 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.755573    2566 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 14:55:28.762949 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.755576    2566 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 14:55:28.763512 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.755578    2566 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 14:55:28.763512 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.755581    2566 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 14:55:28.763512 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.755584    2566 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 14:55:28.763512 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.755587    2566 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 14:55:28.763512 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.755591    2566 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 14:55:28.763512 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.755593    2566 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 14:55:28.763512 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.755596    2566 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 14:55:28.763512 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.755599    2566 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 14:55:28.763512 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.755601    2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 14:55:28.763512 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.755604    2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 14:55:28.763512 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.755607    2566 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 14:55:28.763512 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.755610    2566 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 14:55:28.763512 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.755612    2566 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 14:55:28.763512 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.755615    2566 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 14:55:28.763512 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.755618    2566 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 14:55:28.763512 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.755620    2566 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 14:55:28.763512 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.755623    2566 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 14:55:28.763512 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.755626    2566 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 14:55:28.763512 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.755629    2566 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 14:55:28.763512 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.755633    2566 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 14:55:28.763979 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.755635    2566 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 14:55:28.763979 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.755638    2566 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 14:55:28.763979 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.755640    2566 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 14:55:28.763979 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.755644    2566 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 14:55:28.763979 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.755646    2566 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 14:55:28.763979 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.755649    2566 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 14:55:28.763979 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.755652    2566 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 14:55:28.763979 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.755655    2566 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 14:55:28.763979 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.755657    2566 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 14:55:28.763979 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.755660    2566 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 14:55:28.763979 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.755662    2566 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 14:55:28.763979 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.755665    2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 14:55:28.763979 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.755668    2566 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 14:55:28.763979 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.755670    2566 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 14:55:28.763979 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.755673    2566 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 14:55:28.763979 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.755677    2566 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 14:55:28.763979 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.755680    2566 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 14:55:28.763979 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.755683    2566 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 14:55:28.763979 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.755685    2566 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 14:55:28.763979 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.755688    2566 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 14:55:28.764459 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.755690    2566 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 14:55:28.764459 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.755694    2566 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 14:55:28.764459 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.755696    2566 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 14:55:28.764459 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.755699    2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 14:55:28.764459 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.755701    2566 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 14:55:28.764459 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.755704    2566 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 14:55:28.764459 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.755707    2566 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 14:55:28.764459 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.755710    2566 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 14:55:28.764459 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.755713    2566 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 14:55:28.764459 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.755715    2566 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 14:55:28.764459 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.755718    2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 14:55:28.764459 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.755722    2566 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 14:55:28.764459 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.755726    2566 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 14:55:28.764459 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.755730    2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 14:55:28.764459 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.755734    2566 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 14:55:28.764459 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.755737    2566 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 14:55:28.764459 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.755741    2566 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 14:55:28.764459 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.755744    2566 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 14:55:28.764459 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.755747    2566 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 14:55:28.764909 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.755750    2566 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 14:55:28.764909 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.755752    2566 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 14:55:28.764909 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.755755    2566 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 14:55:28.764909 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.755758    2566 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 14:55:28.764909 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.755760    2566 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 14:55:28.764909 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.755763    2566 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 14:55:28.764909 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.755766    2566 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 14:55:28.764909 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.755768    2566 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 14:55:28.764909 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.755772    2566 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 14:55:28.764909 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.755775    2566 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 14:55:28.764909 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.755778    2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 14:55:28.764909 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.755780    2566 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 14:55:28.764909 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.755783    2566 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 14:55:28.764909 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.755786    2566 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 14:55:28.764909 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.755788    2566 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 14:55:28.764909 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.755792    2566 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 14:55:28.764909 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.755796    2566 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 14:55:28.764909 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.755800    2566 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 14:55:28.764909 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.755803    2566 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 14:55:28.765455 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.755806    2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 14:55:28.765455 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.755809    2566 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 14:55:28.765455 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.755812    2566 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 14:55:28.765455 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.755815    2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 14:55:28.765455 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.755818    2566 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 14:55:28.765455 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.755822    2566 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 14:55:28.765455 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.755824    2566 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 14:55:28.765455 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.755827    2566 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 14:55:28.765455 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.755830    2566 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 14:55:28.765455 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.755832    2566 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 14:55:28.765455 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.755835    2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 14:55:28.765455 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.755838    2566 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 14:55:28.765455 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.755841    2566 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 14:55:28.765455 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.755844    2566 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 14:55:28.765455 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.755846    2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 14:55:28.765811 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:28.755873    2566 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 02 14:55:28.765811 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.755912    2566 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 14:55:28.765811 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.755917    2566 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 14:55:28.765811 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.755919    2566 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 14:55:28.765811 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.755922    2566 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 14:55:28.765811 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.755925    2566 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 14:55:28.765811 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.755928    2566 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 14:55:28.765811 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.755932    2566 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 14:55:28.765811 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.755934    2566 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 14:55:28.765811 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.755937    2566 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 14:55:28.765811 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.755940    2566 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 14:55:28.765811 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.755942    2566 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 14:55:28.765811 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.755945    2566 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 14:55:28.765811 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.755948    2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 14:55:28.765811 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.755950    2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 14:55:28.765811 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.755953    2566 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 14:55:28.766186 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.755955    2566 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 14:55:28.766186 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.755958    2566 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 14:55:28.766186 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.755961    2566 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 14:55:28.766186 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.755963    2566 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 14:55:28.766186 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.755966    2566 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 14:55:28.766186 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.755968    2566 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 14:55:28.766186 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.755973    2566 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 14:55:28.766186 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.755975    2566 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 14:55:28.766186 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.755978    2566 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 14:55:28.766186 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.755981    2566 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 14:55:28.766186 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.755983    2566 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 14:55:28.766186 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.755986    2566 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 14:55:28.766186 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.755988    2566 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 14:55:28.766186 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.755992    2566 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 14:55:28.766186 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.755994    2566 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 14:55:28.766186 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.755997    2566 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 14:55:28.766186 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.755999    2566 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 14:55:28.766186 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.756002    2566 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 14:55:28.766186 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.756005    2566 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 14:55:28.766186 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.756008    2566 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 14:55:28.766648 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.756010    2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 14:55:28.766648 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.756013    2566 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 14:55:28.766648 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.756016    2566 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 14:55:28.766648 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.756020    2566 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 14:55:28.766648 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.756023    2566 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 14:55:28.766648 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.756025    2566 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 14:55:28.766648 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.756028    2566 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 14:55:28.766648 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.756031    2566 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 14:55:28.766648 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.756033    2566 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 14:55:28.766648 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.756036    2566 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 14:55:28.766648 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.756039    2566 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 14:55:28.766648 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.756041    2566 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 14:55:28.766648 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.756044    2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 14:55:28.766648 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.756047    2566 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 14:55:28.766648 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.756049    2566 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 14:55:28.766648 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.756052    2566 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 14:55:28.766648 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.756055    2566 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 14:55:28.766648 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.756058    2566 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 14:55:28.766648 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.756061    2566 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 14:55:28.766648 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.756064    2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 14:55:28.767122 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.756067    2566 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 14:55:28.767122 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.756070    2566 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 14:55:28.767122 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.756074    2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 14:55:28.767122 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.756077    2566 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 14:55:28.767122 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.756080    2566 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 14:55:28.767122 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.756082    2566 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 14:55:28.767122 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.756085    2566 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 14:55:28.767122 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.756088    2566 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 14:55:28.767122 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.756091    2566 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 14:55:28.767122 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.756094    2566 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 14:55:28.767122 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.756096    2566 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 14:55:28.767122 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.756099    2566 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 14:55:28.767122 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.756102    2566 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 14:55:28.767122 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.756104    2566 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 14:55:28.767122 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.756108    2566 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 14:55:28.767122 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.756112    2566 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 14:55:28.767122 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.756115    2566 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 14:55:28.767122 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.756117    2566 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 14:55:28.767122 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.756120    2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 14:55:28.767589 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.756135    2566 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 14:55:28.767589 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.756140    2566 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 14:55:28.767589 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.756144    2566 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 14:55:28.767589 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.756148    2566 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 14:55:28.767589 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.756153    2566 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 14:55:28.767589 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.756158    2566 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 14:55:28.767589 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.756161    2566 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 14:55:28.767589 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.756164    2566 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 14:55:28.767589 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.756166    2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 14:55:28.767589 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.756169    2566 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 14:55:28.767589 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.756172    2566 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 14:55:28.767589 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.756176    2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 14:55:28.767589 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.756179    2566 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 14:55:28.767589 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.756182    2566 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 14:55:28.767589 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.756184    2566 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 14:55:28.767589 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.756187    2566 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 14:55:28.767589 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.756189    2566 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 14:55:28.767589 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.756192    2566 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 14:55:28.767589 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.756195    2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 14:55:28.768037 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.756198    2566 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 14:55:28.768037 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.756201    2566 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 14:55:28.768037 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.756205    2566 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 14:55:28.768037 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.756207    2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 14:55:28.768037 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.756210    2566 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 14:55:28.768037 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.756213    2566 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 14:55:28.768037 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.756216    2566 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 14:55:28.768037 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.756220    2566 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 14:55:28.768037 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.756223    2566 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 14:55:28.768037 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.756228    2566 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 14:55:28.768037 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.756231    2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 14:55:28.768037 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.756234    2566 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 14:55:28.768037 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.756236    2566 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 14:55:28.768037 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:28.763187    2566 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Nov 02 14:55:28.768037 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:28.763201    2566 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Nov 02 14:55:28.768037 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:28.763217    2566 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Nov 02 14:55:28.768037 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:28.763222    2566 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Nov 02 14:55:28.768037 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.763298    2566 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 14:55:28.768037 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.763303    2566 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 14:55:28.768534 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.763306    2566 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 14:55:28.768534 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.763309    2566 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 14:55:28.768534 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.763312    2566 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 14:55:28.768534 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.763315    2566 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 14:55:28.768534 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.763317    2566 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 14:55:28.768534 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.763320    2566 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 14:55:28.768534 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.763323    2566 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 14:55:28.768534 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.763327    2566 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 14:55:28.768534 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.763331    2566 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 14:55:28.768534 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.763335    2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 14:55:28.768534 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.763338    2566 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 14:55:28.768534 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.763341    2566 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 14:55:28.768534 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.763344    2566 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 14:55:28.768534 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.763347    2566 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 14:55:28.768534 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.763355    2566 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 14:55:28.768534 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.763359    2566 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 14:55:28.768534 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.763361    2566 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 14:55:28.768534 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.763364    2566 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 14:55:28.768534 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.763367    2566 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 14:55:28.768998 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.763369    2566 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 14:55:28.768998 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.763372    2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 14:55:28.768998 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.763375    2566 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 14:55:28.768998 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.763378    2566 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 14:55:28.768998 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.763382    2566 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 14:55:28.768998 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.763384    2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 14:55:28.768998 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.763387    2566 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 14:55:28.768998 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.763390    2566 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 14:55:28.768998 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.763393    2566 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 14:55:28.768998 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.763396    2566 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 14:55:28.768998 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.763398    2566 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 14:55:28.768998 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.763401    2566 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 14:55:28.768998 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.763404    2566 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 14:55:28.768998 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.763407    2566 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 14:55:28.768998 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.763409    2566 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 14:55:28.768998 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.763412    2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 14:55:28.768998 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.763415    2566 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 14:55:28.768998 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.763417    2566 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 14:55:28.768998 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.763420    2566 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 14:55:28.769499 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.763423    2566 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 14:55:28.769499 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.763426    2566 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 14:55:28.769499 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.763428    2566 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 14:55:28.769499 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.763431    2566 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 14:55:28.769499 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.763434    2566 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 14:55:28.769499 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.763436    2566 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 14:55:28.769499 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.763439    2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 14:55:28.769499 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.763441    2566 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 14:55:28.769499 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.763444    2566 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 14:55:28.769499 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.763453    2566 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 14:55:28.769499 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.763456    2566 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 14:55:28.769499 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.763458    2566 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 14:55:28.769499 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.763461    2566 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 14:55:28.769499 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.763463    2566 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 14:55:28.769499 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.763466    2566 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 14:55:28.769499 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.763469    2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 14:55:28.769499 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.763472    2566 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 14:55:28.769499 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.763475    2566 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 14:55:28.769499 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.763477    2566 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 14:55:28.770074 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.763480    2566 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 14:55:28.770074 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.763483    2566 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 14:55:28.770074 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.763486    2566 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 14:55:28.770074 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.763489    2566 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 14:55:28.770074 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.763491    2566 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 14:55:28.770074 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.763494    2566 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 14:55:28.770074 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.763498    2566 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 14:55:28.770074 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.763502    2566 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 14:55:28.770074 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.763505    2566 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 14:55:28.770074 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.763508    2566 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 14:55:28.770074 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.763511    2566 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 14:55:28.770074 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.763514    2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 14:55:28.770074 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.763516    2566 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 14:55:28.770074 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.763519    2566 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 14:55:28.770074 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.763522    2566 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 14:55:28.770074 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.763525    2566 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 14:55:28.770074 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.763528    2566 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 14:55:28.770074 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.763530    2566 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 14:55:28.770074 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.763533    2566 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 14:55:28.770074 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.763535    2566 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 14:55:28.770809 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.763538    2566 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 14:55:28.770809 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.763541    2566 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 14:55:28.770809 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.763543    2566 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 14:55:28.770809 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.763552    2566 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 14:55:28.770809 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.763555    2566 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 14:55:28.770809 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.763557    2566 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 14:55:28.770809 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.763560    2566 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 14:55:28.770809 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.763563    2566 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 14:55:28.770809 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.763565    2566 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 14:55:28.770809 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.763568    2566 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 14:55:28.770809 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.763571    2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 14:55:28.770809 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.763574    2566 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 14:55:28.770809 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.763577    2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 14:55:28.770809 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.763579    2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 14:55:28.770809 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.763582    2566 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 14:55:28.770809 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.763585    2566 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 14:55:28.770809 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.763587    2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 14:55:28.770809 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.763590    2566 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 14:55:28.770809 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.763593    2566 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 14:55:28.770809 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.763595    2566 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 14:55:28.771593 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.763598    2566 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 14:55:28.771593 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.763600    2566 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 14:55:28.771593 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.763603    2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 14:55:28.771593 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.763606    2566 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 14:55:28.771593 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.763608    2566 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 14:55:28.771593 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.763611    2566 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 14:55:28.771593 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.763613    2566 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 14:55:28.771593 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:28.763642    2566 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 02 14:55:28.771593 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.763834    2566 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 14:55:28.771593 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.763839    2566 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 14:55:28.771593 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.763842    2566 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 14:55:28.771593 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.763845    2566 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 14:55:28.771593 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.763848    2566 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 14:55:28.771593 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.763850    2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 14:55:28.771593 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.763853    2566 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 14:55:28.771593 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.763857    2566 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 14:55:28.772054 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.763867    2566 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 14:55:28.772054 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.763870    2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 14:55:28.772054 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.763873    2566 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 14:55:28.772054 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.763876    2566 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 14:55:28.772054 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.763878    2566 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 14:55:28.772054 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.763881    2566 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 14:55:28.772054 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.763884    2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 14:55:28.772054 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.763887    2566 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 14:55:28.772054 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.763890    2566 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 14:55:28.772054 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.763893    2566 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 14:55:28.772054 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.763896    2566 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 14:55:28.772054 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.763898    2566 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 14:55:28.772054 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.763901    2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 14:55:28.772054 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.763904    2566 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 14:55:28.772054 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.763906    2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 14:55:28.772054 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.763909    2566 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 14:55:28.772054 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.763912    2566 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 14:55:28.772054 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.763915    2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 14:55:28.772054 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.763917    2566 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 14:55:28.772642 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.763920    2566 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 14:55:28.772642 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.763922    2566 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 14:55:28.772642 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.763926    2566 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 14:55:28.772642 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.763929    2566 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 14:55:28.772642 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.763933    2566 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 14:55:28.772642 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.763935    2566 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 14:55:28.772642 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.763938    2566 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 14:55:28.772642 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.763941    2566 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 14:55:28.772642 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.763943    2566 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 14:55:28.772642 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.763946    2566 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 14:55:28.772642 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.763949    2566 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 14:55:28.772642 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.763951    2566 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 14:55:28.772642 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.763954    2566 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 14:55:28.772642 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.763956    2566 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 14:55:28.772642 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.763965    2566 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 14:55:28.772642 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.763967    2566 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 14:55:28.772642 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.763970    2566 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 14:55:28.772642 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.763972    2566 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 14:55:28.772642 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.763975    2566 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 14:55:28.773228 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.763978    2566 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 14:55:28.773228 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.763981    2566 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 14:55:28.773228 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.763984    2566 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 14:55:28.773228 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.763986    2566 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 14:55:28.773228 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.763989    2566 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 14:55:28.773228 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.763992    2566 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 14:55:28.773228 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.764001    2566 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 14:55:28.773228 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.764004    2566 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 14:55:28.773228 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.764006    2566 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 14:55:28.773228 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.764009    2566 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 14:55:28.773228 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.764012    2566 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 14:55:28.773228 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.764014    2566 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 14:55:28.773228 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.764017    2566 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 14:55:28.773228 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.764020    2566 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 14:55:28.773228 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.764022    2566 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 14:55:28.773228 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.764025    2566 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 14:55:28.773228 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.764028    2566 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 14:55:28.773228 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.764030    2566 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 14:55:28.773228 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.764033    2566 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 14:55:28.773228 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.764035    2566 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 14:55:28.773972 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.764038    2566 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 14:55:28.773972 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.764041    2566 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 14:55:28.773972 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.764043    2566 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 14:55:28.773972 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.764046    2566 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 14:55:28.773972 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.764048    2566 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 14:55:28.773972 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.764051    2566 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 14:55:28.773972 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.764053    2566 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 14:55:28.773972 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.764056    2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 14:55:28.773972 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.764064    2566 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 14:55:28.773972 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.764067    2566 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 14:55:28.773972 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.764070    2566 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 14:55:28.773972 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.764073    2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 14:55:28.773972 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.764075    2566 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 14:55:28.773972 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.764078    2566 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 14:55:28.773972 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.764080    2566 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 14:55:28.773972 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.764083    2566 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 14:55:28.773972 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.764085    2566 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 14:55:28.773972 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.764088    2566 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 14:55:28.773972 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.764090    2566 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 14:55:28.773972 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.764093    2566 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 14:55:28.774477 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.764095    2566 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 14:55:28.774477 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.764098    2566 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 14:55:28.774477 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.764101    2566 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 14:55:28.774477 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.764103    2566 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 14:55:28.774477 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.764106    2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 14:55:28.774477 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.764108    2566 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 14:55:28.774477 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.764111    2566 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 14:55:28.774477 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.764113    2566 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 14:55:28.774477 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.764116    2566 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 14:55:28.774477 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.764118    2566 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 14:55:28.774477 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.764121    2566 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 14:55:28.774477 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.764139    2566 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 14:55:28.774477 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.764144    2566 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 14:55:28.774477 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.764147    2566 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 14:55:28.774477 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.764150    2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 14:55:28.774477 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.764153    2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 14:55:28.774477 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.764155    2566 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 14:55:28.774477 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.764158    2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 14:55:28.774477 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.764160    2566 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 14:55:28.774941 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.764163    2566 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 14:55:28.774941 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:28.764189    2566 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 02 14:55:28.774941 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.764249    2566 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 14:55:28.774941 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.764253    2566 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 14:55:28.774941 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.764256    2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 14:55:28.774941 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.764259    2566 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 14:55:28.774941 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.764262    2566 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 14:55:28.774941 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.764265    2566 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 14:55:28.774941 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.764268    2566 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 14:55:28.774941 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.764271    2566 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 14:55:28.774941 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.764274    2566 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 14:55:28.774941 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.764277    2566 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 14:55:28.774941 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.764279    2566 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 14:55:28.774941 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.764282    2566 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 14:55:28.774941 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.764285    2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 14:55:28.774941 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.764287    2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 14:55:28.775339 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.764290    2566 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 14:55:28.775339 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.764292    2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 14:55:28.775339 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.764295    2566 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 14:55:28.775339 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.764298    2566 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 14:55:28.775339 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.764300    2566 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 14:55:28.775339 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.764303    2566 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 14:55:28.775339 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.764305    2566 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 14:55:28.775339 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.764308    2566 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 14:55:28.775339 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.764310    2566 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 14:55:28.775339 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.764313    2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 14:55:28.775339 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.764316    2566 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 14:55:28.775339 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.764320    2566 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 14:55:28.775339 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.764324    2566 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 14:55:28.775339 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.764327    2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 14:55:28.775339 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.764330    2566 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 14:55:28.775339 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.764333    2566 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 14:55:28.775339 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.764335    2566 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 14:55:28.775339 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.764338    2566 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 14:55:28.775339 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.764341    2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 14:55:28.775779 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.764350    2566 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 14:55:28.775779 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.764353    2566 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 14:55:28.775779 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.764355    2566 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 14:55:28.775779 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.764358    2566 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 14:55:28.775779 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.764360    2566 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 14:55:28.775779 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.764363    2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 14:55:28.775779 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.764366    2566 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 14:55:28.775779 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.764368    2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 14:55:28.775779 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.764371    2566 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 14:55:28.775779 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.764374    2566 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 14:55:28.775779 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.764376    2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 14:55:28.775779 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.764379    2566 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 14:55:28.775779 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.764381    2566 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 14:55:28.775779 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.764384    2566 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 14:55:28.775779 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.764388    2566 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 14:55:28.775779 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.764391    2566 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 14:55:28.775779 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.764394    2566 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 14:55:28.775779 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.764396    2566 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 14:55:28.775779 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.764399    2566 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 14:55:28.775779 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.764401    2566 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 14:55:28.776288 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.764404    2566 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 14:55:28.776288 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.764407    2566 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 14:55:28.776288 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.764410    2566 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 14:55:28.776288 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.764412    2566 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 14:55:28.776288 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.764415    2566 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 14:55:28.776288 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.764418    2566 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 14:55:28.776288 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.764420    2566 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 14:55:28.776288 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.764423    2566 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 14:55:28.776288 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.764425    2566 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 14:55:28.776288 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.764428    2566 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 14:55:28.776288 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.764430    2566 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 14:55:28.776288 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.764433    2566 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 14:55:28.776288 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.764435    2566 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 14:55:28.776288 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.764444    2566 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 14:55:28.776288 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.764447    2566 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 14:55:28.776288 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.764452    2566 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 14:55:28.776288 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.764454    2566 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 14:55:28.776288 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.764457    2566 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 14:55:28.776288 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.764459    2566 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 14:55:28.776288 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.764462    2566 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 14:55:28.776818 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.764465    2566 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 14:55:28.776818 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.764468    2566 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 14:55:28.776818 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.764471    2566 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 14:55:28.776818 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.764473    2566 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 14:55:28.776818 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.764476    2566 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 14:55:28.776818 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.764479    2566 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 14:55:28.776818 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.764481    2566 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 14:55:28.776818 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.764484    2566 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 14:55:28.776818 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.764486    2566 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 14:55:28.776818 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.764489    2566 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 14:55:28.776818 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.764491    2566 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 14:55:28.776818 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.764494    2566 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 14:55:28.776818 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.764497    2566 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 14:55:28.776818 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.764499    2566 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 14:55:28.776818 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.764502    2566 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 14:55:28.776818 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.764504    2566 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 14:55:28.776818 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.764507    2566 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 14:55:28.776818 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.764510    2566 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 14:55:28.776818 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.764512    2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 14:55:28.776818 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.764514    2566 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 14:55:28.777313 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.764518    2566 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 14:55:28.777313 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.764521    2566 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 14:55:28.777313 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.764523    2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 14:55:28.777313 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.764526    2566 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 14:55:28.777313 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.764528    2566 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 14:55:28.777313 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.764531    2566 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 14:55:28.777313 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.764539    2566 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 14:55:28.777313 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.764543    2566 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 14:55:28.777313 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.764545    2566 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 14:55:28.777313 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.764548    2566 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 14:55:28.777313 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.764550    2566 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 14:55:28.777313 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.764553    2566 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 14:55:28.777313 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:28.764555    2566 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 14:55:28.777313 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:28.764614    2566 server.go:957] "Client rotation is on, will bootstrap in background"
Nov 02 14:55:28.777313 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:28.766859    2566 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Nov 02 14:55:28.777313 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:28.767798    2566 server.go:1014] "Starting client certificate rotation"
Nov 02 14:55:28.777313 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:28.767896    2566 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 02 14:55:28.777313 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:28.767924    2566 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 02 14:55:28.777737 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:28.772525    2566 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 02 14:55:28.777737 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:28.773724    2566 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 02 14:55:28.785448 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:28.785430    2566 log.go:25] "Validated CRI v1 runtime API"
Nov 02 14:55:28.789066 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:28.789048    2566 log.go:25] "Validated CRI v1 image API"
Nov 02 14:55:28.789894 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:28.789871    2566 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Nov 02 14:55:28.794379 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:28.794359    2566 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 b930873d-65fe-4796-a85b-b436842c6376:/dev/nvme0n1p3 cfc27493-9538-44d8-99e4-7a391d34725d:/dev/nvme0n1p4]
Nov 02 14:55:28.794440 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:28.794379    2566 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}]
Nov 02 14:55:28.800803 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:28.800684    2566 manager.go:217] Machine: {Timestamp:2026-11-02 14:55:28.798655712 +0000 UTC m=+0.317679003 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099729 MemoryCapacity:33164328960 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2d3573186e82525e7d99d71f5cc5d2 SystemUUID:ec2d3573-186e-8252-5e7d-99d71f5cc5d2 BootID:58ecc053-6c06-4b52-acf5-b49dc2859d14 Filesystems:[{Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582164480 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632865792 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582164480 Type:vfs Inodes:1048576 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:d3:c5:0b:7d Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:d3:c5:0b:7d Speed:0 Mtu:9001} {Name:ovs-system MacAddress:0e:3d:93:9a:5a:d4 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164328960 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Nov 02 14:55:28.800803 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:28.800800    2566 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Nov 02 14:55:28.800931 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:28.800919    2566 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Nov 02 14:55:28.803862 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:28.803834    2566 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Nov 02 14:55:28.804045 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:28.803864    2566 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-135-138.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Nov 02 14:55:28.804139 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:28.804050    2566 topology_manager.go:143] "Creating topology manager with none policy"
Nov 02 14:55:28.804139 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:28.804062    2566 container_manager_linux.go:311] "Creating device plugin manager"
Nov 02 14:55:28.804139 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:28.804076    2566 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 02 14:55:28.804883 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:28.804869    2566 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 02 14:55:28.806049 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:28.806037    2566 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Nov 02 14:55:28.809947 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:28.809932    2566 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Nov 02 14:55:28.810097 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:28.810086    2566 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Nov 02 14:55:28.810658 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:28.810647    2566 kubelet.go:493] "Attempting to sync node with API server"
Nov 02 14:55:28.810715 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:28.810665    2566 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Nov 02 14:55:28.810715 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:28.810679    2566 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Nov 02 14:55:28.810715 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:28.810694    2566 kubelet.go:405] "Adding apiserver pod source"
Nov 02 14:55:28.810715 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:28.810706    2566 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Nov 02 14:55:28.811161 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:28.811147    2566 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 02 14:55:28.813405 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:28.813379    2566 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Nov 02 14:55:28.820797 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:28.820777    2566 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 02 14:55:28.821680 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:28.821665    2566 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Nov 02 14:55:28.821745 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:28.821698    2566 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Nov 02 14:55:28.821864 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:28.821853    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Nov 02 14:55:28.821902 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:28.821871    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Nov 02 14:55:28.821902 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:28.821877    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Nov 02 14:55:28.821902 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:28.821882    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Nov 02 14:55:28.821902 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:28.821888    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Nov 02 14:55:28.821902 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:28.821894    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Nov 02 14:55:28.822337 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:28.822327    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Nov 02 14:55:28.822367 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:28.822337    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Nov 02 14:55:28.822367 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:28.822344    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Nov 02 14:55:28.822367 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:28.822350    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Nov 02 14:55:28.822367 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:28.822359    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Nov 02 14:55:28.822367 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:28.822368    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Nov 02 14:55:28.823061 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:28.823049    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Nov 02 14:55:28.823094 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:28.823065    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Nov 02 14:55:28.826851 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:28.826834    2566 server.go:1263] "Started kubelet"
Nov 02 14:55:28.827008 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:28.826908    2566 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Nov 02 14:55:28.827057 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:28.826949    2566 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Nov 02 14:55:28.827057 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:28.827045    2566 server_v1.go:49] "podresources" method="list" useActivePods=true
Nov 02 14:55:28.827702 ip-10-0-135-138 systemd[1]: Started Kubernetes Kubelet.
Nov 02 14:55:28.828958 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:28.828940    2566 server.go:317] "Adding debug handlers to kubelet server"
Nov 02 14:55:28.829437 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:28.829413    2566 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Nov 02 14:55:28.834326 ip-10-0-135-138 kubenswrapper[2566]: E1102 14:55:28.834296    2566 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Nov 02 14:55:28.836781 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:28.836759    2566 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Nov 02 14:55:28.837895 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:28.837877    2566 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Nov 02 14:55:28.838052 ip-10-0-135-138 kubenswrapper[2566]: E1102 14:55:28.837988    2566 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-138.ec2.internal\" not found"
Nov 02 14:55:28.838733 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:28.838717    2566 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Nov 02 14:55:28.838806 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:28.838722    2566 volume_manager.go:309] "The desired_state_of_world populator starts"
Nov 02 14:55:28.838806 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:28.838749    2566 volume_manager.go:311] "Starting Kubelet Volume Manager"
Nov 02 14:55:28.838879 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:28.838839    2566 reconstruct.go:97] "Volume reconstruction finished"
Nov 02 14:55:28.838879 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:28.838846    2566 reconciler.go:29] "Reconciler: start to sync state"
Nov 02 14:55:28.839608 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:28.839592    2566 factory.go:153] Registering CRI-O factory
Nov 02 14:55:28.839608 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:28.839608    2566 factory.go:223] Registration of the crio container factory successfully
Nov 02 14:55:28.839738 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:28.839647    2566 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Nov 02 14:55:28.839738 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:28.839663    2566 factory.go:55] Registering systemd factory
Nov 02 14:55:28.839738 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:28.839668    2566 factory.go:223] Registration of the systemd container factory successfully
Nov 02 14:55:28.839738 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:28.839681    2566 factory.go:103] Registering Raw factory
Nov 02 14:55:28.839738 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:28.839722    2566 manager.go:1199] Started watching for new ooms in manager
Nov 02 14:55:28.841002 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:28.840987    2566 manager.go:319] Starting recovery of all containers
Nov 02 14:55:28.854136 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:28.853386    2566 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Nov 02 14:55:28.854136 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:28.853775    2566 manager.go:324] Recovery completed
Nov 02 14:55:28.858784 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:28.858770    2566 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 14:55:28.859467 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:28.859450    2566 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-qrsg6"
Nov 02 14:55:28.859680 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:28.859663    2566 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-135-138.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Nov 02 14:55:28.859735 ip-10-0-135-138 kubenswrapper[2566]: E1102 14:55:28.859676    2566 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-135-138.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Nov 02 14:55:28.860592 ip-10-0-135-138 kubenswrapper[2566]: E1102 14:55:28.859685    2566 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-138.ec2.internal.18e44230c4bbcf99  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-138.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-135-138.ec2.internal,},FirstTimestamp:2026-11-02 14:55:28.826806169 +0000 UTC m=+0.345829459,LastTimestamp:2026-11-02 14:55:28.826806169 +0000 UTC m=+0.345829459,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-138.ec2.internal,}"
Nov 02 14:55:28.860703 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:28.860688    2566 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-138.ec2.internal" event="NodeHasSufficientMemory"
Nov 02 14:55:28.860744 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:28.860717    2566 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-138.ec2.internal" event="NodeHasNoDiskPressure"
Nov 02 14:55:28.860744 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:28.860728    2566 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-138.ec2.internal" event="NodeHasSufficientPID"
Nov 02 14:55:28.861307 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:28.861288    2566 cpu_manager.go:226] "Starting" policy="none"
Nov 02 14:55:28.861307 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:28.861307    2566 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Nov 02 14:55:28.861426 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:28.861290    2566 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 14:55:28.861426 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:28.861327    2566 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Nov 02 14:55:28.862355 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:28.862339    2566 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-138.ec2.internal" event="NodeHasSufficientMemory"
Nov 02 14:55:28.862414 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:28.862372    2566 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-138.ec2.internal" event="NodeHasNoDiskPressure"
Nov 02 14:55:28.862414 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:28.862386    2566 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-138.ec2.internal" event="NodeHasSufficientPID"
Nov 02 14:55:28.871640 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:28.871623    2566 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-qrsg6"
Nov 02 14:55:28.878461 ip-10-0-135-138 kubenswrapper[2566]: E1102 14:55:28.878357    2566 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-138.ec2.internal.18e44230c6c10aba  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-138.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-135-138.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-135-138.ec2.internal,},FirstTimestamp:2026-11-02 14:55:28.860703418 +0000 UTC m=+0.379726712,LastTimestamp:2026-11-02 14:55:28.860703418 +0000 UTC m=+0.379726712,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-138.ec2.internal,}"
Nov 02 14:55:28.878875 ip-10-0-135-138 kubenswrapper[2566]: E1102 14:55:28.878845    2566 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service"
Nov 02 14:55:28.878955 ip-10-0-135-138 kubenswrapper[2566]: E1102 14:55:28.878857    2566 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-135-138.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node"
Nov 02 14:55:28.878955 ip-10-0-135-138 kubenswrapper[2566]: E1102 14:55:28.878853    2566 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver"
Nov 02 14:55:28.917630 ip-10-0-135-138 kubenswrapper[2566]: E1102 14:55:28.917517    2566 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-138.ec2.internal.18e44230c6c15419  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-138.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-135-138.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-135-138.ec2.internal,},FirstTimestamp:2026-11-02 14:55:28.860722201 +0000 UTC m=+0.379745492,LastTimestamp:2026-11-02 14:55:28.860722201 +0000 UTC m=+0.379745492,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-138.ec2.internal,}"
Nov 02 14:55:29.005241 ip-10-0-135-138 kubenswrapper[2566]: E1102 14:55:28.936431    2566 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-138.ec2.internal.18e44230c6c17beb  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-138.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-135-138.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-135-138.ec2.internal,},FirstTimestamp:2026-11-02 14:55:28.860732395 +0000 UTC m=+0.379755688,LastTimestamp:2026-11-02 14:55:28.860732395 +0000 UTC m=+0.379755688,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-138.ec2.internal,}"
Nov 02 14:55:29.005241 ip-10-0-135-138 kubenswrapper[2566]: E1102 14:55:28.938383    2566 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-138.ec2.internal\" not found"
Nov 02 14:55:29.005241 ip-10-0-135-138 kubenswrapper[2566]: E1102 14:55:28.962009    2566 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-135-138.ec2.internal.18e44230c6c10aba\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-138.ec2.internal.18e44230c6c10aba  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-138.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-135-138.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-135-138.ec2.internal,},FirstTimestamp:2026-11-02 14:55:28.860703418 +0000 UTC m=+0.379726712,LastTimestamp:2026-11-02 14:55:28.862354913 +0000 UTC m=+0.381378207,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-138.ec2.internal,}"
Nov 02 14:55:29.005241 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:28.965074    2566 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Nov 02 14:55:29.005241 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:28.966228    2566 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Nov 02 14:55:29.005241 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:28.966254    2566 status_manager.go:249] "Starting to sync pod status with apiserver"
Nov 02 14:55:29.005241 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:28.966286    2566 kubelet.go:2521] "Starting kubelet main sync loop"
Nov 02 14:55:29.005241 ip-10-0-135-138 kubenswrapper[2566]: E1102 14:55:28.966330    2566 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Nov 02 14:55:29.005241 ip-10-0-135-138 kubenswrapper[2566]: E1102 14:55:28.988099    2566 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-135-138.ec2.internal.18e44230c6c15419\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-138.ec2.internal.18e44230c6c15419  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-138.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-135-138.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-135-138.ec2.internal,},FirstTimestamp:2026-11-02 14:55:28.860722201 +0000 UTC m=+0.379745492,LastTimestamp:2026-11-02 14:55:28.862378557 +0000 UTC m=+0.381401850,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-138.ec2.internal,}"
Nov 02 14:55:29.005241 ip-10-0-135-138 kubenswrapper[2566]: E1102 14:55:29.001786    2566 reflector.go:204] "Failed to watch" err="failed to list *v1.RuntimeClass: runtimeclasses.node.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"runtimeclasses\" in API group \"node.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass"
Nov 02 14:55:29.005241 ip-10-0-135-138 kubenswrapper[2566]: E1102 14:55:29.001738    2566 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-135-138.ec2.internal.18e44230c6c17beb\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-138.ec2.internal.18e44230c6c17beb  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-138.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-135-138.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-135-138.ec2.internal,},FirstTimestamp:2026-11-02 14:55:28.860732395 +0000 UTC m=+0.379755688,LastTimestamp:2026-11-02 14:55:28.862392351 +0000 UTC m=+0.381415645,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-138.ec2.internal,}"
Nov 02 14:55:29.005241 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:29.004783    2566 policy_none.go:50] "Start"
Nov 02 14:55:29.005241 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:29.004801    2566 memory_manager.go:187] "Starting memorymanager" policy="None"
Nov 02 14:55:29.005241 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:29.004813    2566 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Nov 02 14:55:29.007950 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:29.006092    2566 policy_none.go:44] "Start"
Nov 02 14:55:29.038998 ip-10-0-135-138 kubenswrapper[2566]: E1102 14:55:29.038977    2566 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-138.ec2.internal\" not found"
Nov 02 14:55:29.040710 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:29.040652    2566 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Nov 02 14:55:29.044664 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:29.044647    2566 manager.go:341] "Starting Device Plugin manager"
Nov 02 14:55:29.045367 ip-10-0-135-138 kubenswrapper[2566]: E1102 14:55:29.045349    2566 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Nov 02 14:55:29.045367 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:29.045366    2566 server.go:87] "Starting device plugin registration server"
Nov 02 14:55:29.045677 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:29.045665    2566 eviction_manager.go:194] "Eviction manager: starting control loop"
Nov 02 14:55:29.045734 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:29.045676    2566 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Nov 02 14:55:29.045808 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:29.045787    2566 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Nov 02 14:55:29.045914 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:29.045887    2566 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Nov 02 14:55:29.045914 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:29.045896    2566 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Nov 02 14:55:29.046536 ip-10-0-135-138 kubenswrapper[2566]: E1102 14:55:29.046516    2566 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Nov 02 14:55:29.046623 ip-10-0-135-138 kubenswrapper[2566]: E1102 14:55:29.046561    2566 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-135-138.ec2.internal\" not found"
Nov 02 14:55:29.066540 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:29.066521    2566 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-138.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-135-138.ec2.internal"]
Nov 02 14:55:29.066592 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:29.066576    2566 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 14:55:29.067521 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:29.067503    2566 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-138.ec2.internal" event="NodeHasSufficientMemory"
Nov 02 14:55:29.067612 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:29.067542    2566 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-138.ec2.internal" event="NodeHasNoDiskPressure"
Nov 02 14:55:29.067612 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:29.067557    2566 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-138.ec2.internal" event="NodeHasSufficientPID"
Nov 02 14:55:29.068920 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:29.068904    2566 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 14:55:29.069096 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:29.069081    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-138.ec2.internal"
Nov 02 14:55:29.069159 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:29.069118    2566 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 14:55:29.069746 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:29.069730    2566 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-138.ec2.internal" event="NodeHasSufficientMemory"
Nov 02 14:55:29.069832 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:29.069756    2566 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-138.ec2.internal" event="NodeHasNoDiskPressure"
Nov 02 14:55:29.069832 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:29.069765    2566 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-138.ec2.internal" event="NodeHasSufficientPID"
Nov 02 14:55:29.069832 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:29.069807    2566 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-138.ec2.internal" event="NodeHasSufficientMemory"
Nov 02 14:55:29.069832 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:29.069831    2566 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-138.ec2.internal" event="NodeHasNoDiskPressure"
Nov 02 14:55:29.070023 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:29.069844    2566 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-138.ec2.internal" event="NodeHasSufficientPID"
Nov 02 14:55:29.070902 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:29.070884    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-138.ec2.internal"
Nov 02 14:55:29.070981 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:29.070912    2566 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 14:55:29.071571 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:29.071554    2566 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-138.ec2.internal" event="NodeHasSufficientMemory"
Nov 02 14:55:29.071645 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:29.071591    2566 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-138.ec2.internal" event="NodeHasNoDiskPressure"
Nov 02 14:55:29.071645 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:29.071602    2566 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-138.ec2.internal" event="NodeHasSufficientPID"
Nov 02 14:55:29.089083 ip-10-0-135-138 kubenswrapper[2566]: E1102 14:55:29.089067    2566 nodelease.go:50] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-135-138.ec2.internal\" not found" node="ip-10-0-135-138.ec2.internal"
Nov 02 14:55:29.106983 ip-10-0-135-138 kubenswrapper[2566]: E1102 14:55:29.106962    2566 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-138.ec2.internal\" not found" node="ip-10-0-135-138.ec2.internal"
Nov 02 14:55:29.110922 ip-10-0-135-138 kubenswrapper[2566]: E1102 14:55:29.110905    2566 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-138.ec2.internal\" not found" node="ip-10-0-135-138.ec2.internal"
Nov 02 14:55:29.140825 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:29.140801    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a98e99ec11875ba668469a960777e8a7-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-138.ec2.internal\" (UID: \"a98e99ec11875ba668469a960777e8a7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-138.ec2.internal"
Nov 02 14:55:29.140920 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:29.140834    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/9a8741d4cd8a99bb77c79c43a5144068-config\") pod \"kube-apiserver-proxy-ip-10-0-135-138.ec2.internal\" (UID: \"9a8741d4cd8a99bb77c79c43a5144068\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-138.ec2.internal"
Nov 02 14:55:29.140920 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:29.140854    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a98e99ec11875ba668469a960777e8a7-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-138.ec2.internal\" (UID: \"a98e99ec11875ba668469a960777e8a7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-138.ec2.internal"
Nov 02 14:55:29.146356 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:29.146340    2566 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 14:55:29.147255 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:29.147242    2566 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-138.ec2.internal" event="NodeHasSufficientMemory"
Nov 02 14:55:29.147325 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:29.147270    2566 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-138.ec2.internal" event="NodeHasNoDiskPressure"
Nov 02 14:55:29.147325 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:29.147282    2566 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-138.ec2.internal" event="NodeHasSufficientPID"
Nov 02 14:55:29.147325 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:29.147307    2566 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-135-138.ec2.internal"
Nov 02 14:55:29.167574 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:29.167556    2566 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-135-138.ec2.internal"
Nov 02 14:55:29.167642 ip-10-0-135-138 kubenswrapper[2566]: E1102 14:55:29.167580    2566 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-135-138.ec2.internal\": node \"ip-10-0-135-138.ec2.internal\" not found"
Nov 02 14:55:29.210613 ip-10-0-135-138 kubenswrapper[2566]: E1102 14:55:29.210581    2566 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-138.ec2.internal\" not found"
Nov 02 14:55:29.241613 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:29.241587    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a98e99ec11875ba668469a960777e8a7-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-138.ec2.internal\" (UID: \"a98e99ec11875ba668469a960777e8a7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-138.ec2.internal"
Nov 02 14:55:29.241699 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:29.241618    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a98e99ec11875ba668469a960777e8a7-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-138.ec2.internal\" (UID: \"a98e99ec11875ba668469a960777e8a7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-138.ec2.internal"
Nov 02 14:55:29.241699 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:29.241635    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/9a8741d4cd8a99bb77c79c43a5144068-config\") pod \"kube-apiserver-proxy-ip-10-0-135-138.ec2.internal\" (UID: \"9a8741d4cd8a99bb77c79c43a5144068\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-138.ec2.internal"
Nov 02 14:55:29.241699 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:29.241683    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a98e99ec11875ba668469a960777e8a7-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-138.ec2.internal\" (UID: \"a98e99ec11875ba668469a960777e8a7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-138.ec2.internal"
Nov 02 14:55:29.241805 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:29.241699    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a98e99ec11875ba668469a960777e8a7-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-138.ec2.internal\" (UID: \"a98e99ec11875ba668469a960777e8a7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-138.ec2.internal"
Nov 02 14:55:29.241805 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:29.241762    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/9a8741d4cd8a99bb77c79c43a5144068-config\") pod \"kube-apiserver-proxy-ip-10-0-135-138.ec2.internal\" (UID: \"9a8741d4cd8a99bb77c79c43a5144068\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-138.ec2.internal"
Nov 02 14:55:29.310999 ip-10-0-135-138 kubenswrapper[2566]: E1102 14:55:29.310918    2566 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-138.ec2.internal\" not found"
Nov 02 14:55:29.411136 ip-10-0-135-138 kubenswrapper[2566]: E1102 14:55:29.411096    2566 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-138.ec2.internal\" not found"
Nov 02 14:55:29.411269 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:29.411169    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-138.ec2.internal"
Nov 02 14:55:29.414406 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:29.414393    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-138.ec2.internal"
Nov 02 14:55:29.511975 ip-10-0-135-138 kubenswrapper[2566]: E1102 14:55:29.511941    2566 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-138.ec2.internal\" not found"
Nov 02 14:55:29.612494 ip-10-0-135-138 kubenswrapper[2566]: E1102 14:55:29.612430    2566 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-138.ec2.internal\" not found"
Nov 02 14:55:29.713208 ip-10-0-135-138 kubenswrapper[2566]: E1102 14:55:29.713012    2566 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-138.ec2.internal\" not found"
Nov 02 14:55:29.768067 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:29.768039    2566 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Nov 02 14:55:29.814328 ip-10-0-135-138 kubenswrapper[2566]: E1102 14:55:29.814295    2566 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-138.ec2.internal\" not found"
Nov 02 14:55:29.892538 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:29.892464    2566 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 02 14:55:29.905113 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:29.905086    2566 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 02 14:55:29.905291 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:29.905215    2566 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 02 14:55:29.920507 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:29.920480    2566 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 02 14:55:29.938249 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:29.938220    2566 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-138.ec2.internal"
Nov 02 14:55:29.964247 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:29.964219    2566 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 02 14:55:29.965179 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:29.965164    2566 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-138.ec2.internal"
Nov 02 14:55:29.988722 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:29.988695    2566 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 02 14:55:30.005741 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:30.005691    2566 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda98e99ec11875ba668469a960777e8a7.slice/crio-9d80c77407d3e67f6198e00be9d36a6e7f9c402b652e9d03c548331917a7ac4c WatchSource:0}: Error finding container 9d80c77407d3e67f6198e00be9d36a6e7f9c402b652e9d03c548331917a7ac4c: Status 404 returned error can't find the container with id 9d80c77407d3e67f6198e00be9d36a6e7f9c402b652e9d03c548331917a7ac4c
Nov 02 14:55:30.005978 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:30.005952    2566 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a8741d4cd8a99bb77c79c43a5144068.slice/crio-777cd23a55bfad60d07f0c9cc7b69832952228d10ba770ce6b855beb1aa7fb9f WatchSource:0}: Error finding container 777cd23a55bfad60d07f0c9cc7b69832952228d10ba770ce6b855beb1aa7fb9f: Status 404 returned error can't find the container with id 777cd23a55bfad60d07f0c9cc7b69832952228d10ba770ce6b855beb1aa7fb9f
Nov 02 14:55:30.008155 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:30.008086    2566 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-01 14:50:29 +0000 UTC" deadline="2028-06-02 03:33:52.722405054 +0000 UTC"
Nov 02 14:55:30.008155 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:30.008151    2566 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13860h38m22.714257318s"
Nov 02 14:55:30.010757 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:30.010742    2566 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 02 14:55:30.812317 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:30.812277    2566 apiserver.go:52] "Watching apiserver"
Nov 02 14:55:30.837232 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:30.837201    2566 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Nov 02 14:55:30.859840 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:30.859808    2566 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Nov 02 14:55:30.860286 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:30.860254    2566 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-node-tuning-operator/tuned-sq4l4","openshift-dns/node-resolver-jpz27","openshift-image-registry/node-ca-q9nx7","openshift-multus/multus-additional-cni-plugins-fbnnl","openshift-network-operator/iptables-alerter-pl8gn","kube-system/global-pull-secret-syncer-vl6zb","kube-system/konnectivity-agent-d5t6w","openshift-multus/multus-7slzh","openshift-multus/network-metrics-daemon-484ld","openshift-network-diagnostics/network-check-target-wns8r","openshift-ovn-kubernetes/ovnkube-node-f8btv","kube-system/kube-apiserver-proxy-ip-10-0-135-138.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6cj82","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-138.ec2.internal"]
Nov 02 14:55:30.862047 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:30.862024    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-d5t6w"
Nov 02 14:55:30.864602 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:30.864004    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6cj82"
Nov 02 14:55:30.865780 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:30.865405    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-q9nx7"
Nov 02 14:55:30.867069 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:30.867052    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-sq4l4"
Nov 02 14:55:30.868516 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:30.868496    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-fbnnl"
Nov 02 14:55:30.869928 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:30.869906    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vl6zb"
Nov 02 14:55:30.870012 ip-10-0-135-138 kubenswrapper[2566]: E1102 14:55:30.869983    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vl6zb" podUID="332f4a2d-a241-4c71-8ebb-a99618c0d3dc"
Nov 02 14:55:30.872463 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:30.871660    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-pl8gn"
Nov 02 14:55:30.873189 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:30.873168    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-7slzh"
Nov 02 14:55:30.874681 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:30.874659    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-484ld"
Nov 02 14:55:30.874790 ip-10-0-135-138 kubenswrapper[2566]: E1102 14:55:30.874736    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-484ld" podUID="caa8898b-6611-49b8-845d-3b9bdac6691f"
Nov 02 14:55:30.876744 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:30.876643    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wns8r"
Nov 02 14:55:30.876744 ip-10-0-135-138 kubenswrapper[2566]: E1102 14:55:30.876698    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wns8r" podUID="0d6c5872-e278-4463-953d-e1388a001d0d"
Nov 02 14:55:30.878066 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:30.878040    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-f8btv"
Nov 02 14:55:30.879511 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:30.879489    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-jpz27"
Nov 02 14:55:30.879616 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:30.879555    2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Nov 02 14:55:30.881779 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:30.881755    2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Nov 02 14:55:30.881878 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:30.881824    2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Nov 02 14:55:30.882102 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:30.882084    2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-btkxl\""
Nov 02 14:55:30.882565 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:30.882294    2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-zhczs\""
Nov 02 14:55:30.882565 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:30.882476    2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Nov 02 14:55:30.894044 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:30.894019    2566 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Nov 02 14:55:30.894159 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:30.894087    2566 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-wcldz"
Nov 02 14:55:30.901892 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:30.901868    2566 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-wcldz"
Nov 02 14:55:30.939991 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:30.939968    2566 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Nov 02 14:55:30.953665 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:30.953635    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/65ce39fc-05c7-433b-81f4-c1b219f1bd71-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-f8btv\" (UID: \"65ce39fc-05c7-433b-81f4-c1b219f1bd71\") " pod="openshift-ovn-kubernetes/ovnkube-node-f8btv"
Nov 02 14:55:30.953807 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:30.953672    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/93d7d50e-b320-4217-b183-2bfc921f6c9d-cni-binary-copy\") pod \"multus-7slzh\" (UID: \"93d7d50e-b320-4217-b183-2bfc921f6c9d\") " pod="openshift-multus/multus-7slzh"
Nov 02 14:55:30.953807 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:30.953697    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/93d7d50e-b320-4217-b183-2bfc921f6c9d-etc-kubernetes\") pod \"multus-7slzh\" (UID: \"93d7d50e-b320-4217-b183-2bfc921f6c9d\") " pod="openshift-multus/multus-7slzh"
Nov 02 14:55:30.953807 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:30.953723    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-prmck\" (UniqueName: \"kubernetes.io/projected/e1da99c3-8294-4e34-9d33-47e04a6182d3-kube-api-access-prmck\") pod \"tuned-sq4l4\" (UID: \"e1da99c3-8294-4e34-9d33-47e04a6182d3\") " pod="openshift-cluster-node-tuning-operator/tuned-sq4l4"
Nov 02 14:55:30.953807 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:30.953751    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-548bv\" (UniqueName: \"kubernetes.io/projected/b1715623-afd0-4d0e-9b63-b38499cec1e2-kube-api-access-548bv\") pod \"multus-additional-cni-plugins-fbnnl\" (UID: \"b1715623-afd0-4d0e-9b63-b38499cec1e2\") " pod="openshift-multus/multus-additional-cni-plugins-fbnnl"
Nov 02 14:55:30.953807 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:30.953769    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/65ce39fc-05c7-433b-81f4-c1b219f1bd71-run-systemd\") pod \"ovnkube-node-f8btv\" (UID: \"65ce39fc-05c7-433b-81f4-c1b219f1bd71\") " pod="openshift-ovn-kubernetes/ovnkube-node-f8btv"
Nov 02 14:55:30.953807 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:30.953789    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/65ce39fc-05c7-433b-81f4-c1b219f1bd71-etc-openvswitch\") pod \"ovnkube-node-f8btv\" (UID: \"65ce39fc-05c7-433b-81f4-c1b219f1bd71\") " pod="openshift-ovn-kubernetes/ovnkube-node-f8btv"
Nov 02 14:55:30.954092 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:30.953844    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b1715623-afd0-4d0e-9b63-b38499cec1e2-cnibin\") pod \"multus-additional-cni-plugins-fbnnl\" (UID: \"b1715623-afd0-4d0e-9b63-b38499cec1e2\") " pod="openshift-multus/multus-additional-cni-plugins-fbnnl"
Nov 02 14:55:30.954092 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:30.953909    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/65ce39fc-05c7-433b-81f4-c1b219f1bd71-host-kubelet\") pod \"ovnkube-node-f8btv\" (UID: \"65ce39fc-05c7-433b-81f4-c1b219f1bd71\") " pod="openshift-ovn-kubernetes/ovnkube-node-f8btv"
Nov 02 14:55:30.954092 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:30.953981    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/93d7d50e-b320-4217-b183-2bfc921f6c9d-host-var-lib-cni-bin\") pod \"multus-7slzh\" (UID: \"93d7d50e-b320-4217-b183-2bfc921f6c9d\") " pod="openshift-multus/multus-7slzh"
Nov 02 14:55:30.954092 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:30.954016    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/93d7d50e-b320-4217-b183-2bfc921f6c9d-host-var-lib-kubelet\") pod \"multus-7slzh\" (UID: \"93d7d50e-b320-4217-b183-2bfc921f6c9d\") " pod="openshift-multus/multus-7slzh"
Nov 02 14:55:30.954092 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:30.954043    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/93d7d50e-b320-4217-b183-2bfc921f6c9d-hostroot\") pod \"multus-7slzh\" (UID: \"93d7d50e-b320-4217-b183-2bfc921f6c9d\") " pod="openshift-multus/multus-7slzh"
Nov 02 14:55:30.954092 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:30.954068    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/93d7d50e-b320-4217-b183-2bfc921f6c9d-multus-conf-dir\") pod \"multus-7slzh\" (UID: \"93d7d50e-b320-4217-b183-2bfc921f6c9d\") " pod="openshift-multus/multus-7slzh"
Nov 02 14:55:30.954421 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:30.954114    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7fe9ba5e-a867-4ac3-8a6f-a316c6f64db4-host-slash\") pod \"iptables-alerter-pl8gn\" (UID: \"7fe9ba5e-a867-4ac3-8a6f-a316c6f64db4\") " pod="openshift-network-operator/iptables-alerter-pl8gn"
Nov 02 14:55:30.954421 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:30.954181    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/65ce39fc-05c7-433b-81f4-c1b219f1bd71-host-slash\") pod \"ovnkube-node-f8btv\" (UID: \"65ce39fc-05c7-433b-81f4-c1b219f1bd71\") " pod="openshift-ovn-kubernetes/ovnkube-node-f8btv"
Nov 02 14:55:30.954421 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:30.954219    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/65ce39fc-05c7-433b-81f4-c1b219f1bd71-node-log\") pod \"ovnkube-node-f8btv\" (UID: \"65ce39fc-05c7-433b-81f4-c1b219f1bd71\") " pod="openshift-ovn-kubernetes/ovnkube-node-f8btv"
Nov 02 14:55:30.954421 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:30.954246    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/65ce39fc-05c7-433b-81f4-c1b219f1bd71-host-cni-bin\") pod \"ovnkube-node-f8btv\" (UID: \"65ce39fc-05c7-433b-81f4-c1b219f1bd71\") " pod="openshift-ovn-kubernetes/ovnkube-node-f8btv"
Nov 02 14:55:30.954421 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:30.954289    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/b1715623-afd0-4d0e-9b63-b38499cec1e2-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-fbnnl\" (UID: \"b1715623-afd0-4d0e-9b63-b38499cec1e2\") " pod="openshift-multus/multus-additional-cni-plugins-fbnnl"
Nov 02 14:55:30.954421 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:30.954333    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/65ce39fc-05c7-433b-81f4-c1b219f1bd71-systemd-units\") pod \"ovnkube-node-f8btv\" (UID: \"65ce39fc-05c7-433b-81f4-c1b219f1bd71\") " pod="openshift-ovn-kubernetes/ovnkube-node-f8btv"
Nov 02 14:55:30.954421 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:30.954365    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/65ce39fc-05c7-433b-81f4-c1b219f1bd71-run-openvswitch\") pod \"ovnkube-node-f8btv\" (UID: \"65ce39fc-05c7-433b-81f4-c1b219f1bd71\") " pod="openshift-ovn-kubernetes/ovnkube-node-f8btv"
Nov 02 14:55:30.954421 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:30.954392    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/65ce39fc-05c7-433b-81f4-c1b219f1bd71-env-overrides\") pod \"ovnkube-node-f8btv\" (UID: \"65ce39fc-05c7-433b-81f4-c1b219f1bd71\") " pod="openshift-ovn-kubernetes/ovnkube-node-f8btv"
Nov 02 14:55:30.954822 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:30.954422    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/68b781b2-be96-4c5c-8daa-2238fd0a9623-agent-certs\") pod \"konnectivity-agent-d5t6w\" (UID: \"68b781b2-be96-4c5c-8daa-2238fd0a9623\") " pod="kube-system/konnectivity-agent-d5t6w"
Nov 02 14:55:30.954822 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:30.954446    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e1da99c3-8294-4e34-9d33-47e04a6182d3-tmp\") pod \"tuned-sq4l4\" (UID: \"e1da99c3-8294-4e34-9d33-47e04a6182d3\") " pod="openshift-cluster-node-tuning-operator/tuned-sq4l4"
Nov 02 14:55:30.954822 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:30.954476    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/65ce39fc-05c7-433b-81f4-c1b219f1bd71-ovnkube-script-lib\") pod \"ovnkube-node-f8btv\" (UID: \"65ce39fc-05c7-433b-81f4-c1b219f1bd71\") " pod="openshift-ovn-kubernetes/ovnkube-node-f8btv"
Nov 02 14:55:30.954822 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:30.954502    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/54dc6898-f041-4ed6-9fc9-1b5159d00d27-socket-dir\") pod \"aws-ebs-csi-driver-node-6cj82\" (UID: \"54dc6898-f041-4ed6-9fc9-1b5159d00d27\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6cj82"
Nov 02 14:55:30.954822 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:30.954543    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hz5t9\" (UniqueName: \"kubernetes.io/projected/caa8898b-6611-49b8-845d-3b9bdac6691f-kube-api-access-hz5t9\") pod \"network-metrics-daemon-484ld\" (UID: \"caa8898b-6611-49b8-845d-3b9bdac6691f\") " pod="openshift-multus/network-metrics-daemon-484ld"
Nov 02 14:55:30.954822 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:30.954569    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e1da99c3-8294-4e34-9d33-47e04a6182d3-sys\") pod \"tuned-sq4l4\" (UID: \"e1da99c3-8294-4e34-9d33-47e04a6182d3\") " pod="openshift-cluster-node-tuning-operator/tuned-sq4l4"
Nov 02 14:55:30.954822 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:30.954593    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e1da99c3-8294-4e34-9d33-47e04a6182d3-var-lib-kubelet\") pod \"tuned-sq4l4\" (UID: \"e1da99c3-8294-4e34-9d33-47e04a6182d3\") " pod="openshift-cluster-node-tuning-operator/tuned-sq4l4"
Nov 02 14:55:30.954822 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:30.954626    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b1715623-afd0-4d0e-9b63-b38499cec1e2-cni-binary-copy\") pod \"multus-additional-cni-plugins-fbnnl\" (UID: \"b1715623-afd0-4d0e-9b63-b38499cec1e2\") " pod="openshift-multus/multus-additional-cni-plugins-fbnnl"
Nov 02 14:55:30.954822 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:30.954652    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/65ce39fc-05c7-433b-81f4-c1b219f1bd71-host-cni-netd\") pod \"ovnkube-node-f8btv\" (UID: \"65ce39fc-05c7-433b-81f4-c1b219f1bd71\") " pod="openshift-ovn-kubernetes/ovnkube-node-f8btv"
Nov 02 14:55:30.954822 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:30.954677    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/720ef290-024b-4ed2-8417-92290f360dc9-hosts-file\") pod \"node-resolver-jpz27\" (UID: \"720ef290-024b-4ed2-8417-92290f360dc9\") " pod="openshift-dns/node-resolver-jpz27"
Nov 02 14:55:30.954822 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:30.954703    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/68b781b2-be96-4c5c-8daa-2238fd0a9623-konnectivity-ca\") pod \"konnectivity-agent-d5t6w\" (UID: \"68b781b2-be96-4c5c-8daa-2238fd0a9623\") " pod="kube-system/konnectivity-agent-d5t6w"
Nov 02 14:55:30.954822 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:30.954728    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/54dc6898-f041-4ed6-9fc9-1b5159d00d27-device-dir\") pod \"aws-ebs-csi-driver-node-6cj82\" (UID: \"54dc6898-f041-4ed6-9fc9-1b5159d00d27\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6cj82"
Nov 02 14:55:30.954822 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:30.954806    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/54dc6898-f041-4ed6-9fc9-1b5159d00d27-etc-selinux\") pod \"aws-ebs-csi-driver-node-6cj82\" (UID: \"54dc6898-f041-4ed6-9fc9-1b5159d00d27\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6cj82"
Nov 02 14:55:30.955506 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:30.954840    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e4983b70-40f4-4e1e-a5da-80194d5d4786-host\") pod \"node-ca-q9nx7\" (UID: \"e4983b70-40f4-4e1e-a5da-80194d5d4786\") " pod="openshift-image-registry/node-ca-q9nx7"
Nov 02 14:55:30.955506 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:30.954867    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7758v\" (UniqueName: \"kubernetes.io/projected/e4983b70-40f4-4e1e-a5da-80194d5d4786-kube-api-access-7758v\") pod \"node-ca-q9nx7\" (UID: \"e4983b70-40f4-4e1e-a5da-80194d5d4786\") " pod="openshift-image-registry/node-ca-q9nx7"
Nov 02 14:55:30.955506 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:30.954892    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/93d7d50e-b320-4217-b183-2bfc921f6c9d-host-var-lib-cni-multus\") pod \"multus-7slzh\" (UID: \"93d7d50e-b320-4217-b183-2bfc921f6c9d\") " pod="openshift-multus/multus-7slzh"
Nov 02 14:55:30.955506 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:30.954921    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/65ce39fc-05c7-433b-81f4-c1b219f1bd71-ovn-node-metrics-cert\") pod \"ovnkube-node-f8btv\" (UID: \"65ce39fc-05c7-433b-81f4-c1b219f1bd71\") " pod="openshift-ovn-kubernetes/ovnkube-node-f8btv"
Nov 02 14:55:30.955506 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:30.954950    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/332f4a2d-a241-4c71-8ebb-a99618c0d3dc-dbus\") pod \"global-pull-secret-syncer-vl6zb\" (UID: \"332f4a2d-a241-4c71-8ebb-a99618c0d3dc\") " pod="kube-system/global-pull-secret-syncer-vl6zb"
Nov 02 14:55:30.955506 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:30.954976    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/65ce39fc-05c7-433b-81f4-c1b219f1bd71-run-ovn\") pod \"ovnkube-node-f8btv\" (UID: \"65ce39fc-05c7-433b-81f4-c1b219f1bd71\") " pod="openshift-ovn-kubernetes/ovnkube-node-f8btv"
Nov 02 14:55:30.955506 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:30.955000    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/93d7d50e-b320-4217-b183-2bfc921f6c9d-multus-daemon-config\") pod \"multus-7slzh\" (UID: \"93d7d50e-b320-4217-b183-2bfc921f6c9d\") " pod="openshift-multus/multus-7slzh"
Nov 02 14:55:30.955506 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:30.955033    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/93d7d50e-b320-4217-b183-2bfc921f6c9d-host-run-multus-certs\") pod \"multus-7slzh\" (UID: \"93d7d50e-b320-4217-b183-2bfc921f6c9d\") " pod="openshift-multus/multus-7slzh"
Nov 02 14:55:30.955506 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:30.955060    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e1da99c3-8294-4e34-9d33-47e04a6182d3-etc-sysctl-conf\") pod \"tuned-sq4l4\" (UID: \"e1da99c3-8294-4e34-9d33-47e04a6182d3\") " pod="openshift-cluster-node-tuning-operator/tuned-sq4l4"
Nov 02 14:55:30.955506 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:30.955085    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e1da99c3-8294-4e34-9d33-47e04a6182d3-etc-tuned\") pod \"tuned-sq4l4\" (UID: \"e1da99c3-8294-4e34-9d33-47e04a6182d3\") " pod="openshift-cluster-node-tuning-operator/tuned-sq4l4"
Nov 02 14:55:30.955506 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:30.955111    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/7fe9ba5e-a867-4ac3-8a6f-a316c6f64db4-iptables-alerter-script\") pod \"iptables-alerter-pl8gn\" (UID: \"7fe9ba5e-a867-4ac3-8a6f-a316c6f64db4\") " pod="openshift-network-operator/iptables-alerter-pl8gn"
Nov 02 14:55:30.955506 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:30.955137    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/65ce39fc-05c7-433b-81f4-c1b219f1bd71-host-run-ovn-kubernetes\") pod \"ovnkube-node-f8btv\" (UID: \"65ce39fc-05c7-433b-81f4-c1b219f1bd71\") " pod="openshift-ovn-kubernetes/ovnkube-node-f8btv"
Nov 02 14:55:30.955506 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:30.955171    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ps49c\" (UniqueName: \"kubernetes.io/projected/65ce39fc-05c7-433b-81f4-c1b219f1bd71-kube-api-access-ps49c\") pod \"ovnkube-node-f8btv\" (UID: \"65ce39fc-05c7-433b-81f4-c1b219f1bd71\") " pod="openshift-ovn-kubernetes/ovnkube-node-f8btv"
Nov 02 14:55:30.955506 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:30.955196    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/54dc6898-f041-4ed6-9fc9-1b5159d00d27-sys-fs\") pod \"aws-ebs-csi-driver-node-6cj82\" (UID: \"54dc6898-f041-4ed6-9fc9-1b5159d00d27\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6cj82"
Nov 02 14:55:30.955506 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:30.955239    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/93d7d50e-b320-4217-b183-2bfc921f6c9d-system-cni-dir\") pod \"multus-7slzh\" (UID: \"93d7d50e-b320-4217-b183-2bfc921f6c9d\") " pod="openshift-multus/multus-7slzh"
Nov 02 14:55:30.955506 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:30.955355    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/93d7d50e-b320-4217-b183-2bfc921f6c9d-multus-cni-dir\") pod \"multus-7slzh\" (UID: \"93d7d50e-b320-4217-b183-2bfc921f6c9d\") " pod="openshift-multus/multus-7slzh"
Nov 02 14:55:30.955506 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:30.955381    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/93d7d50e-b320-4217-b183-2bfc921f6c9d-cnibin\") pod \"multus-7slzh\" (UID: \"93d7d50e-b320-4217-b183-2bfc921f6c9d\") " pod="openshift-multus/multus-7slzh"
Nov 02 14:55:30.956382 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:30.955410    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e1da99c3-8294-4e34-9d33-47e04a6182d3-run\") pod \"tuned-sq4l4\" (UID: \"e1da99c3-8294-4e34-9d33-47e04a6182d3\") " pod="openshift-cluster-node-tuning-operator/tuned-sq4l4"
Nov 02 14:55:30.956382 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:30.955437    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/caa8898b-6611-49b8-845d-3b9bdac6691f-metrics-certs\") pod \"network-metrics-daemon-484ld\" (UID: \"caa8898b-6611-49b8-845d-3b9bdac6691f\") " pod="openshift-multus/network-metrics-daemon-484ld"
Nov 02 14:55:30.956382 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:30.955464    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hpfk5\" (UniqueName: \"kubernetes.io/projected/0d6c5872-e278-4463-953d-e1388a001d0d-kube-api-access-hpfk5\") pod \"network-check-target-wns8r\" (UID: \"0d6c5872-e278-4463-953d-e1388a001d0d\") " pod="openshift-network-diagnostics/network-check-target-wns8r"
Nov 02 14:55:30.956382 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:30.955494    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e1da99c3-8294-4e34-9d33-47e04a6182d3-etc-sysctl-d\") pod \"tuned-sq4l4\" (UID: \"e1da99c3-8294-4e34-9d33-47e04a6182d3\") " pod="openshift-cluster-node-tuning-operator/tuned-sq4l4"
Nov 02 14:55:30.956382 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:30.955520    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/b1715623-afd0-4d0e-9b63-b38499cec1e2-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-fbnnl\" (UID: \"b1715623-afd0-4d0e-9b63-b38499cec1e2\") " pod="openshift-multus/multus-additional-cni-plugins-fbnnl"
Nov 02 14:55:30.956382 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:30.955558    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/65ce39fc-05c7-433b-81f4-c1b219f1bd71-host-run-netns\") pod \"ovnkube-node-f8btv\" (UID: \"65ce39fc-05c7-433b-81f4-c1b219f1bd71\") " pod="openshift-ovn-kubernetes/ovnkube-node-f8btv"
Nov 02 14:55:30.956382 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:30.955586    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/65ce39fc-05c7-433b-81f4-c1b219f1bd71-var-lib-openvswitch\") pod \"ovnkube-node-f8btv\" (UID: \"65ce39fc-05c7-433b-81f4-c1b219f1bd71\") " pod="openshift-ovn-kubernetes/ovnkube-node-f8btv"
Nov 02 14:55:30.956382 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:30.955613    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/65ce39fc-05c7-433b-81f4-c1b219f1bd71-ovnkube-config\") pod \"ovnkube-node-f8btv\" (UID: \"65ce39fc-05c7-433b-81f4-c1b219f1bd71\") " pod="openshift-ovn-kubernetes/ovnkube-node-f8btv"
Nov 02 14:55:30.956382 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:30.955639    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/93d7d50e-b320-4217-b183-2bfc921f6c9d-multus-socket-dir-parent\") pod \"multus-7slzh\" (UID: \"93d7d50e-b320-4217-b183-2bfc921f6c9d\") " pod="openshift-multus/multus-7slzh"
Nov 02 14:55:30.956382 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:30.955666    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ndjwt\" (UniqueName: \"kubernetes.io/projected/93d7d50e-b320-4217-b183-2bfc921f6c9d-kube-api-access-ndjwt\") pod \"multus-7slzh\" (UID: \"93d7d50e-b320-4217-b183-2bfc921f6c9d\") " pod="openshift-multus/multus-7slzh"
Nov 02 14:55:30.956382 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:30.955696    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e1da99c3-8294-4e34-9d33-47e04a6182d3-etc-modprobe-d\") pod \"tuned-sq4l4\" (UID: \"e1da99c3-8294-4e34-9d33-47e04a6182d3\") " pod="openshift-cluster-node-tuning-operator/tuned-sq4l4"
Nov 02 14:55:30.956382 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:30.955722    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e1da99c3-8294-4e34-9d33-47e04a6182d3-lib-modules\") pod \"tuned-sq4l4\" (UID: \"e1da99c3-8294-4e34-9d33-47e04a6182d3\") " pod="openshift-cluster-node-tuning-operator/tuned-sq4l4"
Nov 02 14:55:30.956382 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:30.955748    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/332f4a2d-a241-4c71-8ebb-a99618c0d3dc-original-pull-secret\") pod \"global-pull-secret-syncer-vl6zb\" (UID: \"332f4a2d-a241-4c71-8ebb-a99618c0d3dc\") " pod="kube-system/global-pull-secret-syncer-vl6zb"
Nov 02 14:55:30.956382 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:30.955775    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/65ce39fc-05c7-433b-81f4-c1b219f1bd71-log-socket\") pod \"ovnkube-node-f8btv\" (UID: \"65ce39fc-05c7-433b-81f4-c1b219f1bd71\") " pod="openshift-ovn-kubernetes/ovnkube-node-f8btv"
Nov 02 14:55:30.956382 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:30.955800    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/54dc6898-f041-4ed6-9fc9-1b5159d00d27-registration-dir\") pod \"aws-ebs-csi-driver-node-6cj82\" (UID: \"54dc6898-f041-4ed6-9fc9-1b5159d00d27\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6cj82"
Nov 02 14:55:30.956382 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:30.955839    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e4983b70-40f4-4e1e-a5da-80194d5d4786-serviceca\") pod \"node-ca-q9nx7\" (UID: \"e4983b70-40f4-4e1e-a5da-80194d5d4786\") " pod="openshift-image-registry/node-ca-q9nx7"
Nov 02 14:55:30.957192 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:30.955866    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e1da99c3-8294-4e34-9d33-47e04a6182d3-etc-sysconfig\") pod \"tuned-sq4l4\" (UID: \"e1da99c3-8294-4e34-9d33-47e04a6182d3\") " pod="openshift-cluster-node-tuning-operator/tuned-sq4l4"
Nov 02 14:55:30.957192 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:30.955894    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b1715623-afd0-4d0e-9b63-b38499cec1e2-tuning-conf-dir\") pod \"multus-additional-cni-plugins-fbnnl\" (UID: \"b1715623-afd0-4d0e-9b63-b38499cec1e2\") " pod="openshift-multus/multus-additional-cni-plugins-fbnnl"
Nov 02 14:55:30.957192 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:30.955921    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f28t5\" (UniqueName: \"kubernetes.io/projected/720ef290-024b-4ed2-8417-92290f360dc9-kube-api-access-f28t5\") pod \"node-resolver-jpz27\" (UID: \"720ef290-024b-4ed2-8417-92290f360dc9\") " pod="openshift-dns/node-resolver-jpz27"
Nov 02 14:55:30.957192 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:30.955949    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/93d7d50e-b320-4217-b183-2bfc921f6c9d-host-run-k8s-cni-cncf-io\") pod \"multus-7slzh\" (UID: \"93d7d50e-b320-4217-b183-2bfc921f6c9d\") " pod="openshift-multus/multus-7slzh"
Nov 02 14:55:30.957192 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:30.955980    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e1da99c3-8294-4e34-9d33-47e04a6182d3-etc-kubernetes\") pod \"tuned-sq4l4\" (UID: \"e1da99c3-8294-4e34-9d33-47e04a6182d3\") " pod="openshift-cluster-node-tuning-operator/tuned-sq4l4"
Nov 02 14:55:30.957192 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:30.956008    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e1da99c3-8294-4e34-9d33-47e04a6182d3-host\") pod \"tuned-sq4l4\" (UID: \"e1da99c3-8294-4e34-9d33-47e04a6182d3\") " pod="openshift-cluster-node-tuning-operator/tuned-sq4l4"
Nov 02 14:55:30.957192 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:30.956033    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/332f4a2d-a241-4c71-8ebb-a99618c0d3dc-kubelet-config\") pod \"global-pull-secret-syncer-vl6zb\" (UID: \"332f4a2d-a241-4c71-8ebb-a99618c0d3dc\") " pod="kube-system/global-pull-secret-syncer-vl6zb"
Nov 02 14:55:30.957192 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:30.956062    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8rqfc\" (UniqueName: \"kubernetes.io/projected/7fe9ba5e-a867-4ac3-8a6f-a316c6f64db4-kube-api-access-8rqfc\") pod \"iptables-alerter-pl8gn\" (UID: \"7fe9ba5e-a867-4ac3-8a6f-a316c6f64db4\") " pod="openshift-network-operator/iptables-alerter-pl8gn"
Nov 02 14:55:30.957192 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:30.956089    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/54dc6898-f041-4ed6-9fc9-1b5159d00d27-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6cj82\" (UID: \"54dc6898-f041-4ed6-9fc9-1b5159d00d27\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6cj82"
Nov 02 14:55:30.957192 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:30.956116    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/93d7d50e-b320-4217-b183-2bfc921f6c9d-host-run-netns\") pod \"multus-7slzh\" (UID: \"93d7d50e-b320-4217-b183-2bfc921f6c9d\") " pod="openshift-multus/multus-7slzh"
Nov 02 14:55:30.957192 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:30.956147    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e1da99c3-8294-4e34-9d33-47e04a6182d3-etc-systemd\") pod \"tuned-sq4l4\" (UID: \"e1da99c3-8294-4e34-9d33-47e04a6182d3\") " pod="openshift-cluster-node-tuning-operator/tuned-sq4l4"
Nov 02 14:55:30.957192 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:30.956173    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/720ef290-024b-4ed2-8417-92290f360dc9-tmp-dir\") pod \"node-resolver-jpz27\" (UID: \"720ef290-024b-4ed2-8417-92290f360dc9\") " pod="openshift-dns/node-resolver-jpz27"
Nov 02 14:55:30.957192 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:30.956223    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8bt8x\" (UniqueName: \"kubernetes.io/projected/54dc6898-f041-4ed6-9fc9-1b5159d00d27-kube-api-access-8bt8x\") pod \"aws-ebs-csi-driver-node-6cj82\" (UID: \"54dc6898-f041-4ed6-9fc9-1b5159d00d27\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6cj82"
Nov 02 14:55:30.957192 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:30.956252    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b1715623-afd0-4d0e-9b63-b38499cec1e2-system-cni-dir\") pod \"multus-additional-cni-plugins-fbnnl\" (UID: \"b1715623-afd0-4d0e-9b63-b38499cec1e2\") " pod="openshift-multus/multus-additional-cni-plugins-fbnnl"
Nov 02 14:55:30.971563 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:30.971512    2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-138.ec2.internal" event={"ID":"a98e99ec11875ba668469a960777e8a7","Type":"ContainerStarted","Data":"9d80c77407d3e67f6198e00be9d36a6e7f9c402b652e9d03c548331917a7ac4c"}
Nov 02 14:55:30.973044 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:30.973015    2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-138.ec2.internal" event={"ID":"9a8741d4cd8a99bb77c79c43a5144068","Type":"ContainerStarted","Data":"777cd23a55bfad60d07f0c9cc7b69832952228d10ba770ce6b855beb1aa7fb9f"}
Nov 02 14:55:30.989219 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:30.989180    2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Nov 02 14:55:30.989395 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:30.989307    2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Nov 02 14:55:30.989467 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:30.989432    2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Nov 02 14:55:30.989467 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:30.989438    2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Nov 02 14:55:30.989573 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:30.989565    2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Nov 02 14:55:30.989785 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:30.989656    2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Nov 02 14:55:30.989785 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:30.989668    2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Nov 02 14:55:30.989785 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:30.989708    2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Nov 02 14:55:30.989785 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:30.989722    2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-7pvkf\""
Nov 02 14:55:30.989785 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:30.989745    2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-kws2w\""
Nov 02 14:55:30.990039 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:30.989855    2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Nov 02 14:55:30.990039 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:30.989884    2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Nov 02 14:55:30.990039 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:30.989901    2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-62zsq\""
Nov 02 14:55:30.990495 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:30.990247    2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Nov 02 14:55:30.990495 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:30.990335    2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Nov 02 14:55:30.990495 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:30.990372    2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Nov 02 14:55:30.990495 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:30.990464    2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Nov 02 14:55:30.991639 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:30.991617    2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Nov 02 14:55:30.991760 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:30.991648    2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Nov 02 14:55:30.991760 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:30.991657    2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Nov 02 14:55:30.991760 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:30.991659    2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-v7sjw\""
Nov 02 14:55:30.991760 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:30.991692    2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Nov 02 14:55:31.003706 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:31.003688    2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Nov 02 14:55:31.004789 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:31.004342    2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Nov 02 14:55:31.004789 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:31.004388    2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-2fmch\""
Nov 02 14:55:31.004789 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:31.004404    2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-25j7m\""
Nov 02 14:55:31.004789 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:31.004392    2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Nov 02 14:55:31.004789 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:31.004405    2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-x78pf\""
Nov 02 14:55:31.004789 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:31.004431    2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Nov 02 14:55:31.004789 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:31.004460    2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Nov 02 14:55:31.056827 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:31.056788    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e1da99c3-8294-4e34-9d33-47e04a6182d3-lib-modules\") pod \"tuned-sq4l4\" (UID: \"e1da99c3-8294-4e34-9d33-47e04a6182d3\") " pod="openshift-cluster-node-tuning-operator/tuned-sq4l4"
Nov 02 14:55:31.057010 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:31.056833    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/332f4a2d-a241-4c71-8ebb-a99618c0d3dc-original-pull-secret\") pod \"global-pull-secret-syncer-vl6zb\" (UID: \"332f4a2d-a241-4c71-8ebb-a99618c0d3dc\") " pod="kube-system/global-pull-secret-syncer-vl6zb"
Nov 02 14:55:31.057010 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:31.056861    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/65ce39fc-05c7-433b-81f4-c1b219f1bd71-log-socket\") pod \"ovnkube-node-f8btv\" (UID: \"65ce39fc-05c7-433b-81f4-c1b219f1bd71\") " pod="openshift-ovn-kubernetes/ovnkube-node-f8btv"
Nov 02 14:55:31.057010 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:31.056888    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/54dc6898-f041-4ed6-9fc9-1b5159d00d27-registration-dir\") pod \"aws-ebs-csi-driver-node-6cj82\" (UID: \"54dc6898-f041-4ed6-9fc9-1b5159d00d27\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6cj82"
Nov 02 14:55:31.057010 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:31.056910    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e4983b70-40f4-4e1e-a5da-80194d5d4786-serviceca\") pod \"node-ca-q9nx7\" (UID: \"e4983b70-40f4-4e1e-a5da-80194d5d4786\") " pod="openshift-image-registry/node-ca-q9nx7"
Nov 02 14:55:31.057010 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:31.056932    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e1da99c3-8294-4e34-9d33-47e04a6182d3-etc-sysconfig\") pod \"tuned-sq4l4\" (UID: \"e1da99c3-8294-4e34-9d33-47e04a6182d3\") " pod="openshift-cluster-node-tuning-operator/tuned-sq4l4"
Nov 02 14:55:31.057010 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:31.056957    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b1715623-afd0-4d0e-9b63-b38499cec1e2-tuning-conf-dir\") pod \"multus-additional-cni-plugins-fbnnl\" (UID: \"b1715623-afd0-4d0e-9b63-b38499cec1e2\") " pod="openshift-multus/multus-additional-cni-plugins-fbnnl"
Nov 02 14:55:31.057010 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:31.056969    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e1da99c3-8294-4e34-9d33-47e04a6182d3-lib-modules\") pod \"tuned-sq4l4\" (UID: \"e1da99c3-8294-4e34-9d33-47e04a6182d3\") " pod="openshift-cluster-node-tuning-operator/tuned-sq4l4"
Nov 02 14:55:31.057010 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:31.056982    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f28t5\" (UniqueName: \"kubernetes.io/projected/720ef290-024b-4ed2-8417-92290f360dc9-kube-api-access-f28t5\") pod \"node-resolver-jpz27\" (UID: \"720ef290-024b-4ed2-8417-92290f360dc9\") " pod="openshift-dns/node-resolver-jpz27"
Nov 02 14:55:31.057010 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:31.057007    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/93d7d50e-b320-4217-b183-2bfc921f6c9d-host-run-k8s-cni-cncf-io\") pod \"multus-7slzh\" (UID: \"93d7d50e-b320-4217-b183-2bfc921f6c9d\") " pod="openshift-multus/multus-7slzh"
Nov 02 14:55:31.057385 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:31.057032    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e1da99c3-8294-4e34-9d33-47e04a6182d3-etc-kubernetes\") pod \"tuned-sq4l4\" (UID: \"e1da99c3-8294-4e34-9d33-47e04a6182d3\") " pod="openshift-cluster-node-tuning-operator/tuned-sq4l4"
Nov 02 14:55:31.057385 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:31.057047    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/54dc6898-f041-4ed6-9fc9-1b5159d00d27-registration-dir\") pod \"aws-ebs-csi-driver-node-6cj82\" (UID: \"54dc6898-f041-4ed6-9fc9-1b5159d00d27\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6cj82"
Nov 02 14:55:31.057385 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:31.057056    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e1da99c3-8294-4e34-9d33-47e04a6182d3-host\") pod \"tuned-sq4l4\" (UID: \"e1da99c3-8294-4e34-9d33-47e04a6182d3\") " pod="openshift-cluster-node-tuning-operator/tuned-sq4l4"
Nov 02 14:55:31.057385 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:31.057110    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e1da99c3-8294-4e34-9d33-47e04a6182d3-host\") pod \"tuned-sq4l4\" (UID: \"e1da99c3-8294-4e34-9d33-47e04a6182d3\") " pod="openshift-cluster-node-tuning-operator/tuned-sq4l4"
Nov 02 14:55:31.057385 ip-10-0-135-138 kubenswrapper[2566]: E1102 14:55:31.057183    2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 02 14:55:31.057385 ip-10-0-135-138 kubenswrapper[2566]: E1102 14:55:31.057256    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/332f4a2d-a241-4c71-8ebb-a99618c0d3dc-original-pull-secret podName:332f4a2d-a241-4c71-8ebb-a99618c0d3dc nodeName:}" failed. No retries permitted until 2026-11-02 14:55:31.557234668 +0000 UTC m=+3.076257954 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/332f4a2d-a241-4c71-8ebb-a99618c0d3dc-original-pull-secret") pod "global-pull-secret-syncer-vl6zb" (UID: "332f4a2d-a241-4c71-8ebb-a99618c0d3dc") : object "kube-system"/"original-pull-secret" not registered
Nov 02 14:55:31.057685 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:31.057436    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/332f4a2d-a241-4c71-8ebb-a99618c0d3dc-kubelet-config\") pod \"global-pull-secret-syncer-vl6zb\" (UID: \"332f4a2d-a241-4c71-8ebb-a99618c0d3dc\") " pod="kube-system/global-pull-secret-syncer-vl6zb"
Nov 02 14:55:31.057685 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:31.057497    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8rqfc\" (UniqueName: \"kubernetes.io/projected/7fe9ba5e-a867-4ac3-8a6f-a316c6f64db4-kube-api-access-8rqfc\") pod \"iptables-alerter-pl8gn\" (UID: \"7fe9ba5e-a867-4ac3-8a6f-a316c6f64db4\") " pod="openshift-network-operator/iptables-alerter-pl8gn"
Nov 02 14:55:31.057685 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:31.057524    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/65ce39fc-05c7-433b-81f4-c1b219f1bd71-log-socket\") pod \"ovnkube-node-f8btv\" (UID: \"65ce39fc-05c7-433b-81f4-c1b219f1bd71\") " pod="openshift-ovn-kubernetes/ovnkube-node-f8btv"
Nov 02 14:55:31.057685 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:31.057527    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/54dc6898-f041-4ed6-9fc9-1b5159d00d27-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6cj82\" (UID: \"54dc6898-f041-4ed6-9fc9-1b5159d00d27\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6cj82"
Nov 02 14:55:31.057685 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:31.057572    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/93d7d50e-b320-4217-b183-2bfc921f6c9d-host-run-netns\") pod \"multus-7slzh\" (UID: \"93d7d50e-b320-4217-b183-2bfc921f6c9d\") " pod="openshift-multus/multus-7slzh"
Nov 02 14:55:31.057685 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:31.057599    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/54dc6898-f041-4ed6-9fc9-1b5159d00d27-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6cj82\" (UID: \"54dc6898-f041-4ed6-9fc9-1b5159d00d27\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6cj82"
Nov 02 14:55:31.057685 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:31.057601    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e1da99c3-8294-4e34-9d33-47e04a6182d3-etc-systemd\") pod \"tuned-sq4l4\" (UID: \"e1da99c3-8294-4e34-9d33-47e04a6182d3\") " pod="openshift-cluster-node-tuning-operator/tuned-sq4l4"
Nov 02 14:55:31.057685 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:31.057624    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e4983b70-40f4-4e1e-a5da-80194d5d4786-serviceca\") pod \"node-ca-q9nx7\" (UID: \"e4983b70-40f4-4e1e-a5da-80194d5d4786\") " pod="openshift-image-registry/node-ca-q9nx7"
Nov 02 14:55:31.057685 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:31.057636    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/720ef290-024b-4ed2-8417-92290f360dc9-tmp-dir\") pod \"node-resolver-jpz27\" (UID: \"720ef290-024b-4ed2-8417-92290f360dc9\") " pod="openshift-dns/node-resolver-jpz27"
Nov 02 14:55:31.057685 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:31.057643    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e1da99c3-8294-4e34-9d33-47e04a6182d3-etc-systemd\") pod \"tuned-sq4l4\" (UID: \"e1da99c3-8294-4e34-9d33-47e04a6182d3\") " pod="openshift-cluster-node-tuning-operator/tuned-sq4l4"
Nov 02 14:55:31.057685 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:31.057666    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8bt8x\" (UniqueName: \"kubernetes.io/projected/54dc6898-f041-4ed6-9fc9-1b5159d00d27-kube-api-access-8bt8x\") pod \"aws-ebs-csi-driver-node-6cj82\" (UID: \"54dc6898-f041-4ed6-9fc9-1b5159d00d27\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6cj82"
Nov 02 14:55:31.058120 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:31.057693    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b1715623-afd0-4d0e-9b63-b38499cec1e2-system-cni-dir\") pod \"multus-additional-cni-plugins-fbnnl\" (UID: \"b1715623-afd0-4d0e-9b63-b38499cec1e2\") " pod="openshift-multus/multus-additional-cni-plugins-fbnnl"
Nov 02 14:55:31.058120 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:31.057696    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/93d7d50e-b320-4217-b183-2bfc921f6c9d-host-run-netns\") pod \"multus-7slzh\" (UID: \"93d7d50e-b320-4217-b183-2bfc921f6c9d\") " pod="openshift-multus/multus-7slzh"
Nov 02 14:55:31.058120 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:31.057708    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e1da99c3-8294-4e34-9d33-47e04a6182d3-etc-sysconfig\") pod \"tuned-sq4l4\" (UID: \"e1da99c3-8294-4e34-9d33-47e04a6182d3\") " pod="openshift-cluster-node-tuning-operator/tuned-sq4l4"
Nov 02 14:55:31.058120 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:31.057724    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/65ce39fc-05c7-433b-81f4-c1b219f1bd71-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-f8btv\" (UID: \"65ce39fc-05c7-433b-81f4-c1b219f1bd71\") " pod="openshift-ovn-kubernetes/ovnkube-node-f8btv"
Nov 02 14:55:31.058120 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:31.057902    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/332f4a2d-a241-4c71-8ebb-a99618c0d3dc-kubelet-config\") pod \"global-pull-secret-syncer-vl6zb\" (UID: \"332f4a2d-a241-4c71-8ebb-a99618c0d3dc\") " pod="kube-system/global-pull-secret-syncer-vl6zb"
Nov 02 14:55:31.058120 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:31.057952    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/65ce39fc-05c7-433b-81f4-c1b219f1bd71-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-f8btv\" (UID: \"65ce39fc-05c7-433b-81f4-c1b219f1bd71\") " pod="openshift-ovn-kubernetes/ovnkube-node-f8btv"
Nov 02 14:55:31.058120 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:31.058022    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e1da99c3-8294-4e34-9d33-47e04a6182d3-etc-kubernetes\") pod \"tuned-sq4l4\" (UID: \"e1da99c3-8294-4e34-9d33-47e04a6182d3\") " pod="openshift-cluster-node-tuning-operator/tuned-sq4l4"
Nov 02 14:55:31.058120 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:31.058092    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b1715623-afd0-4d0e-9b63-b38499cec1e2-tuning-conf-dir\") pod \"multus-additional-cni-plugins-fbnnl\" (UID: \"b1715623-afd0-4d0e-9b63-b38499cec1e2\") " pod="openshift-multus/multus-additional-cni-plugins-fbnnl"
Nov 02 14:55:31.058475 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:31.058127    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/93d7d50e-b320-4217-b183-2bfc921f6c9d-cni-binary-copy\") pod \"multus-7slzh\" (UID: \"93d7d50e-b320-4217-b183-2bfc921f6c9d\") " pod="openshift-multus/multus-7slzh"
Nov 02 14:55:31.058475 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:31.058145    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/93d7d50e-b320-4217-b183-2bfc921f6c9d-etc-kubernetes\") pod \"multus-7slzh\" (UID: \"93d7d50e-b320-4217-b183-2bfc921f6c9d\") " pod="openshift-multus/multus-7slzh"
Nov 02 14:55:31.058475 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:31.058160    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-prmck\" (UniqueName: \"kubernetes.io/projected/e1da99c3-8294-4e34-9d33-47e04a6182d3-kube-api-access-prmck\") pod \"tuned-sq4l4\" (UID: \"e1da99c3-8294-4e34-9d33-47e04a6182d3\") " pod="openshift-cluster-node-tuning-operator/tuned-sq4l4"
Nov 02 14:55:31.058475 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:31.058177    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-548bv\" (UniqueName: \"kubernetes.io/projected/b1715623-afd0-4d0e-9b63-b38499cec1e2-kube-api-access-548bv\") pod \"multus-additional-cni-plugins-fbnnl\" (UID: \"b1715623-afd0-4d0e-9b63-b38499cec1e2\") " pod="openshift-multus/multus-additional-cni-plugins-fbnnl"
Nov 02 14:55:31.058475 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:31.058298    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b1715623-afd0-4d0e-9b63-b38499cec1e2-system-cni-dir\") pod \"multus-additional-cni-plugins-fbnnl\" (UID: \"b1715623-afd0-4d0e-9b63-b38499cec1e2\") " pod="openshift-multus/multus-additional-cni-plugins-fbnnl"
Nov 02 14:55:31.058475 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:31.058324    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/93d7d50e-b320-4217-b183-2bfc921f6c9d-etc-kubernetes\") pod \"multus-7slzh\" (UID: \"93d7d50e-b320-4217-b183-2bfc921f6c9d\") " pod="openshift-multus/multus-7slzh"
Nov 02 14:55:31.058475 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:31.058335    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/720ef290-024b-4ed2-8417-92290f360dc9-tmp-dir\") pod \"node-resolver-jpz27\" (UID: \"720ef290-024b-4ed2-8417-92290f360dc9\") " pod="openshift-dns/node-resolver-jpz27"
Nov 02 14:55:31.058475 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:31.058387    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/65ce39fc-05c7-433b-81f4-c1b219f1bd71-run-systemd\") pod \"ovnkube-node-f8btv\" (UID: \"65ce39fc-05c7-433b-81f4-c1b219f1bd71\") " pod="openshift-ovn-kubernetes/ovnkube-node-f8btv"
Nov 02 14:55:31.058475 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:31.058418    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/65ce39fc-05c7-433b-81f4-c1b219f1bd71-etc-openvswitch\") pod \"ovnkube-node-f8btv\" (UID: \"65ce39fc-05c7-433b-81f4-c1b219f1bd71\") " pod="openshift-ovn-kubernetes/ovnkube-node-f8btv"
Nov 02 14:55:31.058475 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:31.058433    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/93d7d50e-b320-4217-b183-2bfc921f6c9d-host-run-k8s-cni-cncf-io\") pod \"multus-7slzh\" (UID: \"93d7d50e-b320-4217-b183-2bfc921f6c9d\") " pod="openshift-multus/multus-7slzh"
Nov 02 14:55:31.058475 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:31.058473    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b1715623-afd0-4d0e-9b63-b38499cec1e2-cnibin\") pod \"multus-additional-cni-plugins-fbnnl\" (UID: \"b1715623-afd0-4d0e-9b63-b38499cec1e2\") " pod="openshift-multus/multus-additional-cni-plugins-fbnnl"
Nov 02 14:55:31.059074 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:31.058491    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/65ce39fc-05c7-433b-81f4-c1b219f1bd71-run-systemd\") pod \"ovnkube-node-f8btv\" (UID: \"65ce39fc-05c7-433b-81f4-c1b219f1bd71\") " pod="openshift-ovn-kubernetes/ovnkube-node-f8btv"
Nov 02 14:55:31.059074 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:31.058493    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/65ce39fc-05c7-433b-81f4-c1b219f1bd71-etc-openvswitch\") pod \"ovnkube-node-f8btv\" (UID: \"65ce39fc-05c7-433b-81f4-c1b219f1bd71\") " pod="openshift-ovn-kubernetes/ovnkube-node-f8btv"
Nov 02 14:55:31.059074 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:31.058546    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/65ce39fc-05c7-433b-81f4-c1b219f1bd71-host-kubelet\") pod \"ovnkube-node-f8btv\" (UID: \"65ce39fc-05c7-433b-81f4-c1b219f1bd71\") " pod="openshift-ovn-kubernetes/ovnkube-node-f8btv"
Nov 02 14:55:31.059074 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:31.058620    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b1715623-afd0-4d0e-9b63-b38499cec1e2-cnibin\") pod \"multus-additional-cni-plugins-fbnnl\" (UID: \"b1715623-afd0-4d0e-9b63-b38499cec1e2\") " pod="openshift-multus/multus-additional-cni-plugins-fbnnl"
Nov 02 14:55:31.059074 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:31.058666    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/93d7d50e-b320-4217-b183-2bfc921f6c9d-host-var-lib-cni-bin\") pod \"multus-7slzh\" (UID: \"93d7d50e-b320-4217-b183-2bfc921f6c9d\") " pod="openshift-multus/multus-7slzh"
Nov 02 14:55:31.059074 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:31.058729    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/93d7d50e-b320-4217-b183-2bfc921f6c9d-host-var-lib-kubelet\") pod \"multus-7slzh\" (UID: \"93d7d50e-b320-4217-b183-2bfc921f6c9d\") " pod="openshift-multus/multus-7slzh"
Nov 02 14:55:31.059074 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:31.058762    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/93d7d50e-b320-4217-b183-2bfc921f6c9d-host-var-lib-cni-bin\") pod \"multus-7slzh\" (UID: \"93d7d50e-b320-4217-b183-2bfc921f6c9d\") " pod="openshift-multus/multus-7slzh"
Nov 02 14:55:31.059074 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:31.058771    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/93d7d50e-b320-4217-b183-2bfc921f6c9d-hostroot\") pod \"multus-7slzh\" (UID: \"93d7d50e-b320-4217-b183-2bfc921f6c9d\") " pod="openshift-multus/multus-7slzh"
Nov 02 14:55:31.059074 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:31.058729    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/65ce39fc-05c7-433b-81f4-c1b219f1bd71-host-kubelet\") pod \"ovnkube-node-f8btv\" (UID: \"65ce39fc-05c7-433b-81f4-c1b219f1bd71\") " pod="openshift-ovn-kubernetes/ovnkube-node-f8btv"
Nov 02 14:55:31.059074 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:31.058796    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/93d7d50e-b320-4217-b183-2bfc921f6c9d-multus-conf-dir\") pod \"multus-7slzh\" (UID: \"93d7d50e-b320-4217-b183-2bfc921f6c9d\") " pod="openshift-multus/multus-7slzh"
Nov 02 14:55:31.059074 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:31.058790    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/93d7d50e-b320-4217-b183-2bfc921f6c9d-host-var-lib-kubelet\") pod \"multus-7slzh\" (UID: \"93d7d50e-b320-4217-b183-2bfc921f6c9d\") " pod="openshift-multus/multus-7slzh"
Nov 02 14:55:31.059074 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:31.058824    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7fe9ba5e-a867-4ac3-8a6f-a316c6f64db4-host-slash\") pod \"iptables-alerter-pl8gn\" (UID: \"7fe9ba5e-a867-4ac3-8a6f-a316c6f64db4\") " pod="openshift-network-operator/iptables-alerter-pl8gn"
Nov 02 14:55:31.059074 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:31.058830    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/93d7d50e-b320-4217-b183-2bfc921f6c9d-hostroot\") pod \"multus-7slzh\" (UID: \"93d7d50e-b320-4217-b183-2bfc921f6c9d\") " pod="openshift-multus/multus-7slzh"
Nov 02 14:55:31.059074 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:31.058848    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/65ce39fc-05c7-433b-81f4-c1b219f1bd71-host-slash\") pod \"ovnkube-node-f8btv\" (UID: \"65ce39fc-05c7-433b-81f4-c1b219f1bd71\") " pod="openshift-ovn-kubernetes/ovnkube-node-f8btv"
Nov 02 14:55:31.059074 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:31.058858    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/93d7d50e-b320-4217-b183-2bfc921f6c9d-multus-conf-dir\") pod \"multus-7slzh\" (UID: \"93d7d50e-b320-4217-b183-2bfc921f6c9d\") " pod="openshift-multus/multus-7slzh"
Nov 02 14:55:31.059074 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:31.058873    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/65ce39fc-05c7-433b-81f4-c1b219f1bd71-node-log\") pod \"ovnkube-node-f8btv\" (UID: \"65ce39fc-05c7-433b-81f4-c1b219f1bd71\") " pod="openshift-ovn-kubernetes/ovnkube-node-f8btv"
Nov 02 14:55:31.059074 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:31.058879    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7fe9ba5e-a867-4ac3-8a6f-a316c6f64db4-host-slash\") pod \"iptables-alerter-pl8gn\" (UID: \"7fe9ba5e-a867-4ac3-8a6f-a316c6f64db4\") " pod="openshift-network-operator/iptables-alerter-pl8gn"
Nov 02 14:55:31.059074 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:31.058899    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/65ce39fc-05c7-433b-81f4-c1b219f1bd71-host-cni-bin\") pod \"ovnkube-node-f8btv\" (UID: \"65ce39fc-05c7-433b-81f4-c1b219f1bd71\") " pod="openshift-ovn-kubernetes/ovnkube-node-f8btv"
Nov 02 14:55:31.059861 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:31.058924    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/65ce39fc-05c7-433b-81f4-c1b219f1bd71-node-log\") pod \"ovnkube-node-f8btv\" (UID: \"65ce39fc-05c7-433b-81f4-c1b219f1bd71\") " pod="openshift-ovn-kubernetes/ovnkube-node-f8btv"
Nov 02 14:55:31.059861 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:31.058933    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/b1715623-afd0-4d0e-9b63-b38499cec1e2-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-fbnnl\" (UID: \"b1715623-afd0-4d0e-9b63-b38499cec1e2\") " pod="openshift-multus/multus-additional-cni-plugins-fbnnl"
Nov 02 14:55:31.059861 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:31.058906    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/65ce39fc-05c7-433b-81f4-c1b219f1bd71-host-slash\") pod \"ovnkube-node-f8btv\" (UID: \"65ce39fc-05c7-433b-81f4-c1b219f1bd71\") " pod="openshift-ovn-kubernetes/ovnkube-node-f8btv"
Nov 02 14:55:31.059861 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:31.058947    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/65ce39fc-05c7-433b-81f4-c1b219f1bd71-host-cni-bin\") pod \"ovnkube-node-f8btv\" (UID: \"65ce39fc-05c7-433b-81f4-c1b219f1bd71\") " pod="openshift-ovn-kubernetes/ovnkube-node-f8btv"
Nov 02 14:55:31.059861 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:31.058963    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/65ce39fc-05c7-433b-81f4-c1b219f1bd71-systemd-units\") pod \"ovnkube-node-f8btv\" (UID: \"65ce39fc-05c7-433b-81f4-c1b219f1bd71\") " pod="openshift-ovn-kubernetes/ovnkube-node-f8btv"
Nov 02 14:55:31.059861 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:31.059020    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/65ce39fc-05c7-433b-81f4-c1b219f1bd71-systemd-units\") pod \"ovnkube-node-f8btv\" (UID: \"65ce39fc-05c7-433b-81f4-c1b219f1bd71\") " pod="openshift-ovn-kubernetes/ovnkube-node-f8btv"
Nov 02 14:55:31.059861 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:31.059023    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/93d7d50e-b320-4217-b183-2bfc921f6c9d-cni-binary-copy\") pod \"multus-7slzh\" (UID: \"93d7d50e-b320-4217-b183-2bfc921f6c9d\") " pod="openshift-multus/multus-7slzh"
Nov 02 14:55:31.059861 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:31.059035    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/65ce39fc-05c7-433b-81f4-c1b219f1bd71-run-openvswitch\") pod \"ovnkube-node-f8btv\" (UID: \"65ce39fc-05c7-433b-81f4-c1b219f1bd71\") " pod="openshift-ovn-kubernetes/ovnkube-node-f8btv"
Nov 02 14:55:31.059861 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:31.059072    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/65ce39fc-05c7-433b-81f4-c1b219f1bd71-run-openvswitch\") pod \"ovnkube-node-f8btv\" (UID: \"65ce39fc-05c7-433b-81f4-c1b219f1bd71\") " pod="openshift-ovn-kubernetes/ovnkube-node-f8btv"
Nov 02 14:55:31.059861 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:31.059073    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/65ce39fc-05c7-433b-81f4-c1b219f1bd71-env-overrides\") pod \"ovnkube-node-f8btv\" (UID: \"65ce39fc-05c7-433b-81f4-c1b219f1bd71\") " pod="openshift-ovn-kubernetes/ovnkube-node-f8btv"
Nov 02 14:55:31.059861 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:31.059096    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/68b781b2-be96-4c5c-8daa-2238fd0a9623-agent-certs\") pod \"konnectivity-agent-d5t6w\" (UID: \"68b781b2-be96-4c5c-8daa-2238fd0a9623\") " pod="kube-system/konnectivity-agent-d5t6w"
Nov 02 14:55:31.059861 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:31.059122    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e1da99c3-8294-4e34-9d33-47e04a6182d3-tmp\") pod \"tuned-sq4l4\" (UID: \"e1da99c3-8294-4e34-9d33-47e04a6182d3\") " pod="openshift-cluster-node-tuning-operator/tuned-sq4l4"
Nov 02 14:55:31.059861 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:31.059147    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/65ce39fc-05c7-433b-81f4-c1b219f1bd71-ovnkube-script-lib\") pod \"ovnkube-node-f8btv\" (UID: \"65ce39fc-05c7-433b-81f4-c1b219f1bd71\") " pod="openshift-ovn-kubernetes/ovnkube-node-f8btv"
Nov 02 14:55:31.059861 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:31.059174    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/54dc6898-f041-4ed6-9fc9-1b5159d00d27-socket-dir\") pod \"aws-ebs-csi-driver-node-6cj82\" (UID: \"54dc6898-f041-4ed6-9fc9-1b5159d00d27\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6cj82"
Nov 02 14:55:31.059861 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:31.059216    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hz5t9\" (UniqueName: \"kubernetes.io/projected/caa8898b-6611-49b8-845d-3b9bdac6691f-kube-api-access-hz5t9\") pod \"network-metrics-daemon-484ld\" (UID: \"caa8898b-6611-49b8-845d-3b9bdac6691f\") " pod="openshift-multus/network-metrics-daemon-484ld"
Nov 02 14:55:31.059861 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:31.059240    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e1da99c3-8294-4e34-9d33-47e04a6182d3-sys\") pod \"tuned-sq4l4\" (UID: \"e1da99c3-8294-4e34-9d33-47e04a6182d3\") " pod="openshift-cluster-node-tuning-operator/tuned-sq4l4"
Nov 02 14:55:31.059861 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:31.059263    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e1da99c3-8294-4e34-9d33-47e04a6182d3-var-lib-kubelet\") pod \"tuned-sq4l4\" (UID: \"e1da99c3-8294-4e34-9d33-47e04a6182d3\") " pod="openshift-cluster-node-tuning-operator/tuned-sq4l4"
Nov 02 14:55:31.060615 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:31.059282    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b1715623-afd0-4d0e-9b63-b38499cec1e2-cni-binary-copy\") pod \"multus-additional-cni-plugins-fbnnl\" (UID: \"b1715623-afd0-4d0e-9b63-b38499cec1e2\") " pod="openshift-multus/multus-additional-cni-plugins-fbnnl"
Nov 02 14:55:31.060615 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:31.059298    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/65ce39fc-05c7-433b-81f4-c1b219f1bd71-host-cni-netd\") pod \"ovnkube-node-f8btv\" (UID: \"65ce39fc-05c7-433b-81f4-c1b219f1bd71\") " pod="openshift-ovn-kubernetes/ovnkube-node-f8btv"
Nov 02 14:55:31.060615 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:31.059314    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/720ef290-024b-4ed2-8417-92290f360dc9-hosts-file\") pod \"node-resolver-jpz27\" (UID: \"720ef290-024b-4ed2-8417-92290f360dc9\") " pod="openshift-dns/node-resolver-jpz27"
Nov 02 14:55:31.060615 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:31.059331    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/68b781b2-be96-4c5c-8daa-2238fd0a9623-konnectivity-ca\") pod \"konnectivity-agent-d5t6w\" (UID: \"68b781b2-be96-4c5c-8daa-2238fd0a9623\") " pod="kube-system/konnectivity-agent-d5t6w"
Nov 02 14:55:31.060615 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:31.059350    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e1da99c3-8294-4e34-9d33-47e04a6182d3-sys\") pod \"tuned-sq4l4\" (UID: \"e1da99c3-8294-4e34-9d33-47e04a6182d3\") " pod="openshift-cluster-node-tuning-operator/tuned-sq4l4"
Nov 02 14:55:31.060615 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:31.059355    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/54dc6898-f041-4ed6-9fc9-1b5159d00d27-device-dir\") pod \"aws-ebs-csi-driver-node-6cj82\" (UID: \"54dc6898-f041-4ed6-9fc9-1b5159d00d27\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6cj82"
Nov 02 14:55:31.060615 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:31.059388    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/65ce39fc-05c7-433b-81f4-c1b219f1bd71-host-cni-netd\") pod \"ovnkube-node-f8btv\" (UID: \"65ce39fc-05c7-433b-81f4-c1b219f1bd71\") " pod="openshift-ovn-kubernetes/ovnkube-node-f8btv"
Nov 02 14:55:31.060615 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:31.059402    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/54dc6898-f041-4ed6-9fc9-1b5159d00d27-device-dir\") pod \"aws-ebs-csi-driver-node-6cj82\" (UID: \"54dc6898-f041-4ed6-9fc9-1b5159d00d27\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6cj82"
Nov 02 14:55:31.060615 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:31.059406    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/54dc6898-f041-4ed6-9fc9-1b5159d00d27-etc-selinux\") pod \"aws-ebs-csi-driver-node-6cj82\" (UID: \"54dc6898-f041-4ed6-9fc9-1b5159d00d27\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6cj82"
Nov 02 14:55:31.060615 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:31.059438    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e4983b70-40f4-4e1e-a5da-80194d5d4786-host\") pod \"node-ca-q9nx7\" (UID: \"e4983b70-40f4-4e1e-a5da-80194d5d4786\") " pod="openshift-image-registry/node-ca-q9nx7"
Nov 02 14:55:31.060615 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:31.059465    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7758v\" (UniqueName: \"kubernetes.io/projected/e4983b70-40f4-4e1e-a5da-80194d5d4786-kube-api-access-7758v\") pod \"node-ca-q9nx7\" (UID: \"e4983b70-40f4-4e1e-a5da-80194d5d4786\") " pod="openshift-image-registry/node-ca-q9nx7"
Nov 02 14:55:31.060615 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:31.059491    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/93d7d50e-b320-4217-b183-2bfc921f6c9d-host-var-lib-cni-multus\") pod \"multus-7slzh\" (UID: \"93d7d50e-b320-4217-b183-2bfc921f6c9d\") " pod="openshift-multus/multus-7slzh"
Nov 02 14:55:31.060615 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:31.059518    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/65ce39fc-05c7-433b-81f4-c1b219f1bd71-ovn-node-metrics-cert\") pod \"ovnkube-node-f8btv\" (UID: \"65ce39fc-05c7-433b-81f4-c1b219f1bd71\") " pod="openshift-ovn-kubernetes/ovnkube-node-f8btv"
Nov 02 14:55:31.060615 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:31.059523    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/54dc6898-f041-4ed6-9fc9-1b5159d00d27-etc-selinux\") pod \"aws-ebs-csi-driver-node-6cj82\" (UID: \"54dc6898-f041-4ed6-9fc9-1b5159d00d27\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6cj82"
Nov 02 14:55:31.060615 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:31.059526    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e4983b70-40f4-4e1e-a5da-80194d5d4786-host\") pod \"node-ca-q9nx7\" (UID: \"e4983b70-40f4-4e1e-a5da-80194d5d4786\") " pod="openshift-image-registry/node-ca-q9nx7"
Nov 02 14:55:31.060615 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:31.059543    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/332f4a2d-a241-4c71-8ebb-a99618c0d3dc-dbus\") pod \"global-pull-secret-syncer-vl6zb\" (UID: \"332f4a2d-a241-4c71-8ebb-a99618c0d3dc\") " pod="kube-system/global-pull-secret-syncer-vl6zb"
Nov 02 14:55:31.060615 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:31.059569    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/65ce39fc-05c7-433b-81f4-c1b219f1bd71-run-ovn\") pod \"ovnkube-node-f8btv\" (UID: \"65ce39fc-05c7-433b-81f4-c1b219f1bd71\") " pod="openshift-ovn-kubernetes/ovnkube-node-f8btv"
Nov 02 14:55:31.060615 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:31.059572    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/720ef290-024b-4ed2-8417-92290f360dc9-hosts-file\") pod \"node-resolver-jpz27\" (UID: \"720ef290-024b-4ed2-8417-92290f360dc9\") " pod="openshift-dns/node-resolver-jpz27"
Nov 02 14:55:31.061470 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:31.059585    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e1da99c3-8294-4e34-9d33-47e04a6182d3-var-lib-kubelet\") pod \"tuned-sq4l4\" (UID: \"e1da99c3-8294-4e34-9d33-47e04a6182d3\") " pod="openshift-cluster-node-tuning-operator/tuned-sq4l4"
Nov 02 14:55:31.061470 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:31.059594    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/93d7d50e-b320-4217-b183-2bfc921f6c9d-multus-daemon-config\") pod \"multus-7slzh\" (UID: \"93d7d50e-b320-4217-b183-2bfc921f6c9d\") " pod="openshift-multus/multus-7slzh"
Nov 02 14:55:31.061470 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:31.059783    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/93d7d50e-b320-4217-b183-2bfc921f6c9d-host-var-lib-cni-multus\") pod \"multus-7slzh\" (UID: \"93d7d50e-b320-4217-b183-2bfc921f6c9d\") " pod="openshift-multus/multus-7slzh"
Nov 02 14:55:31.061470 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:31.059791    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/65ce39fc-05c7-433b-81f4-c1b219f1bd71-run-ovn\") pod \"ovnkube-node-f8btv\" (UID: \"65ce39fc-05c7-433b-81f4-c1b219f1bd71\") " pod="openshift-ovn-kubernetes/ovnkube-node-f8btv"
Nov 02 14:55:31.061470 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:31.059830    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/93d7d50e-b320-4217-b183-2bfc921f6c9d-host-run-multus-certs\") pod \"multus-7slzh\" (UID: \"93d7d50e-b320-4217-b183-2bfc921f6c9d\") " pod="openshift-multus/multus-7slzh"
Nov 02 14:55:31.061470 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:31.059860    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e1da99c3-8294-4e34-9d33-47e04a6182d3-etc-sysctl-conf\") pod \"tuned-sq4l4\" (UID: \"e1da99c3-8294-4e34-9d33-47e04a6182d3\") " pod="openshift-cluster-node-tuning-operator/tuned-sq4l4"
Nov 02 14:55:31.061470 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:31.059889    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e1da99c3-8294-4e34-9d33-47e04a6182d3-etc-tuned\") pod \"tuned-sq4l4\" (UID: \"e1da99c3-8294-4e34-9d33-47e04a6182d3\") " pod="openshift-cluster-node-tuning-operator/tuned-sq4l4"
Nov 02 14:55:31.061470 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:31.059915    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/7fe9ba5e-a867-4ac3-8a6f-a316c6f64db4-iptables-alerter-script\") pod \"iptables-alerter-pl8gn\" (UID: \"7fe9ba5e-a867-4ac3-8a6f-a316c6f64db4\") " pod="openshift-network-operator/iptables-alerter-pl8gn"
Nov 02 14:55:31.061470 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:31.059943    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/65ce39fc-05c7-433b-81f4-c1b219f1bd71-host-run-ovn-kubernetes\") pod \"ovnkube-node-f8btv\" (UID: \"65ce39fc-05c7-433b-81f4-c1b219f1bd71\") " pod="openshift-ovn-kubernetes/ovnkube-node-f8btv"
Nov 02 14:55:31.061470 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:31.059955    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/332f4a2d-a241-4c71-8ebb-a99618c0d3dc-dbus\") pod \"global-pull-secret-syncer-vl6zb\" (UID: \"332f4a2d-a241-4c71-8ebb-a99618c0d3dc\") " pod="kube-system/global-pull-secret-syncer-vl6zb"
Nov 02 14:55:31.061470 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:31.059959    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ps49c\" (UniqueName: \"kubernetes.io/projected/65ce39fc-05c7-433b-81f4-c1b219f1bd71-kube-api-access-ps49c\") pod \"ovnkube-node-f8btv\" (UID: \"65ce39fc-05c7-433b-81f4-c1b219f1bd71\") " pod="openshift-ovn-kubernetes/ovnkube-node-f8btv"
Nov 02 14:55:31.061470 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:31.060180    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/54dc6898-f041-4ed6-9fc9-1b5159d00d27-sys-fs\") pod \"aws-ebs-csi-driver-node-6cj82\" (UID: \"54dc6898-f041-4ed6-9fc9-1b5159d00d27\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6cj82"
Nov 02 14:55:31.061470 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:31.060226    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/93d7d50e-b320-4217-b183-2bfc921f6c9d-system-cni-dir\") pod \"multus-7slzh\" (UID: \"93d7d50e-b320-4217-b183-2bfc921f6c9d\") " pod="openshift-multus/multus-7slzh"
Nov 02 14:55:31.061470 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:31.060257    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/93d7d50e-b320-4217-b183-2bfc921f6c9d-multus-cni-dir\") pod \"multus-7slzh\" (UID: \"93d7d50e-b320-4217-b183-2bfc921f6c9d\") " pod="openshift-multus/multus-7slzh"
Nov 02 14:55:31.061470 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:31.060283    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/93d7d50e-b320-4217-b183-2bfc921f6c9d-cnibin\") pod \"multus-7slzh\" (UID: \"93d7d50e-b320-4217-b183-2bfc921f6c9d\") " pod="openshift-multus/multus-7slzh"
Nov 02 14:55:31.061470 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:31.060306    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e1da99c3-8294-4e34-9d33-47e04a6182d3-run\") pod \"tuned-sq4l4\" (UID: \"e1da99c3-8294-4e34-9d33-47e04a6182d3\") " pod="openshift-cluster-node-tuning-operator/tuned-sq4l4"
Nov 02 14:55:31.061470 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:31.060332    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/caa8898b-6611-49b8-845d-3b9bdac6691f-metrics-certs\") pod \"network-metrics-daemon-484ld\" (UID: \"caa8898b-6611-49b8-845d-3b9bdac6691f\") " pod="openshift-multus/network-metrics-daemon-484ld"
Nov 02 14:55:31.061470 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:31.060343    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/54dc6898-f041-4ed6-9fc9-1b5159d00d27-socket-dir\") pod \"aws-ebs-csi-driver-node-6cj82\" (UID: \"54dc6898-f041-4ed6-9fc9-1b5159d00d27\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6cj82"
Nov 02 14:55:31.062428 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:31.060368    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/93d7d50e-b320-4217-b183-2bfc921f6c9d-host-run-multus-certs\") pod \"multus-7slzh\" (UID: \"93d7d50e-b320-4217-b183-2bfc921f6c9d\") " pod="openshift-multus/multus-7slzh"
Nov 02 14:55:31.062428 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:31.060425    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/54dc6898-f041-4ed6-9fc9-1b5159d00d27-sys-fs\") pod \"aws-ebs-csi-driver-node-6cj82\" (UID: \"54dc6898-f041-4ed6-9fc9-1b5159d00d27\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6cj82"
Nov 02 14:55:31.062428 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:31.060448    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/b1715623-afd0-4d0e-9b63-b38499cec1e2-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-fbnnl\" (UID: \"b1715623-afd0-4d0e-9b63-b38499cec1e2\") " pod="openshift-multus/multus-additional-cni-plugins-fbnnl"
Nov 02 14:55:31.062428 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:31.060474    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/93d7d50e-b320-4217-b183-2bfc921f6c9d-system-cni-dir\") pod \"multus-7slzh\" (UID: \"93d7d50e-b320-4217-b183-2bfc921f6c9d\") " pod="openshift-multus/multus-7slzh"
Nov 02 14:55:31.062428 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:31.060535    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/93d7d50e-b320-4217-b183-2bfc921f6c9d-multus-cni-dir\") pod \"multus-7slzh\" (UID: \"93d7d50e-b320-4217-b183-2bfc921f6c9d\") " pod="openshift-multus/multus-7slzh"
Nov 02 14:55:31.062428 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:31.060583    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/68b781b2-be96-4c5c-8daa-2238fd0a9623-konnectivity-ca\") pod \"konnectivity-agent-d5t6w\" (UID: \"68b781b2-be96-4c5c-8daa-2238fd0a9623\") " pod="kube-system/konnectivity-agent-d5t6w"
Nov 02 14:55:31.062428 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:31.060590    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/93d7d50e-b320-4217-b183-2bfc921f6c9d-cnibin\") pod \"multus-7slzh\" (UID: \"93d7d50e-b320-4217-b183-2bfc921f6c9d\") " pod="openshift-multus/multus-7slzh"
Nov 02 14:55:31.062428 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:31.060616    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/65ce39fc-05c7-433b-81f4-c1b219f1bd71-host-run-ovn-kubernetes\") pod \"ovnkube-node-f8btv\" (UID: \"65ce39fc-05c7-433b-81f4-c1b219f1bd71\") " pod="openshift-ovn-kubernetes/ovnkube-node-f8btv"
Nov 02 14:55:31.062428 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:31.060370    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hpfk5\" (UniqueName: \"kubernetes.io/projected/0d6c5872-e278-4463-953d-e1388a001d0d-kube-api-access-hpfk5\") pod \"network-check-target-wns8r\" (UID: \"0d6c5872-e278-4463-953d-e1388a001d0d\") " pod="openshift-network-diagnostics/network-check-target-wns8r"
Nov 02 14:55:31.062428 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:31.060653    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e1da99c3-8294-4e34-9d33-47e04a6182d3-etc-sysctl-d\") pod \"tuned-sq4l4\" (UID: \"e1da99c3-8294-4e34-9d33-47e04a6182d3\") " pod="openshift-cluster-node-tuning-operator/tuned-sq4l4"
Nov 02 14:55:31.062428 ip-10-0-135-138 kubenswrapper[2566]: E1102 14:55:31.060667    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 14:55:31.062428 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:31.060677    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/b1715623-afd0-4d0e-9b63-b38499cec1e2-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-fbnnl\" (UID: \"b1715623-afd0-4d0e-9b63-b38499cec1e2\") " pod="openshift-multus/multus-additional-cni-plugins-fbnnl"
Nov 02 14:55:31.062428 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:31.060702    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/65ce39fc-05c7-433b-81f4-c1b219f1bd71-host-run-netns\") pod \"ovnkube-node-f8btv\" (UID: \"65ce39fc-05c7-433b-81f4-c1b219f1bd71\") " pod="openshift-ovn-kubernetes/ovnkube-node-f8btv"
Nov 02 14:55:31.062428 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:31.060721    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/65ce39fc-05c7-433b-81f4-c1b219f1bd71-env-overrides\") pod \"ovnkube-node-f8btv\" (UID: \"65ce39fc-05c7-433b-81f4-c1b219f1bd71\") " pod="openshift-ovn-kubernetes/ovnkube-node-f8btv"
Nov 02 14:55:31.062428 ip-10-0-135-138 kubenswrapper[2566]: E1102 14:55:31.060730    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/caa8898b-6611-49b8-845d-3b9bdac6691f-metrics-certs podName:caa8898b-6611-49b8-845d-3b9bdac6691f nodeName:}" failed. No retries permitted until 2026-11-02 14:55:31.560714099 +0000 UTC m=+3.079737398 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/caa8898b-6611-49b8-845d-3b9bdac6691f-metrics-certs") pod "network-metrics-daemon-484ld" (UID: "caa8898b-6611-49b8-845d-3b9bdac6691f") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 14:55:31.062428 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:31.060739    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e1da99c3-8294-4e34-9d33-47e04a6182d3-etc-sysctl-conf\") pod \"tuned-sq4l4\" (UID: \"e1da99c3-8294-4e34-9d33-47e04a6182d3\") " pod="openshift-cluster-node-tuning-operator/tuned-sq4l4"
Nov 02 14:55:31.062428 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:31.060748    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/65ce39fc-05c7-433b-81f4-c1b219f1bd71-host-run-netns\") pod \"ovnkube-node-f8btv\" (UID: \"65ce39fc-05c7-433b-81f4-c1b219f1bd71\") " pod="openshift-ovn-kubernetes/ovnkube-node-f8btv"
Nov 02 14:55:31.063287 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:31.060751    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/65ce39fc-05c7-433b-81f4-c1b219f1bd71-var-lib-openvswitch\") pod \"ovnkube-node-f8btv\" (UID: \"65ce39fc-05c7-433b-81f4-c1b219f1bd71\") " pod="openshift-ovn-kubernetes/ovnkube-node-f8btv"
Nov 02 14:55:31.063287 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:31.060727    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/65ce39fc-05c7-433b-81f4-c1b219f1bd71-ovnkube-script-lib\") pod \"ovnkube-node-f8btv\" (UID: \"65ce39fc-05c7-433b-81f4-c1b219f1bd71\") " pod="openshift-ovn-kubernetes/ovnkube-node-f8btv"
Nov 02 14:55:31.063287 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:31.060822    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e1da99c3-8294-4e34-9d33-47e04a6182d3-run\") pod \"tuned-sq4l4\" (UID: \"e1da99c3-8294-4e34-9d33-47e04a6182d3\") " pod="openshift-cluster-node-tuning-operator/tuned-sq4l4"
Nov 02 14:55:31.063287 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:31.060802    2566 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Nov 02 14:55:31.063287 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:31.060884    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/65ce39fc-05c7-433b-81f4-c1b219f1bd71-var-lib-openvswitch\") pod \"ovnkube-node-f8btv\" (UID: \"65ce39fc-05c7-433b-81f4-c1b219f1bd71\") " pod="openshift-ovn-kubernetes/ovnkube-node-f8btv"
Nov 02 14:55:31.063287 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:31.060929    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/65ce39fc-05c7-433b-81f4-c1b219f1bd71-ovnkube-config\") pod \"ovnkube-node-f8btv\" (UID: \"65ce39fc-05c7-433b-81f4-c1b219f1bd71\") " pod="openshift-ovn-kubernetes/ovnkube-node-f8btv"
Nov 02 14:55:31.063287 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:31.060963    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e1da99c3-8294-4e34-9d33-47e04a6182d3-etc-sysctl-d\") pod \"tuned-sq4l4\" (UID: \"e1da99c3-8294-4e34-9d33-47e04a6182d3\") " pod="openshift-cluster-node-tuning-operator/tuned-sq4l4"
Nov 02 14:55:31.063287 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:31.060968    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/93d7d50e-b320-4217-b183-2bfc921f6c9d-multus-socket-dir-parent\") pod \"multus-7slzh\" (UID: \"93d7d50e-b320-4217-b183-2bfc921f6c9d\") " pod="openshift-multus/multus-7slzh"
Nov 02 14:55:31.063287 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:31.060995    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/7fe9ba5e-a867-4ac3-8a6f-a316c6f64db4-iptables-alerter-script\") pod \"iptables-alerter-pl8gn\" (UID: \"7fe9ba5e-a867-4ac3-8a6f-a316c6f64db4\") " pod="openshift-network-operator/iptables-alerter-pl8gn"
Nov 02 14:55:31.063287 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:31.061002    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ndjwt\" (UniqueName: \"kubernetes.io/projected/93d7d50e-b320-4217-b183-2bfc921f6c9d-kube-api-access-ndjwt\") pod \"multus-7slzh\" (UID: \"93d7d50e-b320-4217-b183-2bfc921f6c9d\") " pod="openshift-multus/multus-7slzh"
Nov 02 14:55:31.063287 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:31.061058    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/93d7d50e-b320-4217-b183-2bfc921f6c9d-multus-socket-dir-parent\") pod \"multus-7slzh\" (UID: \"93d7d50e-b320-4217-b183-2bfc921f6c9d\") " pod="openshift-multus/multus-7slzh"
Nov 02 14:55:31.063287 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:31.061064    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e1da99c3-8294-4e34-9d33-47e04a6182d3-etc-modprobe-d\") pod \"tuned-sq4l4\" (UID: \"e1da99c3-8294-4e34-9d33-47e04a6182d3\") " pod="openshift-cluster-node-tuning-operator/tuned-sq4l4"
Nov 02 14:55:31.063287 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:31.061187    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e1da99c3-8294-4e34-9d33-47e04a6182d3-etc-modprobe-d\") pod \"tuned-sq4l4\" (UID: \"e1da99c3-8294-4e34-9d33-47e04a6182d3\") " pod="openshift-cluster-node-tuning-operator/tuned-sq4l4"
Nov 02 14:55:31.063287 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:31.061522    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/65ce39fc-05c7-433b-81f4-c1b219f1bd71-ovnkube-config\") pod \"ovnkube-node-f8btv\" (UID: \"65ce39fc-05c7-433b-81f4-c1b219f1bd71\") " pod="openshift-ovn-kubernetes/ovnkube-node-f8btv"
Nov 02 14:55:31.063287 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:31.061589    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/93d7d50e-b320-4217-b183-2bfc921f6c9d-multus-daemon-config\") pod \"multus-7slzh\" (UID: \"93d7d50e-b320-4217-b183-2bfc921f6c9d\") " pod="openshift-multus/multus-7slzh"
Nov 02 14:55:31.063287 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:31.061733    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b1715623-afd0-4d0e-9b63-b38499cec1e2-cni-binary-copy\") pod \"multus-additional-cni-plugins-fbnnl\" (UID: \"b1715623-afd0-4d0e-9b63-b38499cec1e2\") " pod="openshift-multus/multus-additional-cni-plugins-fbnnl"
Nov 02 14:55:31.064124 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:31.063860    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/b1715623-afd0-4d0e-9b63-b38499cec1e2-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-fbnnl\" (UID: \"b1715623-afd0-4d0e-9b63-b38499cec1e2\") " pod="openshift-multus/multus-additional-cni-plugins-fbnnl"
Nov 02 14:55:31.068179 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:31.065365    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e1da99c3-8294-4e34-9d33-47e04a6182d3-etc-tuned\") pod \"tuned-sq4l4\" (UID: \"e1da99c3-8294-4e34-9d33-47e04a6182d3\") " pod="openshift-cluster-node-tuning-operator/tuned-sq4l4"
Nov 02 14:55:31.068179 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:31.065466    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/68b781b2-be96-4c5c-8daa-2238fd0a9623-agent-certs\") pod \"konnectivity-agent-d5t6w\" (UID: \"68b781b2-be96-4c5c-8daa-2238fd0a9623\") " pod="kube-system/konnectivity-agent-d5t6w"
Nov 02 14:55:31.068179 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:31.065469    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/65ce39fc-05c7-433b-81f4-c1b219f1bd71-ovn-node-metrics-cert\") pod \"ovnkube-node-f8btv\" (UID: \"65ce39fc-05c7-433b-81f4-c1b219f1bd71\") " pod="openshift-ovn-kubernetes/ovnkube-node-f8btv"
Nov 02 14:55:31.070005 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:31.069978    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e1da99c3-8294-4e34-9d33-47e04a6182d3-tmp\") pod \"tuned-sq4l4\" (UID: \"e1da99c3-8294-4e34-9d33-47e04a6182d3\") " pod="openshift-cluster-node-tuning-operator/tuned-sq4l4"
Nov 02 14:55:31.086806 ip-10-0-135-138 kubenswrapper[2566]: E1102 14:55:31.086772    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 02 14:55:31.086806 ip-10-0-135-138 kubenswrapper[2566]: E1102 14:55:31.086805    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 02 14:55:31.086806 ip-10-0-135-138 kubenswrapper[2566]: E1102 14:55:31.086821    2566 projected.go:196] Error preparing data for projected volume kube-api-access-hpfk5 for pod openshift-network-diagnostics/network-check-target-wns8r: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 14:55:31.087133 ip-10-0-135-138 kubenswrapper[2566]: E1102 14:55:31.086894    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0d6c5872-e278-4463-953d-e1388a001d0d-kube-api-access-hpfk5 podName:0d6c5872-e278-4463-953d-e1388a001d0d nodeName:}" failed. No retries permitted until 2026-11-02 14:55:31.586872841 +0000 UTC m=+3.105896145 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-hpfk5" (UniqueName: "kubernetes.io/projected/0d6c5872-e278-4463-953d-e1388a001d0d-kube-api-access-hpfk5") pod "network-check-target-wns8r" (UID: "0d6c5872-e278-4463-953d-e1388a001d0d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 14:55:31.091474 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:31.091443    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-548bv\" (UniqueName: \"kubernetes.io/projected/b1715623-afd0-4d0e-9b63-b38499cec1e2-kube-api-access-548bv\") pod \"multus-additional-cni-plugins-fbnnl\" (UID: \"b1715623-afd0-4d0e-9b63-b38499cec1e2\") " pod="openshift-multus/multus-additional-cni-plugins-fbnnl"
Nov 02 14:55:31.091610 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:31.091483    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-prmck\" (UniqueName: \"kubernetes.io/projected/e1da99c3-8294-4e34-9d33-47e04a6182d3-kube-api-access-prmck\") pod \"tuned-sq4l4\" (UID: \"e1da99c3-8294-4e34-9d33-47e04a6182d3\") " pod="openshift-cluster-node-tuning-operator/tuned-sq4l4"
Nov 02 14:55:31.091610 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:31.091557    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ps49c\" (UniqueName: \"kubernetes.io/projected/65ce39fc-05c7-433b-81f4-c1b219f1bd71-kube-api-access-ps49c\") pod \"ovnkube-node-f8btv\" (UID: \"65ce39fc-05c7-433b-81f4-c1b219f1bd71\") " pod="openshift-ovn-kubernetes/ovnkube-node-f8btv"
Nov 02 14:55:31.091610 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:31.091580    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f28t5\" (UniqueName: \"kubernetes.io/projected/720ef290-024b-4ed2-8417-92290f360dc9-kube-api-access-f28t5\") pod \"node-resolver-jpz27\" (UID: \"720ef290-024b-4ed2-8417-92290f360dc9\") " pod="openshift-dns/node-resolver-jpz27"
Nov 02 14:55:31.091997 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:31.091972    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8rqfc\" (UniqueName: \"kubernetes.io/projected/7fe9ba5e-a867-4ac3-8a6f-a316c6f64db4-kube-api-access-8rqfc\") pod \"iptables-alerter-pl8gn\" (UID: \"7fe9ba5e-a867-4ac3-8a6f-a316c6f64db4\") " pod="openshift-network-operator/iptables-alerter-pl8gn"
Nov 02 14:55:31.092143 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:31.092119    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7758v\" (UniqueName: \"kubernetes.io/projected/e4983b70-40f4-4e1e-a5da-80194d5d4786-kube-api-access-7758v\") pod \"node-ca-q9nx7\" (UID: \"e4983b70-40f4-4e1e-a5da-80194d5d4786\") " pod="openshift-image-registry/node-ca-q9nx7"
Nov 02 14:55:31.092238 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:31.092119    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ndjwt\" (UniqueName: \"kubernetes.io/projected/93d7d50e-b320-4217-b183-2bfc921f6c9d-kube-api-access-ndjwt\") pod \"multus-7slzh\" (UID: \"93d7d50e-b320-4217-b183-2bfc921f6c9d\") " pod="openshift-multus/multus-7slzh"
Nov 02 14:55:31.093091 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:31.093051    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8bt8x\" (UniqueName: \"kubernetes.io/projected/54dc6898-f041-4ed6-9fc9-1b5159d00d27-kube-api-access-8bt8x\") pod \"aws-ebs-csi-driver-node-6cj82\" (UID: \"54dc6898-f041-4ed6-9fc9-1b5159d00d27\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6cj82"
Nov 02 14:55:31.093527 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:31.093509    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hz5t9\" (UniqueName: \"kubernetes.io/projected/caa8898b-6611-49b8-845d-3b9bdac6691f-kube-api-access-hz5t9\") pod \"network-metrics-daemon-484ld\" (UID: \"caa8898b-6611-49b8-845d-3b9bdac6691f\") " pod="openshift-multus/network-metrics-daemon-484ld"
Nov 02 14:55:31.177907 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:31.177874    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-d5t6w"
Nov 02 14:55:31.187927 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:31.187892    2566 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68b781b2_be96_4c5c_8daa_2238fd0a9623.slice/crio-40258544b5fc72c08d5050ca375b814933292f7409af909ee8add2ebafb2e17f WatchSource:0}: Error finding container 40258544b5fc72c08d5050ca375b814933292f7409af909ee8add2ebafb2e17f: Status 404 returned error can't find the container with id 40258544b5fc72c08d5050ca375b814933292f7409af909ee8add2ebafb2e17f
Nov 02 14:55:31.190582 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:31.190558    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6cj82"
Nov 02 14:55:31.199332 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:31.199307    2566 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54dc6898_f041_4ed6_9fc9_1b5159d00d27.slice/crio-dedf3babecad2f468747d3e72210ea54c17d3ccf467e6671f136263e8ad8c84c WatchSource:0}: Error finding container dedf3babecad2f468747d3e72210ea54c17d3ccf467e6671f136263e8ad8c84c: Status 404 returned error can't find the container with id dedf3babecad2f468747d3e72210ea54c17d3ccf467e6671f136263e8ad8c84c
Nov 02 14:55:31.202129 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:31.201939    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-q9nx7"
Nov 02 14:55:31.206391 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:31.206247    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-sq4l4"
Nov 02 14:55:31.211697 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:31.211665    2566 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4983b70_40f4_4e1e_a5da_80194d5d4786.slice/crio-e13c510d9ba4b82e51fb2f10c06b32bbf9c8e5ebf6225c60f9db08c58e4c2b4c WatchSource:0}: Error finding container e13c510d9ba4b82e51fb2f10c06b32bbf9c8e5ebf6225c60f9db08c58e4c2b4c: Status 404 returned error can't find the container with id e13c510d9ba4b82e51fb2f10c06b32bbf9c8e5ebf6225c60f9db08c58e4c2b4c
Nov 02 14:55:31.218995 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:31.218963    2566 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1da99c3_8294_4e34_9d33_47e04a6182d3.slice/crio-b898b2da645b173244ea65fe7ca474f7ce11c7fe860c17dae8ff4ed8da332f86 WatchSource:0}: Error finding container b898b2da645b173244ea65fe7ca474f7ce11c7fe860c17dae8ff4ed8da332f86: Status 404 returned error can't find the container with id b898b2da645b173244ea65fe7ca474f7ce11c7fe860c17dae8ff4ed8da332f86
Nov 02 14:55:31.221430 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:31.221411    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-fbnnl"
Nov 02 14:55:31.223874 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:31.223850    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-pl8gn"
Nov 02 14:55:31.232304 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:31.232047    2566 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1715623_afd0_4d0e_9b63_b38499cec1e2.slice/crio-d1f7259e798f4c778135523c563af014d044be9176e87cba4459ed3b40ffab7d WatchSource:0}: Error finding container d1f7259e798f4c778135523c563af014d044be9176e87cba4459ed3b40ffab7d: Status 404 returned error can't find the container with id d1f7259e798f4c778135523c563af014d044be9176e87cba4459ed3b40ffab7d
Nov 02 14:55:31.232304 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:31.232129    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-7slzh"
Nov 02 14:55:31.236262 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:31.235889    2566 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7fe9ba5e_a867_4ac3_8a6f_a316c6f64db4.slice/crio-bf709fa05cba77da3f9435c58038e48838b3bfc405abff55963f4d883f2a19ba WatchSource:0}: Error finding container bf709fa05cba77da3f9435c58038e48838b3bfc405abff55963f4d883f2a19ba: Status 404 returned error can't find the container with id bf709fa05cba77da3f9435c58038e48838b3bfc405abff55963f4d883f2a19ba
Nov 02 14:55:31.243009 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:31.242979    2566 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93d7d50e_b320_4217_b183_2bfc921f6c9d.slice/crio-b7ab68c4d9b36e6507e0795c8bb246a797d2758dfa1a5dc067d56ef0ef5b459d WatchSource:0}: Error finding container b7ab68c4d9b36e6507e0795c8bb246a797d2758dfa1a5dc067d56ef0ef5b459d: Status 404 returned error can't find the container with id b7ab68c4d9b36e6507e0795c8bb246a797d2758dfa1a5dc067d56ef0ef5b459d
Nov 02 14:55:31.249623 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:31.249598    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-f8btv"
Nov 02 14:55:31.250622 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:31.250602    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-jpz27"
Nov 02 14:55:31.262110 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:55:31.262084    2566 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65ce39fc_05c7_433b_81f4_c1b219f1bd71.slice/crio-0aae85bb0b0b3446e320e80d65728fda1f1ac14b0af5de6c99416d8bde25db9d WatchSource:0}: Error finding container 0aae85bb0b0b3446e320e80d65728fda1f1ac14b0af5de6c99416d8bde25db9d: Status 404 returned error can't find the container with id 0aae85bb0b0b3446e320e80d65728fda1f1ac14b0af5de6c99416d8bde25db9d
Nov 02 14:55:31.564606 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:31.564571    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/caa8898b-6611-49b8-845d-3b9bdac6691f-metrics-certs\") pod \"network-metrics-daemon-484ld\" (UID: \"caa8898b-6611-49b8-845d-3b9bdac6691f\") " pod="openshift-multus/network-metrics-daemon-484ld"
Nov 02 14:55:31.564714 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:31.564643    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/332f4a2d-a241-4c71-8ebb-a99618c0d3dc-original-pull-secret\") pod \"global-pull-secret-syncer-vl6zb\" (UID: \"332f4a2d-a241-4c71-8ebb-a99618c0d3dc\") " pod="kube-system/global-pull-secret-syncer-vl6zb"
Nov 02 14:55:31.564767 ip-10-0-135-138 kubenswrapper[2566]: E1102 14:55:31.564721    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 14:55:31.564810 ip-10-0-135-138 kubenswrapper[2566]: E1102 14:55:31.564789    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/caa8898b-6611-49b8-845d-3b9bdac6691f-metrics-certs podName:caa8898b-6611-49b8-845d-3b9bdac6691f nodeName:}" failed. No retries permitted until 2026-11-02 14:55:32.564771438 +0000 UTC m=+4.083794743 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/caa8898b-6611-49b8-845d-3b9bdac6691f-metrics-certs") pod "network-metrics-daemon-484ld" (UID: "caa8898b-6611-49b8-845d-3b9bdac6691f") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 14:55:31.564810 ip-10-0-135-138 kubenswrapper[2566]: E1102 14:55:31.564789    2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 02 14:55:31.564889 ip-10-0-135-138 kubenswrapper[2566]: E1102 14:55:31.564844    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/332f4a2d-a241-4c71-8ebb-a99618c0d3dc-original-pull-secret podName:332f4a2d-a241-4c71-8ebb-a99618c0d3dc nodeName:}" failed. No retries permitted until 2026-11-02 14:55:32.564831418 +0000 UTC m=+4.083854699 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/332f4a2d-a241-4c71-8ebb-a99618c0d3dc-original-pull-secret") pod "global-pull-secret-syncer-vl6zb" (UID: "332f4a2d-a241-4c71-8ebb-a99618c0d3dc") : object "kube-system"/"original-pull-secret" not registered
Nov 02 14:55:31.666277 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:31.665577    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hpfk5\" (UniqueName: \"kubernetes.io/projected/0d6c5872-e278-4463-953d-e1388a001d0d-kube-api-access-hpfk5\") pod \"network-check-target-wns8r\" (UID: \"0d6c5872-e278-4463-953d-e1388a001d0d\") " pod="openshift-network-diagnostics/network-check-target-wns8r"
Nov 02 14:55:31.666277 ip-10-0-135-138 kubenswrapper[2566]: E1102 14:55:31.665796    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 02 14:55:31.666277 ip-10-0-135-138 kubenswrapper[2566]: E1102 14:55:31.665816    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 02 14:55:31.666277 ip-10-0-135-138 kubenswrapper[2566]: E1102 14:55:31.665829    2566 projected.go:196] Error preparing data for projected volume kube-api-access-hpfk5 for pod openshift-network-diagnostics/network-check-target-wns8r: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 14:55:31.666277 ip-10-0-135-138 kubenswrapper[2566]: E1102 14:55:31.665887    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0d6c5872-e278-4463-953d-e1388a001d0d-kube-api-access-hpfk5 podName:0d6c5872-e278-4463-953d-e1388a001d0d nodeName:}" failed. No retries permitted until 2026-11-02 14:55:32.665870233 +0000 UTC m=+4.184893516 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-hpfk5" (UniqueName: "kubernetes.io/projected/0d6c5872-e278-4463-953d-e1388a001d0d-kube-api-access-hpfk5") pod "network-check-target-wns8r" (UID: "0d6c5872-e278-4463-953d-e1388a001d0d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 14:55:31.904068 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:31.903933    2566 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-01 14:50:31 +0000 UTC" deadline="2028-08-02 06:23:36.643101216 +0000 UTC"
Nov 02 14:55:31.904068 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:31.904000    2566 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15327h28m4.739105424s"
Nov 02 14:55:31.984004 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:31.983958    2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6cj82" event={"ID":"54dc6898-f041-4ed6-9fc9-1b5159d00d27","Type":"ContainerStarted","Data":"dedf3babecad2f468747d3e72210ea54c17d3ccf467e6671f136263e8ad8c84c"}
Nov 02 14:55:31.988791 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:31.988713    2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-d5t6w" event={"ID":"68b781b2-be96-4c5c-8daa-2238fd0a9623","Type":"ContainerStarted","Data":"40258544b5fc72c08d5050ca375b814933292f7409af909ee8add2ebafb2e17f"}
Nov 02 14:55:31.994868 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:31.994814    2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-jpz27" event={"ID":"720ef290-024b-4ed2-8417-92290f360dc9","Type":"ContainerStarted","Data":"2315db45e98cf6c8210934b607b07cfcaf32866bfffd173ccb4d1c2baae1849c"}
Nov 02 14:55:32.002622 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:32.002567    2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-f8btv" event={"ID":"65ce39fc-05c7-433b-81f4-c1b219f1bd71","Type":"ContainerStarted","Data":"0aae85bb0b0b3446e320e80d65728fda1f1ac14b0af5de6c99416d8bde25db9d"}
Nov 02 14:55:32.019769 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:32.019734    2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-pl8gn" event={"ID":"7fe9ba5e-a867-4ac3-8a6f-a316c6f64db4","Type":"ContainerStarted","Data":"bf709fa05cba77da3f9435c58038e48838b3bfc405abff55963f4d883f2a19ba"}
Nov 02 14:55:32.028412 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:32.028376    2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-7slzh" event={"ID":"93d7d50e-b320-4217-b183-2bfc921f6c9d","Type":"ContainerStarted","Data":"b7ab68c4d9b36e6507e0795c8bb246a797d2758dfa1a5dc067d56ef0ef5b459d"}
Nov 02 14:55:32.044791 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:32.044686    2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fbnnl" event={"ID":"b1715623-afd0-4d0e-9b63-b38499cec1e2","Type":"ContainerStarted","Data":"d1f7259e798f4c778135523c563af014d044be9176e87cba4459ed3b40ffab7d"}
Nov 02 14:55:32.048824 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:32.048737    2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-sq4l4" event={"ID":"e1da99c3-8294-4e34-9d33-47e04a6182d3","Type":"ContainerStarted","Data":"b898b2da645b173244ea65fe7ca474f7ce11c7fe860c17dae8ff4ed8da332f86"}
Nov 02 14:55:32.057081 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:32.057051    2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-q9nx7" event={"ID":"e4983b70-40f4-4e1e-a5da-80194d5d4786","Type":"ContainerStarted","Data":"e13c510d9ba4b82e51fb2f10c06b32bbf9c8e5ebf6225c60f9db08c58e4c2b4c"}
Nov 02 14:55:32.573805 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:32.573721    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/caa8898b-6611-49b8-845d-3b9bdac6691f-metrics-certs\") pod \"network-metrics-daemon-484ld\" (UID: \"caa8898b-6611-49b8-845d-3b9bdac6691f\") " pod="openshift-multus/network-metrics-daemon-484ld"
Nov 02 14:55:32.573805 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:32.573772    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/332f4a2d-a241-4c71-8ebb-a99618c0d3dc-original-pull-secret\") pod \"global-pull-secret-syncer-vl6zb\" (UID: \"332f4a2d-a241-4c71-8ebb-a99618c0d3dc\") " pod="kube-system/global-pull-secret-syncer-vl6zb"
Nov 02 14:55:32.574060 ip-10-0-135-138 kubenswrapper[2566]: E1102 14:55:32.573896    2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 02 14:55:32.574060 ip-10-0-135-138 kubenswrapper[2566]: E1102 14:55:32.573964    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/332f4a2d-a241-4c71-8ebb-a99618c0d3dc-original-pull-secret podName:332f4a2d-a241-4c71-8ebb-a99618c0d3dc nodeName:}" failed. No retries permitted until 2026-11-02 14:55:34.573945767 +0000 UTC m=+6.092969064 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/332f4a2d-a241-4c71-8ebb-a99618c0d3dc-original-pull-secret") pod "global-pull-secret-syncer-vl6zb" (UID: "332f4a2d-a241-4c71-8ebb-a99618c0d3dc") : object "kube-system"/"original-pull-secret" not registered
Nov 02 14:55:32.574060 ip-10-0-135-138 kubenswrapper[2566]: E1102 14:55:32.574035    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 14:55:32.574303 ip-10-0-135-138 kubenswrapper[2566]: E1102 14:55:32.574072    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/caa8898b-6611-49b8-845d-3b9bdac6691f-metrics-certs podName:caa8898b-6611-49b8-845d-3b9bdac6691f nodeName:}" failed. No retries permitted until 2026-11-02 14:55:34.574058596 +0000 UTC m=+6.093081893 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/caa8898b-6611-49b8-845d-3b9bdac6691f-metrics-certs") pod "network-metrics-daemon-484ld" (UID: "caa8898b-6611-49b8-845d-3b9bdac6691f") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 14:55:32.674880 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:32.674833    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hpfk5\" (UniqueName: \"kubernetes.io/projected/0d6c5872-e278-4463-953d-e1388a001d0d-kube-api-access-hpfk5\") pod \"network-check-target-wns8r\" (UID: \"0d6c5872-e278-4463-953d-e1388a001d0d\") " pod="openshift-network-diagnostics/network-check-target-wns8r"
Nov 02 14:55:32.675063 ip-10-0-135-138 kubenswrapper[2566]: E1102 14:55:32.675049    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 02 14:55:32.675128 ip-10-0-135-138 kubenswrapper[2566]: E1102 14:55:32.675071    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 02 14:55:32.675128 ip-10-0-135-138 kubenswrapper[2566]: E1102 14:55:32.675084    2566 projected.go:196] Error preparing data for projected volume kube-api-access-hpfk5 for pod openshift-network-diagnostics/network-check-target-wns8r: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 14:55:32.675252 ip-10-0-135-138 kubenswrapper[2566]: E1102 14:55:32.675162    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0d6c5872-e278-4463-953d-e1388a001d0d-kube-api-access-hpfk5 podName:0d6c5872-e278-4463-953d-e1388a001d0d nodeName:}" failed. No retries permitted until 2026-11-02 14:55:34.675142262 +0000 UTC m=+6.194165566 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-hpfk5" (UniqueName: "kubernetes.io/projected/0d6c5872-e278-4463-953d-e1388a001d0d-kube-api-access-hpfk5") pod "network-check-target-wns8r" (UID: "0d6c5872-e278-4463-953d-e1388a001d0d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 14:55:32.904852 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:32.904756    2566 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-01 14:50:31 +0000 UTC" deadline="2028-07-26 09:52:29.643501768 +0000 UTC"
Nov 02 14:55:32.904852 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:32.904804    2566 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15162h56m56.738702645s"
Nov 02 14:55:32.969375 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:32.969344    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vl6zb"
Nov 02 14:55:32.969527 ip-10-0-135-138 kubenswrapper[2566]: E1102 14:55:32.969483    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vl6zb" podUID="332f4a2d-a241-4c71-8ebb-a99618c0d3dc"
Nov 02 14:55:32.969605 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:32.969573    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-484ld"
Nov 02 14:55:32.969707 ip-10-0-135-138 kubenswrapper[2566]: E1102 14:55:32.969665    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-484ld" podUID="caa8898b-6611-49b8-845d-3b9bdac6691f"
Nov 02 14:55:32.969773 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:32.969731    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wns8r"
Nov 02 14:55:32.969827 ip-10-0-135-138 kubenswrapper[2566]: E1102 14:55:32.969800    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wns8r" podUID="0d6c5872-e278-4463-953d-e1388a001d0d"
Nov 02 14:55:33.085318 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:33.085279    2566 generic.go:356] "Generic (PLEG): container finished" podID="a98e99ec11875ba668469a960777e8a7" containerID="12796333caa3b4bb02b5aaa645b4fe31c754e02b6d8f2ca0346e657eaeb5ab7b" exitCode=0
Nov 02 14:55:33.085518 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:33.085382    2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-138.ec2.internal" event={"ID":"a98e99ec11875ba668469a960777e8a7","Type":"ContainerDied","Data":"12796333caa3b4bb02b5aaa645b4fe31c754e02b6d8f2ca0346e657eaeb5ab7b"}
Nov 02 14:55:33.108942 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:33.108381    2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-138.ec2.internal" event={"ID":"9a8741d4cd8a99bb77c79c43a5144068","Type":"ContainerStarted","Data":"53cf9b2218c1a8bc9581d9f9c00cd11faa4cd2b7637385f17dc66335af2e8fae"}
Nov 02 14:55:34.143680 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:34.143352    2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-138.ec2.internal" event={"ID":"a98e99ec11875ba668469a960777e8a7","Type":"ContainerStarted","Data":"d3908d1d68fe643637975c994cda57abe46a3172eaa4bd89d5123ae4844c17b4"}
Nov 02 14:55:34.148918 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:34.148835    2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fbnnl" event={"ID":"b1715623-afd0-4d0e-9b63-b38499cec1e2","Type":"ContainerStarted","Data":"f62b7e1913bd0bd851d8ec494f42b43881661a821c8e5291037ff952aa42cef8"}
Nov 02 14:55:34.190148 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:34.190083    2566 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-138.ec2.internal" podStartSLOduration=4.190059003 podStartE2EDuration="4.190059003s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 14:55:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:55:33.17651109 +0000 UTC m=+4.695534394" watchObservedRunningTime="2026-11-02 14:55:34.190059003 +0000 UTC m=+5.709082307"
Nov 02 14:55:34.242650 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:34.242585    2566 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-138.ec2.internal" podStartSLOduration=4.242563167 podStartE2EDuration="4.242563167s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 14:55:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:55:34.19037947 +0000 UTC m=+5.709402777" watchObservedRunningTime="2026-11-02 14:55:34.242563167 +0000 UTC m=+5.761586473"
Nov 02 14:55:34.594949 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:34.594883    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/caa8898b-6611-49b8-845d-3b9bdac6691f-metrics-certs\") pod \"network-metrics-daemon-484ld\" (UID: \"caa8898b-6611-49b8-845d-3b9bdac6691f\") " pod="openshift-multus/network-metrics-daemon-484ld"
Nov 02 14:55:34.595173 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:34.594960    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/332f4a2d-a241-4c71-8ebb-a99618c0d3dc-original-pull-secret\") pod \"global-pull-secret-syncer-vl6zb\" (UID: \"332f4a2d-a241-4c71-8ebb-a99618c0d3dc\") " pod="kube-system/global-pull-secret-syncer-vl6zb"
Nov 02 14:55:34.595173 ip-10-0-135-138 kubenswrapper[2566]: E1102 14:55:34.595106    2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 02 14:55:34.595334 ip-10-0-135-138 kubenswrapper[2566]: E1102 14:55:34.595196    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/332f4a2d-a241-4c71-8ebb-a99618c0d3dc-original-pull-secret podName:332f4a2d-a241-4c71-8ebb-a99618c0d3dc nodeName:}" failed. No retries permitted until 2026-11-02 14:55:38.595153053 +0000 UTC m=+10.114176352 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/332f4a2d-a241-4c71-8ebb-a99618c0d3dc-original-pull-secret") pod "global-pull-secret-syncer-vl6zb" (UID: "332f4a2d-a241-4c71-8ebb-a99618c0d3dc") : object "kube-system"/"original-pull-secret" not registered
Nov 02 14:55:34.595649 ip-10-0-135-138 kubenswrapper[2566]: E1102 14:55:34.595626    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 14:55:34.595739 ip-10-0-135-138 kubenswrapper[2566]: E1102 14:55:34.595697    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/caa8898b-6611-49b8-845d-3b9bdac6691f-metrics-certs podName:caa8898b-6611-49b8-845d-3b9bdac6691f nodeName:}" failed. No retries permitted until 2026-11-02 14:55:38.595681534 +0000 UTC m=+10.114704829 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/caa8898b-6611-49b8-845d-3b9bdac6691f-metrics-certs") pod "network-metrics-daemon-484ld" (UID: "caa8898b-6611-49b8-845d-3b9bdac6691f") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 14:55:34.695413 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:34.695370    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hpfk5\" (UniqueName: \"kubernetes.io/projected/0d6c5872-e278-4463-953d-e1388a001d0d-kube-api-access-hpfk5\") pod \"network-check-target-wns8r\" (UID: \"0d6c5872-e278-4463-953d-e1388a001d0d\") " pod="openshift-network-diagnostics/network-check-target-wns8r"
Nov 02 14:55:34.695599 ip-10-0-135-138 kubenswrapper[2566]: E1102 14:55:34.695555    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 02 14:55:34.695599 ip-10-0-135-138 kubenswrapper[2566]: E1102 14:55:34.695581    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 02 14:55:34.695599 ip-10-0-135-138 kubenswrapper[2566]: E1102 14:55:34.695595    2566 projected.go:196] Error preparing data for projected volume kube-api-access-hpfk5 for pod openshift-network-diagnostics/network-check-target-wns8r: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 14:55:34.695771 ip-10-0-135-138 kubenswrapper[2566]: E1102 14:55:34.695653    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0d6c5872-e278-4463-953d-e1388a001d0d-kube-api-access-hpfk5 podName:0d6c5872-e278-4463-953d-e1388a001d0d nodeName:}" failed. No retries permitted until 2026-11-02 14:55:38.695635476 +0000 UTC m=+10.214658762 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-hpfk5" (UniqueName: "kubernetes.io/projected/0d6c5872-e278-4463-953d-e1388a001d0d-kube-api-access-hpfk5") pod "network-check-target-wns8r" (UID: "0d6c5872-e278-4463-953d-e1388a001d0d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 14:55:34.970338 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:34.970230    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-484ld"
Nov 02 14:55:34.970499 ip-10-0-135-138 kubenswrapper[2566]: E1102 14:55:34.970436    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-484ld" podUID="caa8898b-6611-49b8-845d-3b9bdac6691f"
Nov 02 14:55:34.970877 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:34.970856    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wns8r"
Nov 02 14:55:34.970980 ip-10-0-135-138 kubenswrapper[2566]: E1102 14:55:34.970962    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wns8r" podUID="0d6c5872-e278-4463-953d-e1388a001d0d"
Nov 02 14:55:34.971047 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:34.971029    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vl6zb"
Nov 02 14:55:34.971168 ip-10-0-135-138 kubenswrapper[2566]: E1102 14:55:34.971107    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vl6zb" podUID="332f4a2d-a241-4c71-8ebb-a99618c0d3dc"
Nov 02 14:55:35.163868 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:35.163182    2566 generic.go:356] "Generic (PLEG): container finished" podID="b1715623-afd0-4d0e-9b63-b38499cec1e2" containerID="f62b7e1913bd0bd851d8ec494f42b43881661a821c8e5291037ff952aa42cef8" exitCode=0
Nov 02 14:55:35.163868 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:35.163322    2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fbnnl" event={"ID":"b1715623-afd0-4d0e-9b63-b38499cec1e2","Type":"ContainerDied","Data":"f62b7e1913bd0bd851d8ec494f42b43881661a821c8e5291037ff952aa42cef8"}
Nov 02 14:55:36.174211 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:36.173169    2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-d5t6w" event={"ID":"68b781b2-be96-4c5c-8daa-2238fd0a9623","Type":"ContainerStarted","Data":"412e74745bad3c6e2a4554eb5bb7979586be056ddfb30ec19431777f21dad9f6"}
Nov 02 14:55:36.971477 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:36.971397    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-484ld"
Nov 02 14:55:36.972170 ip-10-0-135-138 kubenswrapper[2566]: E1102 14:55:36.972131    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-484ld" podUID="caa8898b-6611-49b8-845d-3b9bdac6691f"
Nov 02 14:55:36.972329 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:36.972297    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wns8r"
Nov 02 14:55:36.972468 ip-10-0-135-138 kubenswrapper[2566]: E1102 14:55:36.972436    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wns8r" podUID="0d6c5872-e278-4463-953d-e1388a001d0d"
Nov 02 14:55:36.972558 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:36.972526    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vl6zb"
Nov 02 14:55:36.972687 ip-10-0-135-138 kubenswrapper[2566]: E1102 14:55:36.972669    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vl6zb" podUID="332f4a2d-a241-4c71-8ebb-a99618c0d3dc"
Nov 02 14:55:37.185399 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:37.185359    2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-q9nx7" event={"ID":"e4983b70-40f4-4e1e-a5da-80194d5d4786","Type":"ContainerStarted","Data":"04928ca293f2d0b4bba327a33411a75bb03f2e565dcb7ac28d70db03ee2eaacf"}
Nov 02 14:55:37.231832 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:37.231725    2566 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-q9nx7" podStartSLOduration=2.911550647 podStartE2EDuration="8.231702859s" totalImagesPullingTime="5.320152212s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 14:55:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:55:37.231168824 +0000 UTC m=+8.750192129" watchObservedRunningTime="2026-11-02 14:55:37.231702859 +0000 UTC m=+8.750726169"
Nov 02 14:55:37.232095 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:37.232058    2566 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-d5t6w" podStartSLOduration=4.334913602 podStartE2EDuration="8.232042193s" totalImagesPullingTime="3.897128591s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:55:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:55:36.219493932 +0000 UTC m=+7.738517241" watchObservedRunningTime="2026-11-02 14:55:37.232042193 +0000 UTC m=+8.751065502"
Nov 02 14:55:37.844650 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:37.844557    2566 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-d5t6w"
Nov 02 14:55:37.845323 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:37.845301    2566 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-d5t6w"
Nov 02 14:55:38.190921 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:38.190825    2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6cj82" event={"ID":"54dc6898-f041-4ed6-9fc9-1b5159d00d27","Type":"ContainerStarted","Data":"b8122c3fbd63c4472e50bee5c2ef9148a58b11f87fd59cf6916438f069607d78"}
Nov 02 14:55:38.632683 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:38.632575    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/332f4a2d-a241-4c71-8ebb-a99618c0d3dc-original-pull-secret\") pod \"global-pull-secret-syncer-vl6zb\" (UID: \"332f4a2d-a241-4c71-8ebb-a99618c0d3dc\") " pod="kube-system/global-pull-secret-syncer-vl6zb"
Nov 02 14:55:38.632854 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:38.632705    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/caa8898b-6611-49b8-845d-3b9bdac6691f-metrics-certs\") pod \"network-metrics-daemon-484ld\" (UID: \"caa8898b-6611-49b8-845d-3b9bdac6691f\") " pod="openshift-multus/network-metrics-daemon-484ld"
Nov 02 14:55:38.632947 ip-10-0-135-138 kubenswrapper[2566]: E1102 14:55:38.632866    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 14:55:38.632947 ip-10-0-135-138 kubenswrapper[2566]: E1102 14:55:38.632940    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/caa8898b-6611-49b8-845d-3b9bdac6691f-metrics-certs podName:caa8898b-6611-49b8-845d-3b9bdac6691f nodeName:}" failed. No retries permitted until 2026-11-02 14:55:46.632919501 +0000 UTC m=+18.151942809 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/caa8898b-6611-49b8-845d-3b9bdac6691f-metrics-certs") pod "network-metrics-daemon-484ld" (UID: "caa8898b-6611-49b8-845d-3b9bdac6691f") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 14:55:38.633426 ip-10-0-135-138 kubenswrapper[2566]: E1102 14:55:38.633395    2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 02 14:55:38.633529 ip-10-0-135-138 kubenswrapper[2566]: E1102 14:55:38.633472    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/332f4a2d-a241-4c71-8ebb-a99618c0d3dc-original-pull-secret podName:332f4a2d-a241-4c71-8ebb-a99618c0d3dc nodeName:}" failed. No retries permitted until 2026-11-02 14:55:46.633447013 +0000 UTC m=+18.152470312 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/332f4a2d-a241-4c71-8ebb-a99618c0d3dc-original-pull-secret") pod "global-pull-secret-syncer-vl6zb" (UID: "332f4a2d-a241-4c71-8ebb-a99618c0d3dc") : object "kube-system"/"original-pull-secret" not registered
Nov 02 14:55:38.734234 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:38.734192    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hpfk5\" (UniqueName: \"kubernetes.io/projected/0d6c5872-e278-4463-953d-e1388a001d0d-kube-api-access-hpfk5\") pod \"network-check-target-wns8r\" (UID: \"0d6c5872-e278-4463-953d-e1388a001d0d\") " pod="openshift-network-diagnostics/network-check-target-wns8r"
Nov 02 14:55:38.734421 ip-10-0-135-138 kubenswrapper[2566]: E1102 14:55:38.734402    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 02 14:55:38.734494 ip-10-0-135-138 kubenswrapper[2566]: E1102 14:55:38.734434    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 02 14:55:38.734494 ip-10-0-135-138 kubenswrapper[2566]: E1102 14:55:38.734449    2566 projected.go:196] Error preparing data for projected volume kube-api-access-hpfk5 for pod openshift-network-diagnostics/network-check-target-wns8r: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 14:55:38.734566 ip-10-0-135-138 kubenswrapper[2566]: E1102 14:55:38.734509    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0d6c5872-e278-4463-953d-e1388a001d0d-kube-api-access-hpfk5 podName:0d6c5872-e278-4463-953d-e1388a001d0d nodeName:}" failed. No retries permitted until 2026-11-02 14:55:46.73449139 +0000 UTC m=+18.253514675 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-hpfk5" (UniqueName: "kubernetes.io/projected/0d6c5872-e278-4463-953d-e1388a001d0d-kube-api-access-hpfk5") pod "network-check-target-wns8r" (UID: "0d6c5872-e278-4463-953d-e1388a001d0d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 14:55:38.970971 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:38.970890    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-484ld"
Nov 02 14:55:38.971195 ip-10-0-135-138 kubenswrapper[2566]: E1102 14:55:38.971049    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-484ld" podUID="caa8898b-6611-49b8-845d-3b9bdac6691f"
Nov 02 14:55:38.971195 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:38.971120    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wns8r"
Nov 02 14:55:38.971307 ip-10-0-135-138 kubenswrapper[2566]: E1102 14:55:38.971218    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wns8r" podUID="0d6c5872-e278-4463-953d-e1388a001d0d"
Nov 02 14:55:38.971307 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:38.971268    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vl6zb"
Nov 02 14:55:38.971411 ip-10-0-135-138 kubenswrapper[2566]: E1102 14:55:38.971341    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vl6zb" podUID="332f4a2d-a241-4c71-8ebb-a99618c0d3dc"
Nov 02 14:55:39.171431 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:39.171164    2566 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Nov 02 14:55:39.198322 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:39.197877    2566 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 02 14:55:39.198322 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:39.197889    2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6cj82" event={"ID":"54dc6898-f041-4ed6-9fc9-1b5159d00d27","Type":"ContainerStarted","Data":"62dc3a1f18fcac92a9c416d9f16b1e42b95f30a3250a0e1f4cfdc19f8a73cd9d"}
Nov 02 14:55:39.749334 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:39.749250    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d5t6w_68b781b2-be96-4c5c-8daa-2238fd0a9623/konnectivity-agent/0.log" limit=1024
Nov 02 14:55:40.052051 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:40.051888    2566 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-02T14:55:39.17118253Z","UUID":"4be7b35c-8013-4ad7-bc3a-a69ac2d96a21","Handler":null,"Name":"","Endpoint":""}
Nov 02 14:55:40.053853 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:40.053550    2566 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Nov 02 14:55:40.053853 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:40.053587    2566 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Nov 02 14:55:40.201746 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:40.201703    2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-jpz27" event={"ID":"720ef290-024b-4ed2-8417-92290f360dc9","Type":"ContainerStarted","Data":"2f216aafd725f96867bcde39718b77d086935e6887a337a9761461db793f5eda"}
Nov 02 14:55:40.203985 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:40.203954    2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-7slzh" event={"ID":"93d7d50e-b320-4217-b183-2bfc921f6c9d","Type":"ContainerStarted","Data":"2ca426bdae249db1f1b1d13744daed53d80dbcf83748dcae5776da4e9d99aa73"}
Nov 02 14:55:40.206266 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:40.206171    2566 generic.go:356] "Generic (PLEG): container finished" podID="b1715623-afd0-4d0e-9b63-b38499cec1e2" containerID="a1be58fa68a97dcb5a116a93e4b895bad3e7a128ad32668f5fd10fe450279f62" exitCode=0
Nov 02 14:55:40.206266 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:40.206221    2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fbnnl" event={"ID":"b1715623-afd0-4d0e-9b63-b38499cec1e2","Type":"ContainerDied","Data":"a1be58fa68a97dcb5a116a93e4b895bad3e7a128ad32668f5fd10fe450279f62"}
Nov 02 14:55:40.245237 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:40.245174    2566 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-jpz27" podStartSLOduration=1.717882075 podStartE2EDuration="10.245159041s" totalImagesPullingTime="8.527276966s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 14:55:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:55:40.244555823 +0000 UTC m=+11.763579126" watchObservedRunningTime="2026-11-02 14:55:40.245159041 +0000 UTC m=+11.764182344"
Nov 02 14:55:40.337064 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:40.336975    2566 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-7slzh" podStartSLOduration=2.1483019309999998 podStartE2EDuration="10.336957266s" totalImagesPullingTime="8.188655335s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 14:55:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:55:40.33664971 +0000 UTC m=+11.855673011" watchObservedRunningTime="2026-11-02 14:55:40.336957266 +0000 UTC m=+11.855980566"
Nov 02 14:55:40.967610 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:40.967577    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wns8r"
Nov 02 14:55:40.967771 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:40.967644    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vl6zb"
Nov 02 14:55:40.967771 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:40.967667    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-484ld"
Nov 02 14:55:40.967771 ip-10-0-135-138 kubenswrapper[2566]: E1102 14:55:40.967742    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vl6zb" podUID="332f4a2d-a241-4c71-8ebb-a99618c0d3dc"
Nov 02 14:55:40.967936 ip-10-0-135-138 kubenswrapper[2566]: E1102 14:55:40.967818    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-484ld" podUID="caa8898b-6611-49b8-845d-3b9bdac6691f"
Nov 02 14:55:40.967936 ip-10-0-135-138 kubenswrapper[2566]: E1102 14:55:40.967876    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wns8r" podUID="0d6c5872-e278-4463-953d-e1388a001d0d"
Nov 02 14:55:41.211337 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:41.211083    2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6cj82" event={"ID":"54dc6898-f041-4ed6-9fc9-1b5159d00d27","Type":"ContainerStarted","Data":"d85f18351aa37acb86f1c6422d62b794ce4ebbbd56cf3a36570b36863cf5b0f2"}
Nov 02 14:55:41.213442 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:41.213410    2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-pl8gn" event={"ID":"7fe9ba5e-a867-4ac3-8a6f-a316c6f64db4","Type":"ContainerStarted","Data":"7199882aa6d82bf116bc08e7c5d8f7779e48cd6e2c74092b44295d4210c0f8c7"}
Nov 02 14:55:41.256988 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:41.256902    2566 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6cj82" podStartSLOduration=3.612680926 podStartE2EDuration="12.256888424s" totalImagesPullingTime="8.644207498s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 14:55:29 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:55:41.256649839 +0000 UTC m=+12.775673142" watchObservedRunningTime="2026-11-02 14:55:41.256888424 +0000 UTC m=+12.775911724"
Nov 02 14:55:41.303165 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:41.303091    2566 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-pl8gn" podStartSLOduration=3.043076996 podStartE2EDuration="11.30307167s" totalImagesPullingTime="8.259994674s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 14:55:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:55:41.30247948 +0000 UTC m=+12.821502786" watchObservedRunningTime="2026-11-02 14:55:41.30307167 +0000 UTC m=+12.822094968"
Nov 02 14:55:42.222011 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:42.221973    2566 generic.go:356] "Generic (PLEG): container finished" podID="b1715623-afd0-4d0e-9b63-b38499cec1e2" containerID="43c4fdb0aaa0f180b3f65b969d1924ef563707eed1d6e01247a4c01a41dda65b" exitCode=0
Nov 02 14:55:42.222513 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:42.222060    2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fbnnl" event={"ID":"b1715623-afd0-4d0e-9b63-b38499cec1e2","Type":"ContainerDied","Data":"43c4fdb0aaa0f180b3f65b969d1924ef563707eed1d6e01247a4c01a41dda65b"}
Nov 02 14:55:42.224236 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:42.224213    2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-sq4l4" event={"ID":"e1da99c3-8294-4e34-9d33-47e04a6182d3","Type":"ContainerStarted","Data":"43cd8599224002cb4b81d710a61556dc4a045e5c1e5cd12736771a3499f71c02"}
Nov 02 14:55:42.315818 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:42.315733    2566 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-sq4l4" podStartSLOduration=2.547945762 podStartE2EDuration="13.31571953s" totalImagesPullingTime="10.767773768s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 14:55:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:55:42.315544292 +0000 UTC m=+13.834567592" watchObservedRunningTime="2026-11-02 14:55:42.31571953 +0000 UTC m=+13.834742829"
Nov 02 14:55:42.967166 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:42.967119    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-484ld"
Nov 02 14:55:42.967338 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:42.967119    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wns8r"
Nov 02 14:55:42.967338 ip-10-0-135-138 kubenswrapper[2566]: E1102 14:55:42.967280    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-484ld" podUID="caa8898b-6611-49b8-845d-3b9bdac6691f"
Nov 02 14:55:42.967338 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:42.967311    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vl6zb"
Nov 02 14:55:42.967484 ip-10-0-135-138 kubenswrapper[2566]: E1102 14:55:42.967420    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wns8r" podUID="0d6c5872-e278-4463-953d-e1388a001d0d"
Nov 02 14:55:42.967523 ip-10-0-135-138 kubenswrapper[2566]: E1102 14:55:42.967499    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vl6zb" podUID="332f4a2d-a241-4c71-8ebb-a99618c0d3dc"
Nov 02 14:55:44.230859 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:44.230816    2566 generic.go:356] "Generic (PLEG): container finished" podID="b1715623-afd0-4d0e-9b63-b38499cec1e2" containerID="d9d301f4d4552c2692a103e04634e1ff319be58f054f325e2e270644e0710c13" exitCode=0
Nov 02 14:55:44.231736 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:44.230880    2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fbnnl" event={"ID":"b1715623-afd0-4d0e-9b63-b38499cec1e2","Type":"ContainerDied","Data":"d9d301f4d4552c2692a103e04634e1ff319be58f054f325e2e270644e0710c13"}
Nov 02 14:55:44.967881 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:44.967840    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wns8r"
Nov 02 14:55:44.968044 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:44.967844    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-484ld"
Nov 02 14:55:44.968044 ip-10-0-135-138 kubenswrapper[2566]: E1102 14:55:44.967963    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wns8r" podUID="0d6c5872-e278-4463-953d-e1388a001d0d"
Nov 02 14:55:44.968044 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:44.967994    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vl6zb"
Nov 02 14:55:44.968231 ip-10-0-135-138 kubenswrapper[2566]: E1102 14:55:44.968093    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-484ld" podUID="caa8898b-6611-49b8-845d-3b9bdac6691f"
Nov 02 14:55:44.968231 ip-10-0-135-138 kubenswrapper[2566]: E1102 14:55:44.968173    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vl6zb" podUID="332f4a2d-a241-4c71-8ebb-a99618c0d3dc"
Nov 02 14:55:46.614226 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:46.614039    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-jpz27_720ef290-024b-4ed2-8417-92290f360dc9/dns-node-resolver/0.log"
Nov 02 14:55:46.701968 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:46.701933    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/caa8898b-6611-49b8-845d-3b9bdac6691f-metrics-certs\") pod \"network-metrics-daemon-484ld\" (UID: \"caa8898b-6611-49b8-845d-3b9bdac6691f\") " pod="openshift-multus/network-metrics-daemon-484ld"
Nov 02 14:55:46.702153 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:46.701981    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/332f4a2d-a241-4c71-8ebb-a99618c0d3dc-original-pull-secret\") pod \"global-pull-secret-syncer-vl6zb\" (UID: \"332f4a2d-a241-4c71-8ebb-a99618c0d3dc\") " pod="kube-system/global-pull-secret-syncer-vl6zb"
Nov 02 14:55:46.702153 ip-10-0-135-138 kubenswrapper[2566]: E1102 14:55:46.702079    2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 02 14:55:46.702153 ip-10-0-135-138 kubenswrapper[2566]: E1102 14:55:46.702084    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 14:55:46.702267 ip-10-0-135-138 kubenswrapper[2566]: E1102 14:55:46.702156    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/332f4a2d-a241-4c71-8ebb-a99618c0d3dc-original-pull-secret podName:332f4a2d-a241-4c71-8ebb-a99618c0d3dc nodeName:}" failed. No retries permitted until 2026-11-02 14:56:02.702139951 +0000 UTC m=+34.221163246 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/332f4a2d-a241-4c71-8ebb-a99618c0d3dc-original-pull-secret") pod "global-pull-secret-syncer-vl6zb" (UID: "332f4a2d-a241-4c71-8ebb-a99618c0d3dc") : object "kube-system"/"original-pull-secret" not registered
Nov 02 14:55:46.702267 ip-10-0-135-138 kubenswrapper[2566]: E1102 14:55:46.702172    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/caa8898b-6611-49b8-845d-3b9bdac6691f-metrics-certs podName:caa8898b-6611-49b8-845d-3b9bdac6691f nodeName:}" failed. No retries permitted until 2026-11-02 14:56:02.702165626 +0000 UTC m=+34.221188904 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/caa8898b-6611-49b8-845d-3b9bdac6691f-metrics-certs") pod "network-metrics-daemon-484ld" (UID: "caa8898b-6611-49b8-845d-3b9bdac6691f") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 14:55:46.802433 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:46.802402    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hpfk5\" (UniqueName: \"kubernetes.io/projected/0d6c5872-e278-4463-953d-e1388a001d0d-kube-api-access-hpfk5\") pod \"network-check-target-wns8r\" (UID: \"0d6c5872-e278-4463-953d-e1388a001d0d\") " pod="openshift-network-diagnostics/network-check-target-wns8r"
Nov 02 14:55:46.802572 ip-10-0-135-138 kubenswrapper[2566]: E1102 14:55:46.802554    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 02 14:55:46.802606 ip-10-0-135-138 kubenswrapper[2566]: E1102 14:55:46.802580    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 02 14:55:46.802606 ip-10-0-135-138 kubenswrapper[2566]: E1102 14:55:46.802590    2566 projected.go:196] Error preparing data for projected volume kube-api-access-hpfk5 for pod openshift-network-diagnostics/network-check-target-wns8r: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 14:55:46.802669 ip-10-0-135-138 kubenswrapper[2566]: E1102 14:55:46.802635    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0d6c5872-e278-4463-953d-e1388a001d0d-kube-api-access-hpfk5 podName:0d6c5872-e278-4463-953d-e1388a001d0d nodeName:}" failed. No retries permitted until 2026-11-02 14:56:02.80262348 +0000 UTC m=+34.321646777 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-hpfk5" (UniqueName: "kubernetes.io/projected/0d6c5872-e278-4463-953d-e1388a001d0d-kube-api-access-hpfk5") pod "network-check-target-wns8r" (UID: "0d6c5872-e278-4463-953d-e1388a001d0d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 14:55:46.966713 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:46.966691    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wns8r"
Nov 02 14:55:46.966812 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:46.966698    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-484ld"
Nov 02 14:55:46.966861 ip-10-0-135-138 kubenswrapper[2566]: E1102 14:55:46.966812    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wns8r" podUID="0d6c5872-e278-4463-953d-e1388a001d0d"
Nov 02 14:55:46.966861 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:46.966701    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vl6zb"
Nov 02 14:55:46.966929 ip-10-0-135-138 kubenswrapper[2566]: E1102 14:55:46.966861    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-484ld" podUID="caa8898b-6611-49b8-845d-3b9bdac6691f"
Nov 02 14:55:46.966977 ip-10-0-135-138 kubenswrapper[2566]: E1102 14:55:46.966935    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vl6zb" podUID="332f4a2d-a241-4c71-8ebb-a99618c0d3dc"
Nov 02 14:55:47.240291 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:47.240209    2566 generic.go:356] "Generic (PLEG): container finished" podID="b1715623-afd0-4d0e-9b63-b38499cec1e2" containerID="24146e9d6d5a628b32ab6acd6a6ab9c79d72dfd83262baf717e2014333fbae11" exitCode=0
Nov 02 14:55:47.240446 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:47.240298    2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fbnnl" event={"ID":"b1715623-afd0-4d0e-9b63-b38499cec1e2","Type":"ContainerDied","Data":"24146e9d6d5a628b32ab6acd6a6ab9c79d72dfd83262baf717e2014333fbae11"}
Nov 02 14:55:47.243408 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:47.243379    2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-f8btv" event={"ID":"65ce39fc-05c7-433b-81f4-c1b219f1bd71","Type":"ContainerStarted","Data":"c44fd54066399ba28237c26cd26352cac33209633f08c79d419c895a9d2e34f3"}
Nov 02 14:55:47.243484 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:47.243441    2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-f8btv" event={"ID":"65ce39fc-05c7-433b-81f4-c1b219f1bd71","Type":"ContainerStarted","Data":"060649697ffb21263f5d80d7a9921cb1cacf6a23a92b4cb71cbd6492eeefc330"}
Nov 02 14:55:47.243484 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:47.243472    2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-f8btv" event={"ID":"65ce39fc-05c7-433b-81f4-c1b219f1bd71","Type":"ContainerStarted","Data":"6a7cbf505dd262df913b6f5bec820811517e1a3b2fb9608c78b11a859b2eeeec"}
Nov 02 14:55:47.243545 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:47.243498    2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-f8btv" event={"ID":"65ce39fc-05c7-433b-81f4-c1b219f1bd71","Type":"ContainerStarted","Data":"14e4a6d1cda51886e3952375a319b9187bc8203387797e279b35dfab53d8c5bd"}
Nov 02 14:55:47.243545 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:47.243526    2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-f8btv" event={"ID":"65ce39fc-05c7-433b-81f4-c1b219f1bd71","Type":"ContainerStarted","Data":"fa60168b4fc8ceecc0291b64362e615dc7e2c257f1623dd63a23ae97e43bbdf6"}
Nov 02 14:55:47.243597 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:47.243550    2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-f8btv" event={"ID":"65ce39fc-05c7-433b-81f4-c1b219f1bd71","Type":"ContainerStarted","Data":"db5f5144eb9d39c324dd5690aceff53904f1ae3c9b4b49eb14ff5f7b094e49c0"}
Nov 02 14:55:47.611492 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:47.611421    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-q9nx7_e4983b70-40f4-4e1e-a5da-80194d5d4786/node-ca/0.log"
Nov 02 14:55:48.028668 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:48.028590    2566 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-d5t6w"
Nov 02 14:55:48.029035 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:48.028735    2566 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 02 14:55:48.029121 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:48.029101    2566 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-d5t6w"
Nov 02 14:55:48.248758 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:48.248723    2566 generic.go:356] "Generic (PLEG): container finished" podID="b1715623-afd0-4d0e-9b63-b38499cec1e2" containerID="5d28dbe3f4717bb50c9f5a7d9e249c9e881f651f7ce67795bcfe4729c7246fb7" exitCode=0
Nov 02 14:55:48.248911 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:48.248799    2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fbnnl" event={"ID":"b1715623-afd0-4d0e-9b63-b38499cec1e2","Type":"ContainerDied","Data":"5d28dbe3f4717bb50c9f5a7d9e249c9e881f651f7ce67795bcfe4729c7246fb7"}
Nov 02 14:55:48.967798 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:48.967760    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-484ld"
Nov 02 14:55:48.967944 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:48.967867    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vl6zb"
Nov 02 14:55:48.967944 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:48.967896    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wns8r"
Nov 02 14:55:48.968003 ip-10-0-135-138 kubenswrapper[2566]: E1102 14:55:48.967967    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vl6zb" podUID="332f4a2d-a241-4c71-8ebb-a99618c0d3dc"
Nov 02 14:55:48.968003 ip-10-0-135-138 kubenswrapper[2566]: E1102 14:55:48.967869    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-484ld" podUID="caa8898b-6611-49b8-845d-3b9bdac6691f"
Nov 02 14:55:48.968069 ip-10-0-135-138 kubenswrapper[2566]: E1102 14:55:48.968039    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wns8r" podUID="0d6c5872-e278-4463-953d-e1388a001d0d"
Nov 02 14:55:49.254672 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:49.254580    2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fbnnl" event={"ID":"b1715623-afd0-4d0e-9b63-b38499cec1e2","Type":"ContainerStarted","Data":"a6784e46e7c0f29946f7cf71f7ac8147c84babd6433cdcca595aceb30879d7fb"}
Nov 02 14:55:49.257734 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:49.257711    2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-f8btv" event={"ID":"65ce39fc-05c7-433b-81f4-c1b219f1bd71","Type":"ContainerStarted","Data":"464fa69622709aee2803f30515986ba2b4e8823a32096f459ce3c32453c9f28f"}
Nov 02 14:55:50.966961 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:50.966926    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vl6zb"
Nov 02 14:55:50.966961 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:50.966965    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-484ld"
Nov 02 14:55:50.967627 ip-10-0-135-138 kubenswrapper[2566]: E1102 14:55:50.967053    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vl6zb" podUID="332f4a2d-a241-4c71-8ebb-a99618c0d3dc"
Nov 02 14:55:50.967627 ip-10-0-135-138 kubenswrapper[2566]: E1102 14:55:50.967086    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-484ld" podUID="caa8898b-6611-49b8-845d-3b9bdac6691f"
Nov 02 14:55:50.967627 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:50.967170    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wns8r"
Nov 02 14:55:50.967627 ip-10-0-135-138 kubenswrapper[2566]: E1102 14:55:50.967283    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wns8r" podUID="0d6c5872-e278-4463-953d-e1388a001d0d"
Nov 02 14:55:51.267178 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:51.267068    2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-f8btv" event={"ID":"65ce39fc-05c7-433b-81f4-c1b219f1bd71","Type":"ContainerStarted","Data":"7c55c8bdf055faeb30a457f8fa5a33ca644fb513e23a6ef683b1952847f6a9bc"}
Nov 02 14:55:51.267394 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:51.267373    2566 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-f8btv"
Nov 02 14:55:51.267497 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:51.267444    2566 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-f8btv"
Nov 02 14:55:51.282874 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:51.282848    2566 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-f8btv"
Nov 02 14:55:51.362539 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:51.362480    2566 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-fbnnl" podStartSLOduration=9.806057932 podStartE2EDuration="21.362463101s" totalImagesPullingTime="11.556405169s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 14:55:30 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:55:49.3168586 +0000 UTC m=+20.835881899" watchObservedRunningTime="2026-11-02 14:55:51.362463101 +0000 UTC m=+22.881486398"
Nov 02 14:55:51.362906 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:51.362876    2566 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-f8btv" podStartSLOduration=5.907216084 podStartE2EDuration="21.362868015s" totalImagesPullingTime="15.455651931s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 14:55:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:55:51.348990854 +0000 UTC m=+22.868014164" watchObservedRunningTime="2026-11-02 14:55:51.362868015 +0000 UTC m=+22.881891319"
Nov 02 14:55:52.269965 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:52.269940    2566 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-f8btv"
Nov 02 14:55:52.285382 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:52.285220    2566 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-f8btv"
Nov 02 14:55:52.966815 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:52.966780    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-484ld"
Nov 02 14:55:52.967154 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:52.966784    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vl6zb"
Nov 02 14:55:52.967154 ip-10-0-135-138 kubenswrapper[2566]: E1102 14:55:52.966891    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-484ld" podUID="caa8898b-6611-49b8-845d-3b9bdac6691f"
Nov 02 14:55:52.967154 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:52.966901    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wns8r"
Nov 02 14:55:52.967154 ip-10-0-135-138 kubenswrapper[2566]: E1102 14:55:52.966991    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vl6zb" podUID="332f4a2d-a241-4c71-8ebb-a99618c0d3dc"
Nov 02 14:55:52.967154 ip-10-0-135-138 kubenswrapper[2566]: E1102 14:55:52.967056    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wns8r" podUID="0d6c5872-e278-4463-953d-e1388a001d0d"
Nov 02 14:55:54.966752 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:54.966718    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vl6zb"
Nov 02 14:55:54.967158 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:54.966757    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wns8r"
Nov 02 14:55:54.967158 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:54.966725    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-484ld"
Nov 02 14:55:54.967158 ip-10-0-135-138 kubenswrapper[2566]: E1102 14:55:54.966842    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vl6zb" podUID="332f4a2d-a241-4c71-8ebb-a99618c0d3dc"
Nov 02 14:55:54.967158 ip-10-0-135-138 kubenswrapper[2566]: E1102 14:55:54.966902    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-484ld" podUID="caa8898b-6611-49b8-845d-3b9bdac6691f"
Nov 02 14:55:54.967158 ip-10-0-135-138 kubenswrapper[2566]: E1102 14:55:54.966965    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wns8r" podUID="0d6c5872-e278-4463-953d-e1388a001d0d"
Nov 02 14:55:56.967170 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:56.967115    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-484ld"
Nov 02 14:55:56.967650 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:56.967149    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wns8r"
Nov 02 14:55:56.967650 ip-10-0-135-138 kubenswrapper[2566]: E1102 14:55:56.967268    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-484ld" podUID="caa8898b-6611-49b8-845d-3b9bdac6691f"
Nov 02 14:55:56.967650 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:56.967312    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vl6zb"
Nov 02 14:55:56.967650 ip-10-0-135-138 kubenswrapper[2566]: E1102 14:55:56.967402    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wns8r" podUID="0d6c5872-e278-4463-953d-e1388a001d0d"
Nov 02 14:55:56.967650 ip-10-0-135-138 kubenswrapper[2566]: E1102 14:55:56.967496    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vl6zb" podUID="332f4a2d-a241-4c71-8ebb-a99618c0d3dc"
Nov 02 14:55:58.968138 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:58.968090    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vl6zb"
Nov 02 14:55:58.968522 ip-10-0-135-138 kubenswrapper[2566]: E1102 14:55:58.968204    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vl6zb" podUID="332f4a2d-a241-4c71-8ebb-a99618c0d3dc"
Nov 02 14:55:58.968522 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:58.968302    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-484ld"
Nov 02 14:55:58.968522 ip-10-0-135-138 kubenswrapper[2566]: E1102 14:55:58.968425    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-484ld" podUID="caa8898b-6611-49b8-845d-3b9bdac6691f"
Nov 02 14:55:58.968522 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:55:58.968494    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wns8r"
Nov 02 14:55:58.968651 ip-10-0-135-138 kubenswrapper[2566]: E1102 14:55:58.968572    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wns8r" podUID="0d6c5872-e278-4463-953d-e1388a001d0d"
Nov 02 14:56:00.967017 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:56:00.966979    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vl6zb"
Nov 02 14:56:00.967017 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:56:00.967017    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wns8r"
Nov 02 14:56:00.967017 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:56:00.967003    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-484ld"
Nov 02 14:56:00.967626 ip-10-0-135-138 kubenswrapper[2566]: E1102 14:56:00.967102    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vl6zb" podUID="332f4a2d-a241-4c71-8ebb-a99618c0d3dc"
Nov 02 14:56:00.967626 ip-10-0-135-138 kubenswrapper[2566]: E1102 14:56:00.967203    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wns8r" podUID="0d6c5872-e278-4463-953d-e1388a001d0d"
Nov 02 14:56:00.967626 ip-10-0-135-138 kubenswrapper[2566]: E1102 14:56:00.967271    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-484ld" podUID="caa8898b-6611-49b8-845d-3b9bdac6691f"
Nov 02 14:56:02.716140 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:56:02.716103    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/caa8898b-6611-49b8-845d-3b9bdac6691f-metrics-certs\") pod \"network-metrics-daemon-484ld\" (UID: \"caa8898b-6611-49b8-845d-3b9bdac6691f\") " pod="openshift-multus/network-metrics-daemon-484ld"
Nov 02 14:56:02.716539 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:56:02.716166    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/332f4a2d-a241-4c71-8ebb-a99618c0d3dc-original-pull-secret\") pod \"global-pull-secret-syncer-vl6zb\" (UID: \"332f4a2d-a241-4c71-8ebb-a99618c0d3dc\") " pod="kube-system/global-pull-secret-syncer-vl6zb"
Nov 02 14:56:02.716539 ip-10-0-135-138 kubenswrapper[2566]: E1102 14:56:02.716272    2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 02 14:56:02.716539 ip-10-0-135-138 kubenswrapper[2566]: E1102 14:56:02.716274    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 14:56:02.716539 ip-10-0-135-138 kubenswrapper[2566]: E1102 14:56:02.716326    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/332f4a2d-a241-4c71-8ebb-a99618c0d3dc-original-pull-secret podName:332f4a2d-a241-4c71-8ebb-a99618c0d3dc nodeName:}" failed. No retries permitted until 2026-11-02 14:56:34.716314273 +0000 UTC m=+66.235337550 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/332f4a2d-a241-4c71-8ebb-a99618c0d3dc-original-pull-secret") pod "global-pull-secret-syncer-vl6zb" (UID: "332f4a2d-a241-4c71-8ebb-a99618c0d3dc") : object "kube-system"/"original-pull-secret" not registered
Nov 02 14:56:02.716539 ip-10-0-135-138 kubenswrapper[2566]: E1102 14:56:02.716340    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/caa8898b-6611-49b8-845d-3b9bdac6691f-metrics-certs podName:caa8898b-6611-49b8-845d-3b9bdac6691f nodeName:}" failed. No retries permitted until 2026-11-02 14:56:34.716333608 +0000 UTC m=+66.235356887 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/caa8898b-6611-49b8-845d-3b9bdac6691f-metrics-certs") pod "network-metrics-daemon-484ld" (UID: "caa8898b-6611-49b8-845d-3b9bdac6691f") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 14:56:02.817280 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:56:02.817241    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hpfk5\" (UniqueName: \"kubernetes.io/projected/0d6c5872-e278-4463-953d-e1388a001d0d-kube-api-access-hpfk5\") pod \"network-check-target-wns8r\" (UID: \"0d6c5872-e278-4463-953d-e1388a001d0d\") " pod="openshift-network-diagnostics/network-check-target-wns8r"
Nov 02 14:56:02.817447 ip-10-0-135-138 kubenswrapper[2566]: E1102 14:56:02.817372    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 02 14:56:02.817447 ip-10-0-135-138 kubenswrapper[2566]: E1102 14:56:02.817397    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 02 14:56:02.817447 ip-10-0-135-138 kubenswrapper[2566]: E1102 14:56:02.817406    2566 projected.go:196] Error preparing data for projected volume kube-api-access-hpfk5 for pod openshift-network-diagnostics/network-check-target-wns8r: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 14:56:02.817540 ip-10-0-135-138 kubenswrapper[2566]: E1102 14:56:02.817454    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0d6c5872-e278-4463-953d-e1388a001d0d-kube-api-access-hpfk5 podName:0d6c5872-e278-4463-953d-e1388a001d0d nodeName:}" failed. No retries permitted until 2026-11-02 14:56:34.817441234 +0000 UTC m=+66.336464532 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-hpfk5" (UniqueName: "kubernetes.io/projected/0d6c5872-e278-4463-953d-e1388a001d0d-kube-api-access-hpfk5") pod "network-check-target-wns8r" (UID: "0d6c5872-e278-4463-953d-e1388a001d0d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 14:56:02.966792 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:56:02.966719    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wns8r"
Nov 02 14:56:02.966937 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:56:02.966719    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-484ld"
Nov 02 14:56:02.966937 ip-10-0-135-138 kubenswrapper[2566]: E1102 14:56:02.966825    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wns8r" podUID="0d6c5872-e278-4463-953d-e1388a001d0d"
Nov 02 14:56:02.966937 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:56:02.966725    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vl6zb"
Nov 02 14:56:02.967040 ip-10-0-135-138 kubenswrapper[2566]: E1102 14:56:02.966907    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-484ld" podUID="caa8898b-6611-49b8-845d-3b9bdac6691f"
Nov 02 14:56:02.967040 ip-10-0-135-138 kubenswrapper[2566]: E1102 14:56:02.966989    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vl6zb" podUID="332f4a2d-a241-4c71-8ebb-a99618c0d3dc"
Nov 02 14:56:04.967099 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:56:04.967055    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wns8r"
Nov 02 14:56:04.967099 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:56:04.967079    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-484ld"
Nov 02 14:56:04.967647 ip-10-0-135-138 kubenswrapper[2566]: E1102 14:56:04.967181    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wns8r" podUID="0d6c5872-e278-4463-953d-e1388a001d0d"
Nov 02 14:56:04.967647 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:56:04.967257    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vl6zb"
Nov 02 14:56:04.967647 ip-10-0-135-138 kubenswrapper[2566]: E1102 14:56:04.967362    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vl6zb" podUID="332f4a2d-a241-4c71-8ebb-a99618c0d3dc"
Nov 02 14:56:04.967647 ip-10-0-135-138 kubenswrapper[2566]: E1102 14:56:04.967465    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-484ld" podUID="caa8898b-6611-49b8-845d-3b9bdac6691f"
Nov 02 14:56:06.967066 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:56:06.967035    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-484ld"
Nov 02 14:56:06.967544 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:56:06.967035    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wns8r"
Nov 02 14:56:06.967544 ip-10-0-135-138 kubenswrapper[2566]: E1102 14:56:06.967159    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-484ld" podUID="caa8898b-6611-49b8-845d-3b9bdac6691f"
Nov 02 14:56:06.967544 ip-10-0-135-138 kubenswrapper[2566]: E1102 14:56:06.967230    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wns8r" podUID="0d6c5872-e278-4463-953d-e1388a001d0d"
Nov 02 14:56:06.967544 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:56:06.967042    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vl6zb"
Nov 02 14:56:06.967544 ip-10-0-135-138 kubenswrapper[2566]: E1102 14:56:06.967291    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vl6zb" podUID="332f4a2d-a241-4c71-8ebb-a99618c0d3dc"
Nov 02 14:56:08.331069 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:56:08.331036    2566 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-wns8r"]
Nov 02 14:56:08.331507 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:56:08.331237    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wns8r"
Nov 02 14:56:08.331507 ip-10-0-135-138 kubenswrapper[2566]: E1102 14:56:08.331352    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wns8r" podUID="0d6c5872-e278-4463-953d-e1388a001d0d"
Nov 02 14:56:08.334912 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:56:08.334885    2566 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-vl6zb"]
Nov 02 14:56:08.335030 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:56:08.334942    2566 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-484ld"]
Nov 02 14:56:08.335093 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:56:08.335030    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-484ld"
Nov 02 14:56:08.335183 ip-10-0-135-138 kubenswrapper[2566]: E1102 14:56:08.335160    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-484ld" podUID="caa8898b-6611-49b8-845d-3b9bdac6691f"
Nov 02 14:56:08.335351 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:56:08.335336    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vl6zb"
Nov 02 14:56:08.335447 ip-10-0-135-138 kubenswrapper[2566]: E1102 14:56:08.335430    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vl6zb" podUID="332f4a2d-a241-4c71-8ebb-a99618c0d3dc"
Nov 02 14:56:09.451684 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:56:09.451657    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-jpz27_720ef290-024b-4ed2-8417-92290f360dc9/dns-node-resolver/0.log"
Nov 02 14:56:09.967288 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:56:09.967255    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vl6zb"
Nov 02 14:56:09.967485 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:56:09.967294    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-484ld"
Nov 02 14:56:09.967485 ip-10-0-135-138 kubenswrapper[2566]: E1102 14:56:09.967369    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vl6zb" podUID="332f4a2d-a241-4c71-8ebb-a99618c0d3dc"
Nov 02 14:56:09.967485 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:56:09.967422    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wns8r"
Nov 02 14:56:09.967649 ip-10-0-135-138 kubenswrapper[2566]: E1102 14:56:09.967499    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wns8r" podUID="0d6c5872-e278-4463-953d-e1388a001d0d"
Nov 02 14:56:09.967649 ip-10-0-135-138 kubenswrapper[2566]: E1102 14:56:09.967545    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-484ld" podUID="caa8898b-6611-49b8-845d-3b9bdac6691f"
Nov 02 14:56:10.646851 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:56:10.646697    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-q9nx7_e4983b70-40f4-4e1e-a5da-80194d5d4786/node-ca/0.log"
Nov 02 14:56:11.967279 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:56:11.967249    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wns8r"
Nov 02 14:56:11.967660 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:56:11.967285    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vl6zb"
Nov 02 14:56:11.967660 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:56:11.967351    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-484ld"
Nov 02 14:56:11.967660 ip-10-0-135-138 kubenswrapper[2566]: E1102 14:56:11.967366    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wns8r" podUID="0d6c5872-e278-4463-953d-e1388a001d0d"
Nov 02 14:56:11.967660 ip-10-0-135-138 kubenswrapper[2566]: E1102 14:56:11.967435    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-484ld" podUID="caa8898b-6611-49b8-845d-3b9bdac6691f"
Nov 02 14:56:11.967660 ip-10-0-135-138 kubenswrapper[2566]: E1102 14:56:11.967500    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vl6zb" podUID="332f4a2d-a241-4c71-8ebb-a99618c0d3dc"
Nov 02 14:56:13.966633 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:56:13.966597    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vl6zb"
Nov 02 14:56:13.967170 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:56:13.966729    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-484ld"
Nov 02 14:56:13.967170 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:56:13.966745    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wns8r"
Nov 02 14:56:13.967170 ip-10-0-135-138 kubenswrapper[2566]: E1102 14:56:13.966751    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vl6zb" podUID="332f4a2d-a241-4c71-8ebb-a99618c0d3dc"
Nov 02 14:56:13.967170 ip-10-0-135-138 kubenswrapper[2566]: E1102 14:56:13.966811    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-484ld" podUID="caa8898b-6611-49b8-845d-3b9bdac6691f"
Nov 02 14:56:13.967170 ip-10-0-135-138 kubenswrapper[2566]: E1102 14:56:13.966861    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wns8r" podUID="0d6c5872-e278-4463-953d-e1388a001d0d"
Nov 02 14:56:15.734249 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:56:15.734220    2566 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-gpjvx"]
Nov 02 14:56:15.738054 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:56:15.738037    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-gpjvx"
Nov 02 14:56:15.754478 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:56:15.754453    2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Nov 02 14:56:15.754705 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:56:15.754688    2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Nov 02 14:56:15.768064 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:56:15.768046    2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Nov 02 14:56:15.768064 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:56:15.768048    2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Nov 02 14:56:15.768230 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:56:15.768069    2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Nov 02 14:56:15.768230 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:56:15.768164    2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Nov 02 14:56:15.768323 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:56:15.768276    2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-nts94\""
Nov 02 14:56:15.798707 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:56:15.798679    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/03b3012d-0bdf-4a78-ad31-c03d1df51ec0-sys\") pod \"node-exporter-gpjvx\" (UID: \"03b3012d-0bdf-4a78-ad31-c03d1df51ec0\") " pod="openshift-monitoring/node-exporter-gpjvx"
Nov 02 14:56:15.798707 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:56:15.798707    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9vxjj\" (UniqueName: \"kubernetes.io/projected/03b3012d-0bdf-4a78-ad31-c03d1df51ec0-kube-api-access-9vxjj\") pod \"node-exporter-gpjvx\" (UID: \"03b3012d-0bdf-4a78-ad31-c03d1df51ec0\") " pod="openshift-monitoring/node-exporter-gpjvx"
Nov 02 14:56:15.798893 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:56:15.798727    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/03b3012d-0bdf-4a78-ad31-c03d1df51ec0-node-exporter-accelerators-collector-config\") pod \"node-exporter-gpjvx\" (UID: \"03b3012d-0bdf-4a78-ad31-c03d1df51ec0\") " pod="openshift-monitoring/node-exporter-gpjvx"
Nov 02 14:56:15.798893 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:56:15.798789    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/03b3012d-0bdf-4a78-ad31-c03d1df51ec0-node-exporter-wtmp\") pod \"node-exporter-gpjvx\" (UID: \"03b3012d-0bdf-4a78-ad31-c03d1df51ec0\") " pod="openshift-monitoring/node-exporter-gpjvx"
Nov 02 14:56:15.798893 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:56:15.798832    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/03b3012d-0bdf-4a78-ad31-c03d1df51ec0-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-gpjvx\" (UID: \"03b3012d-0bdf-4a78-ad31-c03d1df51ec0\") " pod="openshift-monitoring/node-exporter-gpjvx"
Nov 02 14:56:15.798893 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:56:15.798882    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/03b3012d-0bdf-4a78-ad31-c03d1df51ec0-root\") pod \"node-exporter-gpjvx\" (UID: \"03b3012d-0bdf-4a78-ad31-c03d1df51ec0\") " pod="openshift-monitoring/node-exporter-gpjvx"
Nov 02 14:56:15.799052 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:56:15.798914    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/03b3012d-0bdf-4a78-ad31-c03d1df51ec0-node-exporter-textfile\") pod \"node-exporter-gpjvx\" (UID: \"03b3012d-0bdf-4a78-ad31-c03d1df51ec0\") " pod="openshift-monitoring/node-exporter-gpjvx"
Nov 02 14:56:15.799052 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:56:15.798932    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/03b3012d-0bdf-4a78-ad31-c03d1df51ec0-metrics-client-ca\") pod \"node-exporter-gpjvx\" (UID: \"03b3012d-0bdf-4a78-ad31-c03d1df51ec0\") " pod="openshift-monitoring/node-exporter-gpjvx"
Nov 02 14:56:15.799052 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:56:15.798954    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/03b3012d-0bdf-4a78-ad31-c03d1df51ec0-node-exporter-tls\") pod \"node-exporter-gpjvx\" (UID: \"03b3012d-0bdf-4a78-ad31-c03d1df51ec0\") " pod="openshift-monitoring/node-exporter-gpjvx"
Nov 02 14:56:15.899546 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:56:15.899513    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/03b3012d-0bdf-4a78-ad31-c03d1df51ec0-node-exporter-wtmp\") pod \"node-exporter-gpjvx\" (UID: \"03b3012d-0bdf-4a78-ad31-c03d1df51ec0\") " pod="openshift-monitoring/node-exporter-gpjvx"
Nov 02 14:56:15.899546 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:56:15.899544    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/03b3012d-0bdf-4a78-ad31-c03d1df51ec0-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-gpjvx\" (UID: \"03b3012d-0bdf-4a78-ad31-c03d1df51ec0\") " pod="openshift-monitoring/node-exporter-gpjvx"
Nov 02 14:56:15.899754 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:56:15.899563    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/03b3012d-0bdf-4a78-ad31-c03d1df51ec0-root\") pod \"node-exporter-gpjvx\" (UID: \"03b3012d-0bdf-4a78-ad31-c03d1df51ec0\") " pod="openshift-monitoring/node-exporter-gpjvx"
Nov 02 14:56:15.899754 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:56:15.899679    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/03b3012d-0bdf-4a78-ad31-c03d1df51ec0-node-exporter-textfile\") pod \"node-exporter-gpjvx\" (UID: \"03b3012d-0bdf-4a78-ad31-c03d1df51ec0\") " pod="openshift-monitoring/node-exporter-gpjvx"
Nov 02 14:56:15.899754 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:56:15.899714    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/03b3012d-0bdf-4a78-ad31-c03d1df51ec0-metrics-client-ca\") pod \"node-exporter-gpjvx\" (UID: \"03b3012d-0bdf-4a78-ad31-c03d1df51ec0\") " pod="openshift-monitoring/node-exporter-gpjvx"
Nov 02 14:56:15.899754 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:56:15.899717    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/03b3012d-0bdf-4a78-ad31-c03d1df51ec0-root\") pod \"node-exporter-gpjvx\" (UID: \"03b3012d-0bdf-4a78-ad31-c03d1df51ec0\") " pod="openshift-monitoring/node-exporter-gpjvx"
Nov 02 14:56:15.899754 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:56:15.899708    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/03b3012d-0bdf-4a78-ad31-c03d1df51ec0-node-exporter-wtmp\") pod \"node-exporter-gpjvx\" (UID: \"03b3012d-0bdf-4a78-ad31-c03d1df51ec0\") " pod="openshift-monitoring/node-exporter-gpjvx"
Nov 02 14:56:15.899943 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:56:15.899746    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/03b3012d-0bdf-4a78-ad31-c03d1df51ec0-node-exporter-tls\") pod \"node-exporter-gpjvx\" (UID: \"03b3012d-0bdf-4a78-ad31-c03d1df51ec0\") " pod="openshift-monitoring/node-exporter-gpjvx"
Nov 02 14:56:15.899943 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:56:15.899828    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/03b3012d-0bdf-4a78-ad31-c03d1df51ec0-sys\") pod \"node-exporter-gpjvx\" (UID: \"03b3012d-0bdf-4a78-ad31-c03d1df51ec0\") " pod="openshift-monitoring/node-exporter-gpjvx"
Nov 02 14:56:15.899943 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:56:15.899850    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9vxjj\" (UniqueName: \"kubernetes.io/projected/03b3012d-0bdf-4a78-ad31-c03d1df51ec0-kube-api-access-9vxjj\") pod \"node-exporter-gpjvx\" (UID: \"03b3012d-0bdf-4a78-ad31-c03d1df51ec0\") " pod="openshift-monitoring/node-exporter-gpjvx"
Nov 02 14:56:15.899943 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:56:15.899882    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/03b3012d-0bdf-4a78-ad31-c03d1df51ec0-node-exporter-accelerators-collector-config\") pod \"node-exporter-gpjvx\" (UID: \"03b3012d-0bdf-4a78-ad31-c03d1df51ec0\") " pod="openshift-monitoring/node-exporter-gpjvx"
Nov 02 14:56:15.900089 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:56:15.899945    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/03b3012d-0bdf-4a78-ad31-c03d1df51ec0-sys\") pod \"node-exporter-gpjvx\" (UID: \"03b3012d-0bdf-4a78-ad31-c03d1df51ec0\") " pod="openshift-monitoring/node-exporter-gpjvx"
Nov 02 14:56:15.900089 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:56:15.900054    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/03b3012d-0bdf-4a78-ad31-c03d1df51ec0-node-exporter-textfile\") pod \"node-exporter-gpjvx\" (UID: \"03b3012d-0bdf-4a78-ad31-c03d1df51ec0\") " pod="openshift-monitoring/node-exporter-gpjvx"
Nov 02 14:56:15.900438 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:56:15.900409    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/03b3012d-0bdf-4a78-ad31-c03d1df51ec0-metrics-client-ca\") pod \"node-exporter-gpjvx\" (UID: \"03b3012d-0bdf-4a78-ad31-c03d1df51ec0\") " pod="openshift-monitoring/node-exporter-gpjvx"
Nov 02 14:56:15.900489 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:56:15.900409    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/03b3012d-0bdf-4a78-ad31-c03d1df51ec0-node-exporter-accelerators-collector-config\") pod \"node-exporter-gpjvx\" (UID: \"03b3012d-0bdf-4a78-ad31-c03d1df51ec0\") " pod="openshift-monitoring/node-exporter-gpjvx"
Nov 02 14:56:15.903710 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:56:15.903683    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/03b3012d-0bdf-4a78-ad31-c03d1df51ec0-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-gpjvx\" (UID: \"03b3012d-0bdf-4a78-ad31-c03d1df51ec0\") " pod="openshift-monitoring/node-exporter-gpjvx"
Nov 02 14:56:15.903820 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:56:15.903692    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/03b3012d-0bdf-4a78-ad31-c03d1df51ec0-node-exporter-tls\") pod \"node-exporter-gpjvx\" (UID: \"03b3012d-0bdf-4a78-ad31-c03d1df51ec0\") " pod="openshift-monitoring/node-exporter-gpjvx"
Nov 02 14:56:15.924315 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:56:15.924286    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9vxjj\" (UniqueName: \"kubernetes.io/projected/03b3012d-0bdf-4a78-ad31-c03d1df51ec0-kube-api-access-9vxjj\") pod \"node-exporter-gpjvx\" (UID: \"03b3012d-0bdf-4a78-ad31-c03d1df51ec0\") " pod="openshift-monitoring/node-exporter-gpjvx"
Nov 02 14:56:15.967469 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:56:15.967435    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-484ld"
Nov 02 14:56:15.967469 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:56:15.967452    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wns8r"
Nov 02 14:56:15.967469 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:56:15.967451    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vl6zb"
Nov 02 14:56:15.967772 ip-10-0-135-138 kubenswrapper[2566]: E1102 14:56:15.967544    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-484ld" podUID="caa8898b-6611-49b8-845d-3b9bdac6691f"
Nov 02 14:56:15.967772 ip-10-0-135-138 kubenswrapper[2566]: E1102 14:56:15.967626    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wns8r" podUID="0d6c5872-e278-4463-953d-e1388a001d0d"
Nov 02 14:56:15.967772 ip-10-0-135-138 kubenswrapper[2566]: E1102 14:56:15.967671    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vl6zb" podUID="332f4a2d-a241-4c71-8ebb-a99618c0d3dc"
Nov 02 14:56:16.050933 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:56:16.050851    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-gpjvx"
Nov 02 14:56:16.061105 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:56:16.061074    2566 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03b3012d_0bdf_4a78_ad31_c03d1df51ec0.slice/crio-2dffcc175ba039b64912a830e8787c67872dcb2ea3a2d115ce4d37a7558454e3 WatchSource:0}: Error finding container 2dffcc175ba039b64912a830e8787c67872dcb2ea3a2d115ce4d37a7558454e3: Status 404 returned error can't find the container with id 2dffcc175ba039b64912a830e8787c67872dcb2ea3a2d115ce4d37a7558454e3
Nov 02 14:56:16.323753 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:56:16.323669    2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-gpjvx" event={"ID":"03b3012d-0bdf-4a78-ad31-c03d1df51ec0","Type":"ContainerStarted","Data":"2dffcc175ba039b64912a830e8787c67872dcb2ea3a2d115ce4d37a7558454e3"}
Nov 02 14:56:17.326843 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:56:17.326755    2566 generic.go:356] "Generic (PLEG): container finished" podID="03b3012d-0bdf-4a78-ad31-c03d1df51ec0" containerID="799bb67abac0a19dcbe9f1b21ded87c530dea52098d262f901b1c05af88c84fa" exitCode=0
Nov 02 14:56:17.326843 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:56:17.326814    2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-gpjvx" event={"ID":"03b3012d-0bdf-4a78-ad31-c03d1df51ec0","Type":"ContainerDied","Data":"799bb67abac0a19dcbe9f1b21ded87c530dea52098d262f901b1c05af88c84fa"}
Nov 02 14:56:17.967483 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:56:17.967448    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-484ld"
Nov 02 14:56:17.967483 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:56:17.967473    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wns8r"
Nov 02 14:56:17.967690 ip-10-0-135-138 kubenswrapper[2566]: E1102 14:56:17.967602    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-484ld" podUID="caa8898b-6611-49b8-845d-3b9bdac6691f"
Nov 02 14:56:17.967690 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:56:17.967649    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vl6zb"
Nov 02 14:56:17.967768 ip-10-0-135-138 kubenswrapper[2566]: E1102 14:56:17.967717    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wns8r" podUID="0d6c5872-e278-4463-953d-e1388a001d0d"
Nov 02 14:56:17.967812 ip-10-0-135-138 kubenswrapper[2566]: E1102 14:56:17.967767    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vl6zb" podUID="332f4a2d-a241-4c71-8ebb-a99618c0d3dc"
Nov 02 14:56:18.330569 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:56:18.330534    2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-gpjvx" event={"ID":"03b3012d-0bdf-4a78-ad31-c03d1df51ec0","Type":"ContainerStarted","Data":"137b8473521027cc083853df0af10e2e484bdc65a3729f5ee748dfdb8040f6d7"}
Nov 02 14:56:18.330930 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:56:18.330594    2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-gpjvx" event={"ID":"03b3012d-0bdf-4a78-ad31-c03d1df51ec0","Type":"ContainerStarted","Data":"0fc9dd73d7470276d43a0d573b2712ac387f73e982a0df2a73cbc8fef0b35ecb"}
Nov 02 14:56:18.403171 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:56:18.403093    2566 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-gpjvx" podStartSLOduration=2.465284228 podStartE2EDuration="3.403078693s" totalImagesPullingTime="870.409444ms" totalInitContainerRuntime="67.385021ms" isStatefulPod=true podCreationTimestamp="2026-11-02 14:56:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:56:18.402939208 +0000 UTC m=+49.921962510" watchObservedRunningTime="2026-11-02 14:56:18.403078693 +0000 UTC m=+49.922101990"
Nov 02 14:56:19.966758 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:56:19.966719    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wns8r"
Nov 02 14:56:19.967160 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:56:19.966825    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-484ld"
Nov 02 14:56:19.967160 ip-10-0-135-138 kubenswrapper[2566]: E1102 14:56:19.966836    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wns8r" podUID="0d6c5872-e278-4463-953d-e1388a001d0d"
Nov 02 14:56:19.967160 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:56:19.966854    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vl6zb"
Nov 02 14:56:19.967160 ip-10-0-135-138 kubenswrapper[2566]: E1102 14:56:19.966929    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-484ld" podUID="caa8898b-6611-49b8-845d-3b9bdac6691f"
Nov 02 14:56:19.967160 ip-10-0-135-138 kubenswrapper[2566]: E1102 14:56:19.966989    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vl6zb" podUID="332f4a2d-a241-4c71-8ebb-a99618c0d3dc"
Nov 02 14:56:21.967004 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:56:21.966968    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vl6zb"
Nov 02 14:56:21.967004 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:56:21.966998    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wns8r"
Nov 02 14:56:21.967448 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:56:21.966975    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-484ld"
Nov 02 14:56:21.967448 ip-10-0-135-138 kubenswrapper[2566]: E1102 14:56:21.967113    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wns8r" podUID="0d6c5872-e278-4463-953d-e1388a001d0d"
Nov 02 14:56:21.967448 ip-10-0-135-138 kubenswrapper[2566]: E1102 14:56:21.967289    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-484ld" podUID="caa8898b-6611-49b8-845d-3b9bdac6691f"
Nov 02 14:56:21.967448 ip-10-0-135-138 kubenswrapper[2566]: E1102 14:56:21.967291    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vl6zb" podUID="332f4a2d-a241-4c71-8ebb-a99618c0d3dc"
Nov 02 14:56:23.296758 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:56:23.296679    2566 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-138.ec2.internal" event="NodeReady"
Nov 02 14:56:23.297272 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:56:23.296807    2566 kubelet_node_status.go:477] "Fast updating node status as it just became ready"
Nov 02 14:56:23.394100 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:56:23.394058    2566 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-fw8qc"]
Nov 02 14:56:23.420259 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:56:23.420202    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-fw8qc"
Nov 02 14:56:23.421475 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:56:23.421443    2566 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-k6fjk"]
Nov 02 14:56:23.424170 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:56:23.424141    2566 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-kfnkt"]
Nov 02 14:56:23.424443 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:56:23.424243    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-k6fjk"
Nov 02 14:56:23.426230 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:56:23.426205    2566 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-fw8qc"]
Nov 02 14:56:23.426327 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:56:23.426243    2566 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-k6fjk"]
Nov 02 14:56:23.426390 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:56:23.426340    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-kfnkt"
Nov 02 14:56:23.440744 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:56:23.440719    2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Nov 02 14:56:23.440937 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:56:23.440918    2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Nov 02 14:56:23.441109 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:56:23.441091    2566 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-kfnkt"]
Nov 02 14:56:23.450636 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:56:23.450618    2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Nov 02 14:56:23.450727 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:56:23.450639    2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Nov 02 14:56:23.450727 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:56:23.450652    2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Nov 02 14:56:23.450727 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:56:23.450655    2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Nov 02 14:56:23.457030 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:56:23.457009    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/702f7864-7db1-40d5-ba62-33cb9477b7da-data-volume\") pod \"insights-runtime-extractor-k6fjk\" (UID: \"702f7864-7db1-40d5-ba62-33cb9477b7da\") " pod="openshift-insights/insights-runtime-extractor-k6fjk"
Nov 02 14:56:23.457071 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:56:23.457035    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/080a338e-be02-49fb-bebe-f873e3f4017e-metrics-tls\") pod \"dns-default-kfnkt\" (UID: \"080a338e-be02-49fb-bebe-f873e3f4017e\") " pod="openshift-dns/dns-default-kfnkt"
Nov 02 14:56:23.457071 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:56:23.457062    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kw44n\" (UniqueName: \"kubernetes.io/projected/702f7864-7db1-40d5-ba62-33cb9477b7da-kube-api-access-kw44n\") pod \"insights-runtime-extractor-k6fjk\" (UID: \"702f7864-7db1-40d5-ba62-33cb9477b7da\") " pod="openshift-insights/insights-runtime-extractor-k6fjk"
Nov 02 14:56:23.457149 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:56:23.457081    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r67xl\" (UniqueName: \"kubernetes.io/projected/080a338e-be02-49fb-bebe-f873e3f4017e-kube-api-access-r67xl\") pod \"dns-default-kfnkt\" (UID: \"080a338e-be02-49fb-bebe-f873e3f4017e\") " pod="openshift-dns/dns-default-kfnkt"
Nov 02 14:56:23.457216 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:56:23.457203    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/702f7864-7db1-40d5-ba62-33cb9477b7da-crio-socket\") pod \"insights-runtime-extractor-k6fjk\" (UID: \"702f7864-7db1-40d5-ba62-33cb9477b7da\") " pod="openshift-insights/insights-runtime-extractor-k6fjk"
Nov 02 14:56:23.457272 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:56:23.457260    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/080a338e-be02-49fb-bebe-f873e3f4017e-config-volume\") pod \"dns-default-kfnkt\" (UID: \"080a338e-be02-49fb-bebe-f873e3f4017e\") " pod="openshift-dns/dns-default-kfnkt"
Nov 02 14:56:23.457304 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:56:23.457288    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/080a338e-be02-49fb-bebe-f873e3f4017e-tmp-dir\") pod \"dns-default-kfnkt\" (UID: \"080a338e-be02-49fb-bebe-f873e3f4017e\") " pod="openshift-dns/dns-default-kfnkt"
Nov 02 14:56:23.457336 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:56:23.457309    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ghkdw\" (UniqueName: \"kubernetes.io/projected/e4f9e55c-3232-430c-a123-30d17704cd2e-kube-api-access-ghkdw\") pod \"ingress-canary-fw8qc\" (UID: \"e4f9e55c-3232-430c-a123-30d17704cd2e\") " pod="openshift-ingress-canary/ingress-canary-fw8qc"
Nov 02 14:56:23.457391 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:56:23.457375    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/702f7864-7db1-40d5-ba62-33cb9477b7da-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-k6fjk\" (UID: \"702f7864-7db1-40d5-ba62-33cb9477b7da\") " pod="openshift-insights/insights-runtime-extractor-k6fjk"
Nov 02 14:56:23.457428 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:56:23.457398    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e4f9e55c-3232-430c-a123-30d17704cd2e-cert\") pod \"ingress-canary-fw8qc\" (UID: \"e4f9e55c-3232-430c-a123-30d17704cd2e\") " pod="openshift-ingress-canary/ingress-canary-fw8qc"
Nov 02 14:56:23.457459 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:56:23.457435    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/702f7864-7db1-40d5-ba62-33cb9477b7da-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-k6fjk\" (UID: \"702f7864-7db1-40d5-ba62-33cb9477b7da\") " pod="openshift-insights/insights-runtime-extractor-k6fjk"
Nov 02 14:56:23.513903 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:56:23.513869    2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Nov 02 14:56:23.514061 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:56:23.513973    2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-st89p\""
Nov 02 14:56:23.514061 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:56:23.514039    2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Nov 02 14:56:23.514149 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:56:23.514055    2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Nov 02 14:56:23.514149 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:56:23.514102    2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-jmg9z\""
Nov 02 14:56:23.514215 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:56:23.514183    2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-pj6nr\""
Nov 02 14:56:23.557794 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:56:23.557720    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/702f7864-7db1-40d5-ba62-33cb9477b7da-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-k6fjk\" (UID: \"702f7864-7db1-40d5-ba62-33cb9477b7da\") " pod="openshift-insights/insights-runtime-extractor-k6fjk"
Nov 02 14:56:23.557794 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:56:23.557783    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/702f7864-7db1-40d5-ba62-33cb9477b7da-data-volume\") pod \"insights-runtime-extractor-k6fjk\" (UID: \"702f7864-7db1-40d5-ba62-33cb9477b7da\") " pod="openshift-insights/insights-runtime-extractor-k6fjk"
Nov 02 14:56:23.557951 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:56:23.557809    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/080a338e-be02-49fb-bebe-f873e3f4017e-metrics-tls\") pod \"dns-default-kfnkt\" (UID: \"080a338e-be02-49fb-bebe-f873e3f4017e\") " pod="openshift-dns/dns-default-kfnkt"
Nov 02 14:56:23.557951 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:56:23.557847    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kw44n\" (UniqueName: \"kubernetes.io/projected/702f7864-7db1-40d5-ba62-33cb9477b7da-kube-api-access-kw44n\") pod \"insights-runtime-extractor-k6fjk\" (UID: \"702f7864-7db1-40d5-ba62-33cb9477b7da\") " pod="openshift-insights/insights-runtime-extractor-k6fjk"
Nov 02 14:56:23.557951 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:56:23.557871    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r67xl\" (UniqueName: \"kubernetes.io/projected/080a338e-be02-49fb-bebe-f873e3f4017e-kube-api-access-r67xl\") pod \"dns-default-kfnkt\" (UID: \"080a338e-be02-49fb-bebe-f873e3f4017e\") " pod="openshift-dns/dns-default-kfnkt"
Nov 02 14:56:23.557951 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:56:23.557894    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/702f7864-7db1-40d5-ba62-33cb9477b7da-crio-socket\") pod \"insights-runtime-extractor-k6fjk\" (UID: \"702f7864-7db1-40d5-ba62-33cb9477b7da\") " pod="openshift-insights/insights-runtime-extractor-k6fjk"
Nov 02 14:56:23.558168 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:56:23.558006    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/080a338e-be02-49fb-bebe-f873e3f4017e-config-volume\") pod \"dns-default-kfnkt\" (UID: \"080a338e-be02-49fb-bebe-f873e3f4017e\") " pod="openshift-dns/dns-default-kfnkt"
Nov 02 14:56:23.558168 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:56:23.558030    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/080a338e-be02-49fb-bebe-f873e3f4017e-tmp-dir\") pod \"dns-default-kfnkt\" (UID: \"080a338e-be02-49fb-bebe-f873e3f4017e\") " pod="openshift-dns/dns-default-kfnkt"
Nov 02 14:56:23.558168 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:56:23.558057    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ghkdw\" (UniqueName: \"kubernetes.io/projected/e4f9e55c-3232-430c-a123-30d17704cd2e-kube-api-access-ghkdw\") pod \"ingress-canary-fw8qc\" (UID: \"e4f9e55c-3232-430c-a123-30d17704cd2e\") " pod="openshift-ingress-canary/ingress-canary-fw8qc"
Nov 02 14:56:23.558168 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:56:23.558090    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/702f7864-7db1-40d5-ba62-33cb9477b7da-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-k6fjk\" (UID: \"702f7864-7db1-40d5-ba62-33cb9477b7da\") " pod="openshift-insights/insights-runtime-extractor-k6fjk"
Nov 02 14:56:23.558168 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:56:23.558112    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e4f9e55c-3232-430c-a123-30d17704cd2e-cert\") pod \"ingress-canary-fw8qc\" (UID: \"e4f9e55c-3232-430c-a123-30d17704cd2e\") " pod="openshift-ingress-canary/ingress-canary-fw8qc"
Nov 02 14:56:23.558421 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:56:23.558266    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/702f7864-7db1-40d5-ba62-33cb9477b7da-data-volume\") pod \"insights-runtime-extractor-k6fjk\" (UID: \"702f7864-7db1-40d5-ba62-33cb9477b7da\") " pod="openshift-insights/insights-runtime-extractor-k6fjk"
Nov 02 14:56:23.558470 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:56:23.558440    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/702f7864-7db1-40d5-ba62-33cb9477b7da-crio-socket\") pod \"insights-runtime-extractor-k6fjk\" (UID: \"702f7864-7db1-40d5-ba62-33cb9477b7da\") " pod="openshift-insights/insights-runtime-extractor-k6fjk"
Nov 02 14:56:23.558656 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:56:23.558632    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/080a338e-be02-49fb-bebe-f873e3f4017e-tmp-dir\") pod \"dns-default-kfnkt\" (UID: \"080a338e-be02-49fb-bebe-f873e3f4017e\") " pod="openshift-dns/dns-default-kfnkt"
Nov 02 14:56:23.558742 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:56:23.558691    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/702f7864-7db1-40d5-ba62-33cb9477b7da-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-k6fjk\" (UID: \"702f7864-7db1-40d5-ba62-33cb9477b7da\") " pod="openshift-insights/insights-runtime-extractor-k6fjk"
Nov 02 14:56:23.559011 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:56:23.558992    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/080a338e-be02-49fb-bebe-f873e3f4017e-config-volume\") pod \"dns-default-kfnkt\" (UID: \"080a338e-be02-49fb-bebe-f873e3f4017e\") " pod="openshift-dns/dns-default-kfnkt"
Nov 02 14:56:23.560558 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:56:23.560532    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e4f9e55c-3232-430c-a123-30d17704cd2e-cert\") pod \"ingress-canary-fw8qc\" (UID: \"e4f9e55c-3232-430c-a123-30d17704cd2e\") " pod="openshift-ingress-canary/ingress-canary-fw8qc"
Nov 02 14:56:23.560659 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:56:23.560642    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/702f7864-7db1-40d5-ba62-33cb9477b7da-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-k6fjk\" (UID: \"702f7864-7db1-40d5-ba62-33cb9477b7da\") " pod="openshift-insights/insights-runtime-extractor-k6fjk"
Nov 02 14:56:23.560708 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:56:23.560649    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/080a338e-be02-49fb-bebe-f873e3f4017e-metrics-tls\") pod \"dns-default-kfnkt\" (UID: \"080a338e-be02-49fb-bebe-f873e3f4017e\") " pod="openshift-dns/dns-default-kfnkt"
Nov 02 14:56:23.582181 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:56:23.582156    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kw44n\" (UniqueName: \"kubernetes.io/projected/702f7864-7db1-40d5-ba62-33cb9477b7da-kube-api-access-kw44n\") pod \"insights-runtime-extractor-k6fjk\" (UID: \"702f7864-7db1-40d5-ba62-33cb9477b7da\") " pod="openshift-insights/insights-runtime-extractor-k6fjk"
Nov 02 14:56:23.582292 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:56:23.582276    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r67xl\" (UniqueName: \"kubernetes.io/projected/080a338e-be02-49fb-bebe-f873e3f4017e-kube-api-access-r67xl\") pod \"dns-default-kfnkt\" (UID: \"080a338e-be02-49fb-bebe-f873e3f4017e\") " pod="openshift-dns/dns-default-kfnkt"
Nov 02 14:56:23.582478 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:56:23.582462    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ghkdw\" (UniqueName: \"kubernetes.io/projected/e4f9e55c-3232-430c-a123-30d17704cd2e-kube-api-access-ghkdw\") pod \"ingress-canary-fw8qc\" (UID: \"e4f9e55c-3232-430c-a123-30d17704cd2e\") " pod="openshift-ingress-canary/ingress-canary-fw8qc"
Nov 02 14:56:23.733921 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:56:23.733883    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-fw8qc"
Nov 02 14:56:23.742974 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:56:23.742953    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-k6fjk"
Nov 02 14:56:23.748001 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:56:23.747978    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-kfnkt"
Nov 02 14:56:23.967077 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:56:23.967046    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-484ld"
Nov 02 14:56:23.967077 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:56:23.967073    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wns8r"
Nov 02 14:56:23.967261 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:56:23.967208    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vl6zb"
Nov 02 14:56:23.983580 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:56:23.983560    2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Nov 02 14:56:23.983682 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:56:23.983605    2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Nov 02 14:56:23.983727 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:56:23.983688    2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Nov 02 14:56:23.983759 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:56:23.983730    2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-ssfbf\""
Nov 02 14:56:23.983826 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:56:23.983805    2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-dddv8\""
Nov 02 14:56:23.983898 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:56:23.983880    2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-nnwvk\""
Nov 02 14:56:23.984764 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:56:23.984750    2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Nov 02 14:56:24.055608 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:56:24.055575    2566 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-k6fjk"]
Nov 02 14:56:24.060102 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:56:24.060073    2566 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod702f7864_7db1_40d5_ba62_33cb9477b7da.slice/crio-979ce81d18065aab8e148d5e90f911903f8277633576644f10bbfa9486b6cadc WatchSource:0}: Error finding container 979ce81d18065aab8e148d5e90f911903f8277633576644f10bbfa9486b6cadc: Status 404 returned error can't find the container with id 979ce81d18065aab8e148d5e90f911903f8277633576644f10bbfa9486b6cadc
Nov 02 14:56:24.065288 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:56:24.065269    2566 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-fw8qc"]
Nov 02 14:56:24.068521 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:56:24.068501    2566 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-kfnkt"]
Nov 02 14:56:24.071553 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:56:24.071195    2566 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4f9e55c_3232_430c_a123_30d17704cd2e.slice/crio-9881d50d11ec0a6732c1726cbb74624ab2bdebf9c73f8b5222237dafe496e451 WatchSource:0}: Error finding container 9881d50d11ec0a6732c1726cbb74624ab2bdebf9c73f8b5222237dafe496e451: Status 404 returned error can't find the container with id 9881d50d11ec0a6732c1726cbb74624ab2bdebf9c73f8b5222237dafe496e451
Nov 02 14:56:24.072457 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:56:24.072435    2566 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod080a338e_be02_49fb_bebe_f873e3f4017e.slice/crio-03b691e65cf37f7a94beeee735d801b8dc31a3125b18747c22f89e827fd248ba WatchSource:0}: Error finding container 03b691e65cf37f7a94beeee735d801b8dc31a3125b18747c22f89e827fd248ba: Status 404 returned error can't find the container with id 03b691e65cf37f7a94beeee735d801b8dc31a3125b18747c22f89e827fd248ba
Nov 02 14:56:24.284774 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:56:24.284748    2566 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-f8btv"
Nov 02 14:56:24.348548 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:56:24.348512    2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-k6fjk" event={"ID":"702f7864-7db1-40d5-ba62-33cb9477b7da","Type":"ContainerStarted","Data":"d2a29536ce70e299d10ef6915b7f24148627329080c7b3961e79add091d16cc2"}
Nov 02 14:56:24.348980 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:56:24.348574    2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-k6fjk" event={"ID":"702f7864-7db1-40d5-ba62-33cb9477b7da","Type":"ContainerStarted","Data":"979ce81d18065aab8e148d5e90f911903f8277633576644f10bbfa9486b6cadc"}
Nov 02 14:56:24.350183 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:56:24.350069    2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-fw8qc" event={"ID":"e4f9e55c-3232-430c-a123-30d17704cd2e","Type":"ContainerStarted","Data":"9881d50d11ec0a6732c1726cbb74624ab2bdebf9c73f8b5222237dafe496e451"}
Nov 02 14:56:24.353684 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:56:24.351420    2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-kfnkt" event={"ID":"080a338e-be02-49fb-bebe-f873e3f4017e","Type":"ContainerStarted","Data":"03b691e65cf37f7a94beeee735d801b8dc31a3125b18747c22f89e827fd248ba"}
Nov 02 14:56:25.358542 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:56:25.358503    2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-k6fjk" event={"ID":"702f7864-7db1-40d5-ba62-33cb9477b7da","Type":"ContainerStarted","Data":"27c9a58f1e762306879c3907c87bb29b9ddbd27444388c7769694d758e97b1a9"}
Nov 02 14:56:26.366253 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:56:26.366211    2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-k6fjk" event={"ID":"702f7864-7db1-40d5-ba62-33cb9477b7da","Type":"ContainerStarted","Data":"fa2ef8182fe56dbe2a39bd19d4e773e7308e3ee37e624f7b138b88bfae05374a"}
Nov 02 14:56:26.368252 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:56:26.368226    2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-kfnkt" event={"ID":"080a338e-be02-49fb-bebe-f873e3f4017e","Type":"ContainerStarted","Data":"7251b2360449349f926aa5c0572446961a0735851246e7cd0ec4378445a152c8"}
Nov 02 14:56:26.368363 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:56:26.368277    2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-kfnkt" event={"ID":"080a338e-be02-49fb-bebe-f873e3f4017e","Type":"ContainerStarted","Data":"75e6d8f8d58e227dbe8e50847f495044932cd5d699a283520216b61a7c4b304b"}
Nov 02 14:56:26.368441 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:56:26.368373    2566 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-kfnkt"
Nov 02 14:56:26.412148 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:56:26.412068    2566 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-k6fjk" podStartSLOduration=1.320012671 podStartE2EDuration="3.412054698s" totalImagesPullingTime="2.092042027s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 14:56:23 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:56:26.411925589 +0000 UTC m=+57.930948898" watchObservedRunningTime="2026-11-02 14:56:26.412054698 +0000 UTC m=+57.931078034"
Nov 02 14:56:26.457817 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:56:26.457763    2566 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/dns-default-kfnkt" podStartSLOduration=1.623788477 podStartE2EDuration="3.457748439s" totalImagesPullingTime="1.833959962s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:56:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:56:26.457246456 +0000 UTC m=+57.976269756" watchObservedRunningTime="2026-11-02 14:56:26.457748439 +0000 UTC m=+57.976771739"
Nov 02 14:56:27.372818 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:56:27.372778    2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-fw8qc" event={"ID":"e4f9e55c-3232-430c-a123-30d17704cd2e","Type":"ContainerStarted","Data":"b5531be275f29cccd250a5177b92388fb27ce02c8213834f8a784274fa67dfc0"}
Nov 02 14:56:27.416927 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:56:27.416875    2566 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-fw8qc" podStartSLOduration=1.688928057 podStartE2EDuration="4.416856922s" totalImagesPullingTime="2.727928865s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:56:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:56:27.416042734 +0000 UTC m=+58.935066036" watchObservedRunningTime="2026-11-02 14:56:27.416856922 +0000 UTC m=+58.935880222"
Nov 02 14:56:34.746923 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:56:34.746874    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/332f4a2d-a241-4c71-8ebb-a99618c0d3dc-original-pull-secret\") pod \"global-pull-secret-syncer-vl6zb\" (UID: \"332f4a2d-a241-4c71-8ebb-a99618c0d3dc\") " pod="kube-system/global-pull-secret-syncer-vl6zb"
Nov 02 14:56:34.747382 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:56:34.746968    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/caa8898b-6611-49b8-845d-3b9bdac6691f-metrics-certs\") pod \"network-metrics-daemon-484ld\" (UID: \"caa8898b-6611-49b8-845d-3b9bdac6691f\") " pod="openshift-multus/network-metrics-daemon-484ld"
Nov 02 14:56:34.762236 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:56:34.762212    2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Nov 02 14:56:34.769458 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:56:34.769430    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/caa8898b-6611-49b8-845d-3b9bdac6691f-metrics-certs\") pod \"network-metrics-daemon-484ld\" (UID: \"caa8898b-6611-49b8-845d-3b9bdac6691f\") " pod="openshift-multus/network-metrics-daemon-484ld"
Nov 02 14:56:34.775495 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:56:34.775476    2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Nov 02 14:56:34.779482 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:56:34.779466    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/332f4a2d-a241-4c71-8ebb-a99618c0d3dc-original-pull-secret\") pod \"global-pull-secret-syncer-vl6zb\" (UID: \"332f4a2d-a241-4c71-8ebb-a99618c0d3dc\") " pod="kube-system/global-pull-secret-syncer-vl6zb"
Nov 02 14:56:34.794628 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:56:34.794609    2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-ssfbf\""
Nov 02 14:56:34.798385 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:56:34.798369    2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-nnwvk\""
Nov 02 14:56:34.800476 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:56:34.800459    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-484ld"
Nov 02 14:56:34.804097 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:56:34.804081    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vl6zb"
Nov 02 14:56:34.848662 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:56:34.848558    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hpfk5\" (UniqueName: \"kubernetes.io/projected/0d6c5872-e278-4463-953d-e1388a001d0d-kube-api-access-hpfk5\") pod \"network-check-target-wns8r\" (UID: \"0d6c5872-e278-4463-953d-e1388a001d0d\") " pod="openshift-network-diagnostics/network-check-target-wns8r"
Nov 02 14:56:34.865190 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:56:34.865162    2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Nov 02 14:56:34.885155 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:56:34.885100    2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Nov 02 14:56:34.893513 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:56:34.893489    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hpfk5\" (UniqueName: \"kubernetes.io/projected/0d6c5872-e278-4463-953d-e1388a001d0d-kube-api-access-hpfk5\") pod \"network-check-target-wns8r\" (UID: \"0d6c5872-e278-4463-953d-e1388a001d0d\") " pod="openshift-network-diagnostics/network-check-target-wns8r"
Nov 02 14:56:35.103877 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:56:35.103844    2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-dddv8\""
Nov 02 14:56:35.108897 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:56:35.108876    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wns8r"
Nov 02 14:56:35.113622 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:56:35.113602    2566 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-vl6zb"]
Nov 02 14:56:35.116408 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:56:35.116384    2566 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod332f4a2d_a241_4c71_8ebb_a99618c0d3dc.slice/crio-d5d6f959273f30a64b18a421bf431362ed4babd9fee6bef83d84d8bf6665d58d WatchSource:0}: Error finding container d5d6f959273f30a64b18a421bf431362ed4babd9fee6bef83d84d8bf6665d58d: Status 404 returned error can't find the container with id d5d6f959273f30a64b18a421bf431362ed4babd9fee6bef83d84d8bf6665d58d
Nov 02 14:56:35.127316 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:56:35.127296    2566 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-484ld"]
Nov 02 14:56:35.129702 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:56:35.129676    2566 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcaa8898b_6611_49b8_845d_3b9bdac6691f.slice/crio-b236aedf11348eb6319968e96631ba9fb50c69aafadf3aa5a5c86ad633dd1234 WatchSource:0}: Error finding container b236aedf11348eb6319968e96631ba9fb50c69aafadf3aa5a5c86ad633dd1234: Status 404 returned error can't find the container with id b236aedf11348eb6319968e96631ba9fb50c69aafadf3aa5a5c86ad633dd1234
Nov 02 14:56:35.396976 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:56:35.396883    2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-vl6zb" event={"ID":"332f4a2d-a241-4c71-8ebb-a99618c0d3dc","Type":"ContainerStarted","Data":"d5d6f959273f30a64b18a421bf431362ed4babd9fee6bef83d84d8bf6665d58d"}
Nov 02 14:56:35.397988 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:56:35.397963    2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-484ld" event={"ID":"caa8898b-6611-49b8-845d-3b9bdac6691f","Type":"ContainerStarted","Data":"b236aedf11348eb6319968e96631ba9fb50c69aafadf3aa5a5c86ad633dd1234"}
Nov 02 14:56:35.406063 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:56:35.406036    2566 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-wns8r"]
Nov 02 14:56:35.408880 ip-10-0-135-138 kubenswrapper[2566]: W1102 14:56:35.408851    2566 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d6c5872_e278_4463_953d_e1388a001d0d.slice/crio-6917bc2965fdd531105d8ed16d2dea52c30702bfe6a4fd41de68d832fb3dff14 WatchSource:0}: Error finding container 6917bc2965fdd531105d8ed16d2dea52c30702bfe6a4fd41de68d832fb3dff14: Status 404 returned error can't find the container with id 6917bc2965fdd531105d8ed16d2dea52c30702bfe6a4fd41de68d832fb3dff14
Nov 02 14:56:36.375203 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:56:36.375096    2566 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-kfnkt"
Nov 02 14:56:36.404184 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:56:36.404071    2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-484ld" event={"ID":"caa8898b-6611-49b8-845d-3b9bdac6691f","Type":"ContainerStarted","Data":"b4d8dc02af6546d1d270df3fd9d1de8c28a86d80bc2533c05481bad44778adb7"}
Nov 02 14:56:36.405694 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:56:36.405620    2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-wns8r" event={"ID":"0d6c5872-e278-4463-953d-e1388a001d0d","Type":"ContainerStarted","Data":"6917bc2965fdd531105d8ed16d2dea52c30702bfe6a4fd41de68d832fb3dff14"}
Nov 02 14:56:37.411726 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:56:37.411684    2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-484ld" event={"ID":"caa8898b-6611-49b8-845d-3b9bdac6691f","Type":"ContainerStarted","Data":"31f3547bc1c122016575c75785651466e58539d5a91e2bd39e27060b89f1b03a"}
Nov 02 14:56:37.458091 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:56:37.458018    2566 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-484ld" podStartSLOduration=66.379190286 podStartE2EDuration="1m7.457999327s" totalImagesPullingTime="1.078809041s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:55:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:56:37.457147612 +0000 UTC m=+68.976170913" watchObservedRunningTime="2026-11-02 14:56:37.457999327 +0000 UTC m=+68.977022628"
Nov 02 14:56:39.421310 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:56:39.421274    2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-wns8r" event={"ID":"0d6c5872-e278-4463-953d-e1388a001d0d","Type":"ContainerStarted","Data":"ee87648208aec75acb2ac8741544bb4048be949f0dbcc30281dde49ea525923b"}
Nov 02 14:56:39.421745 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:56:39.421401    2566 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-wns8r"
Nov 02 14:56:39.466736 ip-10-0-135-138 kubenswrapper[2566]: I1102 14:56:39.466686    2566 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-wns8r" podStartSLOduration=66.335398788 podStartE2EDuration="1m9.46667335s" totalImagesPullingTime="3.131274562s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:55:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:56:39.465535089 +0000 UTC m=+70.984558391" watchObservedRunningTime="2026-11-02 14:56:39.46667335 +0000 UTC m=+70.985696649"
Nov 02 14:56:39.932533 ip-10-0-135-138 systemd[1]: Stopping Kubernetes Kubelet...
Nov 02 14:56:39.956610 ip-10-0-135-138 systemd[1]: kubelet.service: Deactivated successfully.
Nov 02 14:56:39.956778 ip-10-0-135-138 systemd[1]: Stopped Kubernetes Kubelet.
Nov 02 14:56:39.956828 ip-10-0-135-138 systemd[1]: kubelet.service: Consumed 3.531s CPU time.
Nov 02 14:56:39.973901 ip-10-0-135-138 systemd[1]: Starting Kubernetes Kubelet...
Nov 02 14:56:40.035731 ip-10-0-135-138 kubenswrapper[5304]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 02 14:56:40.035731 ip-10-0-135-138 kubenswrapper[5304]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Nov 02 14:56:40.035731 ip-10-0-135-138 kubenswrapper[5304]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 02 14:56:40.035731 ip-10-0-135-138 kubenswrapper[5304]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 02 14:56:40.037869 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.037780    5304 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 14:56:40.037869 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.037870    5304 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 14:56:40.037944 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.037875    5304 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 14:56:40.037944 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.037878    5304 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 14:56:40.037944 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.037882    5304 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 14:56:40.037944 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.037885    5304 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 14:56:40.037944 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.037888    5304 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 14:56:40.037944 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.037890    5304 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 14:56:40.037944 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.037893    5304 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 14:56:40.037944 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.037897    5304 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 14:56:40.037944 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.037900    5304 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 14:56:40.037944 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.037902    5304 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 14:56:40.037944 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.037905    5304 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 14:56:40.037944 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.037908    5304 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 14:56:40.037944 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.037911    5304 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 14:56:40.037944 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.037914    5304 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 14:56:40.037944 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.037916    5304 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 14:56:40.037944 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.037919    5304 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 14:56:40.037944 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.037922    5304 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 14:56:40.037944 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.037924    5304 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 14:56:40.037944 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.037927    5304 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 14:56:40.038442 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.037930    5304 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 14:56:40.038442 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.037937    5304 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 14:56:40.038442 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.037940    5304 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 14:56:40.038442 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.037943    5304 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 14:56:40.038442 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.037946    5304 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 14:56:40.038442 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.037949    5304 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 14:56:40.038442 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.037952    5304 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 14:56:40.038442 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.037955    5304 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 14:56:40.038442 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.037958    5304 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 14:56:40.038442 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.037961    5304 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 14:56:40.038442 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.037964    5304 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 14:56:40.038442 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.037967    5304 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 14:56:40.038442 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.037970    5304 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 14:56:40.038442 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.037972    5304 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 14:56:40.038442 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.037975    5304 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 14:56:40.038442 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.037978    5304 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 14:56:40.038442 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.037980    5304 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 14:56:40.038442 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.037983    5304 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 14:56:40.038442 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.037985    5304 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 14:56:40.038979 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.037988    5304 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 14:56:40.038979 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.037991    5304 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 14:56:40.038979 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.037993    5304 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 14:56:40.038979 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.037996    5304 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 14:56:40.038979 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.037998    5304 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 14:56:40.038979 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.038001    5304 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 14:56:40.038979 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.038004    5304 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 14:56:40.038979 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.038006    5304 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 14:56:40.038979 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.038009    5304 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 14:56:40.038979 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.038011    5304 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 14:56:40.038979 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.038014    5304 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 14:56:40.038979 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.038018    5304 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 14:56:40.038979 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.038021    5304 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 14:56:40.038979 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.038023    5304 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 14:56:40.038979 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.038027    5304 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 14:56:40.038979 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.038033    5304 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 14:56:40.038979 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.038037    5304 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 14:56:40.038979 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.038040    5304 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 14:56:40.038979 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.038043    5304 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 14:56:40.038979 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.038045    5304 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 14:56:40.039491 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.038048    5304 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 14:56:40.039491 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.038051    5304 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 14:56:40.039491 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.038055    5304 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 14:56:40.039491 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.038059    5304 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 14:56:40.039491 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.038062    5304 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 14:56:40.039491 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.038066    5304 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 14:56:40.039491 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.038069    5304 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 14:56:40.039491 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.038072    5304 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 14:56:40.039491 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.038076    5304 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 14:56:40.039491 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.038081    5304 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 14:56:40.039491 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.038084    5304 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 14:56:40.039491 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.038087    5304 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 14:56:40.039491 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.038089    5304 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 14:56:40.039491 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.038092    5304 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 14:56:40.039491 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.038095    5304 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 14:56:40.039491 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.038098    5304 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 14:56:40.039491 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.038100    5304 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 14:56:40.039491 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.038103    5304 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 14:56:40.039491 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.038106    5304 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 14:56:40.039961 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.038109    5304 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 14:56:40.039961 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.038112    5304 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 14:56:40.039961 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.038114    5304 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 14:56:40.039961 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.038117    5304 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 14:56:40.039961 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.038120    5304 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 14:56:40.039961 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.038138    5304 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 14:56:40.039961 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.038141    5304 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 14:56:40.039961 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.038145    5304 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 14:56:40.039961 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.038148    5304 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 14:56:40.039961 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.038162    5304 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 14:56:40.039961 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.038165    5304 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 14:56:40.039961 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.038168    5304 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 14:56:40.039961 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.038171    5304 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 14:56:40.039961 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.038174    5304 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 14:56:40.039961 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.038178    5304 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 14:56:40.039961 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.038180    5304 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 14:56:40.039961 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.038183    5304 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 14:56:40.039961 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.038186    5304 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 14:56:40.039961 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.038189    5304 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 14:56:40.039961 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.038191    5304 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 14:56:40.040480 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.038194    5304 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 14:56:40.040480 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.038197    5304 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 14:56:40.040480 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.038199    5304 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 14:56:40.040480 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.038202    5304 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 14:56:40.040480 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.038205    5304 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 14:56:40.040480 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.038208    5304 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 14:56:40.040480 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.038210    5304 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 14:56:40.040480 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.039003    5304 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 14:56:40.040480 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.039011    5304 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 14:56:40.040480 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.039014    5304 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 14:56:40.040480 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.039019    5304 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 14:56:40.040480 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.039023    5304 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 14:56:40.040480 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.039026    5304 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 14:56:40.040480 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.039029    5304 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 14:56:40.040480 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.039033    5304 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 14:56:40.040480 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.039036    5304 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 14:56:40.040480 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.039039    5304 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 14:56:40.040480 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.039041    5304 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 14:56:40.040480 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.039044    5304 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 14:56:40.040946 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.039047    5304 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 14:56:40.040946 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.039050    5304 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 14:56:40.040946 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.039052    5304 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 14:56:40.040946 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.039055    5304 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 14:56:40.040946 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.039058    5304 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 14:56:40.040946 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.039061    5304 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 14:56:40.040946 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.039064    5304 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 14:56:40.040946 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.039067    5304 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 14:56:40.040946 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.039069    5304 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 14:56:40.040946 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.039072    5304 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 14:56:40.040946 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.039075    5304 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 14:56:40.040946 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.039078    5304 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 14:56:40.040946 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.039081    5304 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 14:56:40.040946 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.039083    5304 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 14:56:40.040946 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.039086    5304 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 14:56:40.040946 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.039089    5304 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 14:56:40.040946 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.039092    5304 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 14:56:40.040946 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.039095    5304 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 14:56:40.040946 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.039098    5304 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 14:56:40.040946 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.039101    5304 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 14:56:40.041453 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.039103    5304 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 14:56:40.041453 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.039106    5304 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 14:56:40.041453 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.039109    5304 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 14:56:40.041453 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.039112    5304 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 14:56:40.041453 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.039115    5304 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 14:56:40.041453 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.039117    5304 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 14:56:40.041453 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.039134    5304 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 14:56:40.041453 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.039137    5304 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 14:56:40.041453 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.039140    5304 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 14:56:40.041453 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.039143    5304 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 14:56:40.041453 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.039146    5304 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 14:56:40.041453 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.039148    5304 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 14:56:40.041453 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.039151    5304 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 14:56:40.041453 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.039154    5304 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 14:56:40.041453 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.039157    5304 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 14:56:40.041453 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.039160    5304 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 14:56:40.041453 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.039163    5304 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 14:56:40.041453 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.039166    5304 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 14:56:40.041453 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.039168    5304 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 14:56:40.041989 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.039171    5304 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 14:56:40.041989 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.039173    5304 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 14:56:40.041989 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.039176    5304 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 14:56:40.041989 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.039179    5304 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 14:56:40.041989 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.039183    5304 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 14:56:40.041989 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.039185    5304 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 14:56:40.041989 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.039188    5304 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 14:56:40.041989 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.039191    5304 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 14:56:40.041989 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.039193    5304 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 14:56:40.041989 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.039196    5304 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 14:56:40.041989 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.039199    5304 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 14:56:40.041989 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.039202    5304 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 14:56:40.041989 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.039205    5304 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 14:56:40.041989 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.039208    5304 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 14:56:40.041989 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.039211    5304 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 14:56:40.041989 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.039214    5304 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 14:56:40.041989 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.039216    5304 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 14:56:40.041989 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.039219    5304 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 14:56:40.041989 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.039221    5304 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 14:56:40.042529 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.039224    5304 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 14:56:40.042529 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.039227    5304 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 14:56:40.042529 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.039231    5304 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 14:56:40.042529 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.039235    5304 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 14:56:40.042529 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.039238    5304 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 14:56:40.042529 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.039241    5304 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 14:56:40.042529 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.039243    5304 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 14:56:40.042529 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.039246    5304 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 14:56:40.042529 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.039249    5304 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 14:56:40.042529 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.039251    5304 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 14:56:40.042529 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.039254    5304 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 14:56:40.042529 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.039257    5304 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 14:56:40.042529 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.039260    5304 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 14:56:40.042529 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.039263    5304 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 14:56:40.042529 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.039265    5304 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 14:56:40.042529 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.039268    5304 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 14:56:40.042529 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.039271    5304 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 14:56:40.042529 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.039273    5304 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 14:56:40.042529 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.039276    5304 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 14:56:40.043020 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.039279    5304 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 14:56:40.043020 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.039281    5304 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 14:56:40.043020 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.039284    5304 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 14:56:40.043020 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.039287    5304 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 14:56:40.043020 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.039289    5304 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 14:56:40.043020 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.039292    5304 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 14:56:40.043020 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.039295    5304 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 14:56:40.043020 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.039297    5304 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 14:56:40.043020 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.039300    5304 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 14:56:40.043020 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.039306    5304 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 14:56:40.043020 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.039308    5304 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 14:56:40.043020 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.039312    5304 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 14:56:40.043020 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.039315    5304 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 14:56:40.043020 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.039318    5304 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 14:56:40.043020 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.039320    5304 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 14:56:40.043020 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.039323    5304 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 14:56:40.043020 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.039325    5304 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 14:56:40.043020 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.039807    5304 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 14:56:40.043020 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.039812    5304 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 14:56:40.043537 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.039816    5304 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 14:56:40.043537 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.039819    5304 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 14:56:40.043537 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.039821    5304 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 14:56:40.043537 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.039824    5304 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 14:56:40.043537 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.039827    5304 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 14:56:40.043537 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.039830    5304 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 14:56:40.043537 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.039833    5304 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 14:56:40.043537 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.039837    5304 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 14:56:40.043537 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.039839    5304 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 14:56:40.043537 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.039842    5304 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 14:56:40.043537 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.039845    5304 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 14:56:40.043537 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.039848    5304 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 14:56:40.043537 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.039850    5304 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 14:56:40.043537 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.039854    5304 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 14:56:40.043537 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.039858    5304 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 14:56:40.043537 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.039861    5304 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 14:56:40.043537 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.039864    5304 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 14:56:40.043537 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.039867    5304 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 14:56:40.043537 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.039869    5304 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 14:56:40.044034 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.039872    5304 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 14:56:40.044034 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.039874    5304 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 14:56:40.044034 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.039877    5304 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 14:56:40.044034 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.039883    5304 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 14:56:40.044034 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.039885    5304 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 14:56:40.044034 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.039888    5304 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 14:56:40.044034 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.039891    5304 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 14:56:40.044034 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.039893    5304 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 14:56:40.044034 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.039896    5304 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 14:56:40.044034 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.039899    5304 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 14:56:40.044034 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.039902    5304 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 14:56:40.044034 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.039904    5304 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 14:56:40.044034 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.039909    5304 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 14:56:40.044034 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.039912    5304 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 14:56:40.044034 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.039914    5304 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 14:56:40.044034 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.039917    5304 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 14:56:40.044034 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.039920    5304 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 14:56:40.044034 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.039922    5304 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 14:56:40.044034 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.039925    5304 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 14:56:40.044034 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.039928    5304 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 14:56:40.044552 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.039931    5304 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 14:56:40.044552 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.039934    5304 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 14:56:40.044552 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.039937    5304 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 14:56:40.044552 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.039939    5304 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 14:56:40.044552 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.039942    5304 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 14:56:40.044552 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.039944    5304 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 14:56:40.044552 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.039947    5304 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 14:56:40.044552 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.039950    5304 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 14:56:40.044552 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.039953    5304 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 14:56:40.044552 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.039955    5304 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 14:56:40.044552 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.039958    5304 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 14:56:40.044552 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.039960    5304 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 14:56:40.044552 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.039963    5304 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 14:56:40.044552 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.039966    5304 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 14:56:40.044552 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.039969    5304 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 14:56:40.044552 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.039972    5304 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 14:56:40.044552 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.039975    5304 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 14:56:40.044552 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.039978    5304 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 14:56:40.044552 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.039981    5304 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 14:56:40.044552 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.039983    5304 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 14:56:40.045090 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.039986    5304 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 14:56:40.045090 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.039988    5304 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 14:56:40.045090 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.039993    5304 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 14:56:40.045090 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.039997    5304 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 14:56:40.045090 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.040001    5304 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 14:56:40.045090 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.040004    5304 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 14:56:40.045090 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.040007    5304 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 14:56:40.045090 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.040010    5304 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 14:56:40.045090 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.040013    5304 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 14:56:40.045090 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.040016    5304 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 14:56:40.045090 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.040018    5304 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 14:56:40.045090 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.040021    5304 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 14:56:40.045090 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.040024    5304 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 14:56:40.045090 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.040027    5304 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 14:56:40.045090 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.040029    5304 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 14:56:40.045090 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.040032    5304 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 14:56:40.045090 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.040034    5304 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 14:56:40.045090 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.040037    5304 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 14:56:40.045090 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.040040    5304 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 14:56:40.045571 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.040042    5304 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 14:56:40.045571 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.040045    5304 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 14:56:40.045571 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.040047    5304 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 14:56:40.045571 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.040050    5304 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 14:56:40.045571 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.040052    5304 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 14:56:40.045571 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.040055    5304 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 14:56:40.045571 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.040058    5304 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 14:56:40.045571 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.040061    5304 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 14:56:40.045571 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.040064    5304 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 14:56:40.045571 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.040066    5304 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 14:56:40.045571 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.040069    5304 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 14:56:40.045571 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.040072    5304 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 14:56:40.045571 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.040075    5304 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 14:56:40.045571 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.040078    5304 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 14:56:40.045571 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.040080    5304 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 14:56:40.045571 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.040083    5304 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 14:56:40.045571 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.040085    5304 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 14:56:40.045571 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.040089    5304 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 14:56:40.045571 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.040092    5304 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 14:56:40.046094 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.040094    5304 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 14:56:40.046094 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.040097    5304 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 14:56:40.046094 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.040100    5304 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 14:56:40.046094 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.040102    5304 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 14:56:40.046094 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.040106    5304 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 14:56:40.046094 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.040109    5304 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 14:56:40.046094 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.040111    5304 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 14:56:40.046094 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.040219    5304 flags.go:64] FLAG: --address="0.0.0.0"
Nov 02 14:56:40.046094 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.040236    5304 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Nov 02 14:56:40.046094 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.040246    5304 flags.go:64] FLAG: --anonymous-auth="true"
Nov 02 14:56:40.046094 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.040252    5304 flags.go:64] FLAG: --application-metrics-count-limit="100"
Nov 02 14:56:40.046094 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.040256    5304 flags.go:64] FLAG: --authentication-token-webhook="false"
Nov 02 14:56:40.046094 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.040260    5304 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Nov 02 14:56:40.046094 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.040264    5304 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Nov 02 14:56:40.046094 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.040269    5304 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Nov 02 14:56:40.046094 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.040273    5304 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Nov 02 14:56:40.046094 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.040276    5304 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Nov 02 14:56:40.046094 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.040280    5304 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Nov 02 14:56:40.046094 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.040283    5304 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Nov 02 14:56:40.046094 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.040286    5304 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Nov 02 14:56:40.046094 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.040289    5304 flags.go:64] FLAG: --cgroup-root=""
Nov 02 14:56:40.046094 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.040292    5304 flags.go:64] FLAG: --cgroups-per-qos="true"
Nov 02 14:56:40.046094 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.040296    5304 flags.go:64] FLAG: --client-ca-file=""
Nov 02 14:56:40.046683 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.040299    5304 flags.go:64] FLAG: --cloud-provider="external"
Nov 02 14:56:40.046683 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.040302    5304 flags.go:64] FLAG: --cluster-dns="[]"
Nov 02 14:56:40.046683 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.040308    5304 flags.go:64] FLAG: --cluster-domain=""
Nov 02 14:56:40.046683 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.040311    5304 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Nov 02 14:56:40.046683 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.040315    5304 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Nov 02 14:56:40.046683 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.040318    5304 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Nov 02 14:56:40.046683 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.040321    5304 flags.go:64] FLAG: --container-log-max-files="5"
Nov 02 14:56:40.046683 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.040328    5304 flags.go:64] FLAG: --container-log-max-size="10Mi"
Nov 02 14:56:40.046683 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.040331    5304 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Nov 02 14:56:40.046683 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.040334    5304 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Nov 02 14:56:40.046683 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.040338    5304 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Nov 02 14:56:40.046683 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.040341    5304 flags.go:64] FLAG: --contention-profiling="false"
Nov 02 14:56:40.046683 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.040344    5304 flags.go:64] FLAG: --cpu-cfs-quota="true"
Nov 02 14:56:40.046683 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.040347    5304 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Nov 02 14:56:40.046683 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.040350    5304 flags.go:64] FLAG: --cpu-manager-policy="none"
Nov 02 14:56:40.046683 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.040353    5304 flags.go:64] FLAG: --cpu-manager-policy-options=""
Nov 02 14:56:40.046683 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.040357    5304 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Nov 02 14:56:40.046683 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.040361    5304 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Nov 02 14:56:40.046683 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.040364    5304 flags.go:64] FLAG: --enable-debugging-handlers="true"
Nov 02 14:56:40.046683 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.040368    5304 flags.go:64] FLAG: --enable-load-reader="false"
Nov 02 14:56:40.046683 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.040371    5304 flags.go:64] FLAG: --enable-server="true"
Nov 02 14:56:40.046683 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.040374    5304 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Nov 02 14:56:40.046683 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.040377    5304 flags.go:64] FLAG: --event-burst="100"
Nov 02 14:56:40.046683 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.040380    5304 flags.go:64] FLAG: --event-qps="50"
Nov 02 14:56:40.046683 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.040383    5304 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Nov 02 14:56:40.047429 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.040387    5304 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Nov 02 14:56:40.047429 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.040390    5304 flags.go:64] FLAG: --eviction-hard=""
Nov 02 14:56:40.047429 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.040394    5304 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Nov 02 14:56:40.047429 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.040397    5304 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Nov 02 14:56:40.047429 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.040400    5304 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Nov 02 14:56:40.047429 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.040403    5304 flags.go:64] FLAG: --eviction-soft=""
Nov 02 14:56:40.047429 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.040406    5304 flags.go:64] FLAG: --eviction-soft-grace-period=""
Nov 02 14:56:40.047429 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.040409    5304 flags.go:64] FLAG: --exit-on-lock-contention="false"
Nov 02 14:56:40.047429 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.040413    5304 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Nov 02 14:56:40.047429 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.040416    5304 flags.go:64] FLAG: --experimental-mounter-path=""
Nov 02 14:56:40.047429 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.040426    5304 flags.go:64] FLAG: --fail-cgroupv1="true"
Nov 02 14:56:40.047429 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.040430    5304 flags.go:64] FLAG: --fail-swap-on="true"
Nov 02 14:56:40.047429 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.040433    5304 flags.go:64] FLAG: --feature-gates=""
Nov 02 14:56:40.047429 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.040437    5304 flags.go:64] FLAG: --file-check-frequency="20s"
Nov 02 14:56:40.047429 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.040442    5304 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Nov 02 14:56:40.047429 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.040446    5304 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Nov 02 14:56:40.047429 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.040449    5304 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Nov 02 14:56:40.047429 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.040452    5304 flags.go:64] FLAG: --healthz-port="10248"
Nov 02 14:56:40.047429 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.040455    5304 flags.go:64] FLAG: --help="false"
Nov 02 14:56:40.047429 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.040458    5304 flags.go:64] FLAG: --hostname-override="ip-10-0-135-138.ec2.internal"
Nov 02 14:56:40.047429 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.040461    5304 flags.go:64] FLAG: --housekeeping-interval="10s"
Nov 02 14:56:40.047429 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.040464    5304 flags.go:64] FLAG: --http-check-frequency="20s"
Nov 02 14:56:40.047429 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.040467    5304 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Nov 02 14:56:40.048395 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.040471    5304 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Nov 02 14:56:40.048395 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.040477    5304 flags.go:64] FLAG: --image-gc-high-threshold="85"
Nov 02 14:56:40.048395 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.040481    5304 flags.go:64] FLAG: --image-gc-low-threshold="80"
Nov 02 14:56:40.048395 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.040484    5304 flags.go:64] FLAG: --image-service-endpoint=""
Nov 02 14:56:40.048395 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.040487    5304 flags.go:64] FLAG: --kernel-memcg-notification="false"
Nov 02 14:56:40.048395 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.040490    5304 flags.go:64] FLAG: --kube-api-burst="100"
Nov 02 14:56:40.048395 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.040493    5304 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Nov 02 14:56:40.048395 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.040496    5304 flags.go:64] FLAG: --kube-api-qps="50"
Nov 02 14:56:40.048395 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.040499    5304 flags.go:64] FLAG: --kube-reserved=""
Nov 02 14:56:40.048395 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.040503    5304 flags.go:64] FLAG: --kube-reserved-cgroup=""
Nov 02 14:56:40.048395 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.040506    5304 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Nov 02 14:56:40.048395 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.040509    5304 flags.go:64] FLAG: --kubelet-cgroups=""
Nov 02 14:56:40.048395 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.040512    5304 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Nov 02 14:56:40.048395 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.040515    5304 flags.go:64] FLAG: --lock-file=""
Nov 02 14:56:40.048395 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.040518    5304 flags.go:64] FLAG: --log-cadvisor-usage="false"
Nov 02 14:56:40.048395 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.040521    5304 flags.go:64] FLAG: --log-flush-frequency="5s"
Nov 02 14:56:40.048395 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.040524    5304 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Nov 02 14:56:40.048395 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.040535    5304 flags.go:64] FLAG: --log-json-split-stream="false"
Nov 02 14:56:40.048395 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.040538    5304 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Nov 02 14:56:40.048395 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.040541    5304 flags.go:64] FLAG: --log-text-split-stream="false"
Nov 02 14:56:40.048395 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.040545    5304 flags.go:64] FLAG: --logging-format="text"
Nov 02 14:56:40.048395 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.040548    5304 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Nov 02 14:56:40.048395 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.040551    5304 flags.go:64] FLAG: --make-iptables-util-chains="true"
Nov 02 14:56:40.048395 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.040556    5304 flags.go:64] FLAG: --manifest-url=""
Nov 02 14:56:40.049031 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.040559    5304 flags.go:64] FLAG: --manifest-url-header=""
Nov 02 14:56:40.049031 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.040564    5304 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Nov 02 14:56:40.049031 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.040567    5304 flags.go:64] FLAG: --max-open-files="1000000"
Nov 02 14:56:40.049031 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.040571    5304 flags.go:64] FLAG: --max-pods="110"
Nov 02 14:56:40.049031 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.040574    5304 flags.go:64] FLAG: --maximum-dead-containers="-1"
Nov 02 14:56:40.049031 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.040577    5304 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Nov 02 14:56:40.049031 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.040580    5304 flags.go:64] FLAG: --memory-manager-policy="None"
Nov 02 14:56:40.049031 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.040583    5304 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Nov 02 14:56:40.049031 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.040586    5304 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Nov 02 14:56:40.049031 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.040591    5304 flags.go:64] FLAG: --node-ip="0.0.0.0"
Nov 02 14:56:40.049031 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.040594    5304 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Nov 02 14:56:40.049031 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.040602    5304 flags.go:64] FLAG: --node-status-max-images="50"
Nov 02 14:56:40.049031 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.040605    5304 flags.go:64] FLAG: --node-status-update-frequency="10s"
Nov 02 14:56:40.049031 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.040608    5304 flags.go:64] FLAG: --oom-score-adj="-999"
Nov 02 14:56:40.049031 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.040611    5304 flags.go:64] FLAG: --pod-cidr=""
Nov 02 14:56:40.049031 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.040615    5304 flags.go:64] FLAG: --pod-manifest-path=""
Nov 02 14:56:40.049031 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.040618    5304 flags.go:64] FLAG: --pod-max-pids="-1"
Nov 02 14:56:40.049031 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.040621    5304 flags.go:64] FLAG: --pods-per-core="0"
Nov 02 14:56:40.049031 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.040624    5304 flags.go:64] FLAG: --port="10250"
Nov 02 14:56:40.049031 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.040627    5304 flags.go:64] FLAG: --protect-kernel-defaults="false"
Nov 02 14:56:40.049031 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.040630    5304 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0b10dc7a7e78e104b"
Nov 02 14:56:40.049031 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.040633    5304 flags.go:64] FLAG: --qos-reserved=""
Nov 02 14:56:40.049031 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.040636    5304 flags.go:64] FLAG: --read-only-port="10255"
Nov 02 14:56:40.049031 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.040639    5304 flags.go:64] FLAG: --register-node="true"
Nov 02 14:56:40.049031 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.040643    5304 flags.go:64] FLAG: --register-with-taints=""
Nov 02 14:56:40.049031 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.040647    5304 flags.go:64] FLAG: --registry-burst="10"
Nov 02 14:56:40.049698 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.040650    5304 flags.go:64] FLAG: --registry-qps="5"
Nov 02 14:56:40.049698 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.040653    5304 flags.go:64] FLAG: --reserved-cpus=""
Nov 02 14:56:40.049698 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.040656    5304 flags.go:64] FLAG: --reserved-memory=""
Nov 02 14:56:40.049698 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.040660    5304 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Nov 02 14:56:40.049698 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.040663    5304 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Nov 02 14:56:40.049698 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.040668    5304 flags.go:64] FLAG: --rotate-certificates="false"
Nov 02 14:56:40.049698 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.040671    5304 flags.go:64] FLAG: --rotate-server-certificates="false"
Nov 02 14:56:40.049698 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.040674    5304 flags.go:64] FLAG: --runonce="false"
Nov 02 14:56:40.049698 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.040677    5304 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Nov 02 14:56:40.049698 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.040681    5304 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Nov 02 14:56:40.049698 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.040684    5304 flags.go:64] FLAG: --seccomp-default="false"
Nov 02 14:56:40.049698 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.040687    5304 flags.go:64] FLAG: --serialize-image-pulls="true"
Nov 02 14:56:40.049698 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.040690    5304 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Nov 02 14:56:40.049698 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.040693    5304 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Nov 02 14:56:40.049698 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.040696    5304 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Nov 02 14:56:40.049698 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.040700    5304 flags.go:64] FLAG: --storage-driver-password="root"
Nov 02 14:56:40.049698 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.040704    5304 flags.go:64] FLAG: --storage-driver-secure="false"
Nov 02 14:56:40.049698 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.040707    5304 flags.go:64] FLAG: --storage-driver-table="stats"
Nov 02 14:56:40.049698 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.040710    5304 flags.go:64] FLAG: --storage-driver-user="root"
Nov 02 14:56:40.049698 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.040713    5304 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Nov 02 14:56:40.049698 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.040717    5304 flags.go:64] FLAG: --sync-frequency="1m0s"
Nov 02 14:56:40.049698 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.040720    5304 flags.go:64] FLAG: --system-cgroups=""
Nov 02 14:56:40.049698 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.040723    5304 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Nov 02 14:56:40.049698 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.040728    5304 flags.go:64] FLAG: --system-reserved-cgroup=""
Nov 02 14:56:40.049698 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.040731    5304 flags.go:64] FLAG: --tls-cert-file=""
Nov 02 14:56:40.050323 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.040734    5304 flags.go:64] FLAG: --tls-cipher-suites="[]"
Nov 02 14:56:40.050323 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.040738    5304 flags.go:64] FLAG: --tls-min-version=""
Nov 02 14:56:40.050323 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.040741    5304 flags.go:64] FLAG: --tls-private-key-file=""
Nov 02 14:56:40.050323 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.040744    5304 flags.go:64] FLAG: --topology-manager-policy="none"
Nov 02 14:56:40.050323 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.040747    5304 flags.go:64] FLAG: --topology-manager-policy-options=""
Nov 02 14:56:40.050323 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.040750    5304 flags.go:64] FLAG: --topology-manager-scope="container"
Nov 02 14:56:40.050323 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.040753    5304 flags.go:64] FLAG: --v="2"
Nov 02 14:56:40.050323 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.040757    5304 flags.go:64] FLAG: --version="false"
Nov 02 14:56:40.050323 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.040761    5304 flags.go:64] FLAG: --vmodule=""
Nov 02 14:56:40.050323 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.040765    5304 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Nov 02 14:56:40.050323 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.040769    5304 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Nov 02 14:56:40.050323 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.040925    5304 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 14:56:40.050323 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.040930    5304 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 14:56:40.050323 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.040933    5304 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 14:56:40.050323 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.040936    5304 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 14:56:40.050323 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.040939    5304 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 14:56:40.050323 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.040942    5304 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 14:56:40.050323 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.040944    5304 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 14:56:40.050323 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.040947    5304 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 14:56:40.050323 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.040950    5304 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 14:56:40.050323 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.040953    5304 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 14:56:40.050323 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.040956    5304 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 14:56:40.050895 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.040963    5304 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 14:56:40.050895 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.040966    5304 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 14:56:40.050895 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.040969    5304 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 14:56:40.050895 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.040972    5304 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 14:56:40.050895 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.040975    5304 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 14:56:40.050895 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.040978    5304 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 14:56:40.050895 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.040981    5304 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 14:56:40.050895 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.040984    5304 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 14:56:40.050895 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.040986    5304 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 14:56:40.050895 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.040989    5304 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 14:56:40.050895 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.040992    5304 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 14:56:40.050895 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.040994    5304 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 14:56:40.050895 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.040997    5304 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 14:56:40.050895 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.041000    5304 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 14:56:40.050895 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.041002    5304 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 14:56:40.050895 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.041005    5304 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 14:56:40.050895 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.041008    5304 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 14:56:40.050895 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.041010    5304 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 14:56:40.050895 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.041013    5304 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 14:56:40.050895 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.041015    5304 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 14:56:40.051407 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.041018    5304 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 14:56:40.051407 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.041021    5304 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 14:56:40.051407 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.041024    5304 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 14:56:40.051407 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.041027    5304 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 14:56:40.051407 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.041029    5304 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 14:56:40.051407 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.041032    5304 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 14:56:40.051407 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.041034    5304 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 14:56:40.051407 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.041037    5304 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 14:56:40.051407 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.041040    5304 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 14:56:40.051407 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.041042    5304 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 14:56:40.051407 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.041045    5304 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 14:56:40.051407 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.041047    5304 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 14:56:40.051407 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.041051    5304 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 14:56:40.051407 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.041054    5304 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 14:56:40.051407 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.041057    5304 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 14:56:40.051407 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.041060    5304 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 14:56:40.051407 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.041063    5304 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 14:56:40.051407 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.041071    5304 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 14:56:40.051407 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.041074    5304 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 14:56:40.051407 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.041077    5304 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 14:56:40.051981 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.041079    5304 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 14:56:40.051981 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.041082    5304 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 14:56:40.051981 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.041085    5304 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 14:56:40.051981 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.041087    5304 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 14:56:40.051981 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.041090    5304 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 14:56:40.051981 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.041093    5304 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 14:56:40.051981 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.041096    5304 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 14:56:40.051981 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.041099    5304 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 14:56:40.051981 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.041102    5304 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 14:56:40.051981 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.041105    5304 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 14:56:40.051981 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.041107    5304 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 14:56:40.051981 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.041110    5304 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 14:56:40.051981 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.041113    5304 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 14:56:40.051981 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.041116    5304 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 14:56:40.051981 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.041118    5304 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 14:56:40.051981 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.041134    5304 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 14:56:40.051981 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.041137    5304 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 14:56:40.051981 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.041140    5304 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 14:56:40.051981 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.041142    5304 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 14:56:40.051981 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.041145    5304 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 14:56:40.052588 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.041148    5304 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 14:56:40.052588 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.041150    5304 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 14:56:40.052588 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.041153    5304 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 14:56:40.052588 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.041156    5304 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 14:56:40.052588 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.041160    5304 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 14:56:40.052588 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.041164    5304 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 14:56:40.052588 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.041167    5304 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 14:56:40.052588 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.041171    5304 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 14:56:40.052588 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.041173    5304 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 14:56:40.052588 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.041176    5304 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 14:56:40.052588 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.041180    5304 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 14:56:40.052588 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.041183    5304 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 14:56:40.052588 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.041185    5304 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 14:56:40.052588 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.041188    5304 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 14:56:40.052588 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.041190    5304 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 14:56:40.052588 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.041193    5304 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 14:56:40.052588 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.041196    5304 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 14:56:40.052588 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.041198    5304 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 14:56:40.052588 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.041201    5304 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 14:56:40.052588 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.041204    5304 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 14:56:40.053273 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.041207    5304 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 14:56:40.053273 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.041209    5304 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 14:56:40.053273 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.041212    5304 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 14:56:40.053273 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.041214    5304 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 14:56:40.053273 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.041217    5304 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 14:56:40.053273 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.041221    5304 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 14:56:40.053273 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.041224    5304 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 14:56:40.053273 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.041227    5304 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 14:56:40.053273 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.041231    5304 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 14:56:40.053273 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.041233    5304 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 14:56:40.053273 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.041236    5304 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 14:56:40.053273 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.041239    5304 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 14:56:40.053273 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.041241    5304 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 14:56:40.053273 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.041244    5304 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 14:56:40.053273 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.041246    5304 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 14:56:40.053692 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.041273    5304 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 02 14:56:40.053692 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.041312    5304 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 14:56:40.053692 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.041317    5304 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 14:56:40.053692 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.041320    5304 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 14:56:40.053692 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.041323    5304 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 14:56:40.053692 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.041326    5304 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 14:56:40.053692 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.041328    5304 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 14:56:40.053692 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.041331    5304 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 14:56:40.053692 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.041334    5304 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 14:56:40.053692 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.041337    5304 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 14:56:40.053692 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.041340    5304 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 14:56:40.053692 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.041343    5304 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 14:56:40.053692 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.041345    5304 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 14:56:40.053692 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.041348    5304 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 14:56:40.053692 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.041351    5304 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 14:56:40.054303 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.041353    5304 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 14:56:40.054303 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.041356    5304 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 14:56:40.054303 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.041359    5304 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 14:56:40.054303 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.041362    5304 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 14:56:40.054303 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.041364    5304 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 14:56:40.054303 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.041367    5304 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 14:56:40.054303 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.041370    5304 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 14:56:40.054303 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.041372    5304 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 14:56:40.054303 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.041375    5304 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 14:56:40.054303 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.041377    5304 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 14:56:40.054303 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.041380    5304 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 14:56:40.054303 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.041382    5304 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 14:56:40.054303 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.041385    5304 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 14:56:40.054303 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.041387    5304 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 14:56:40.054303 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.041390    5304 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 14:56:40.054303 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.041392    5304 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 14:56:40.054303 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.041395    5304 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 14:56:40.054303 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.041397    5304 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 14:56:40.054303 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.041402    5304 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 14:56:40.054303 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.041406    5304 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 14:56:40.054910 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.041409    5304 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 14:56:40.054910 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.041412    5304 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 14:56:40.054910 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.041415    5304 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 14:56:40.054910 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.041418    5304 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 14:56:40.054910 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.041420    5304 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 14:56:40.054910 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.041424    5304 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 14:56:40.054910 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.041428    5304 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 14:56:40.054910 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.041430    5304 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 14:56:40.054910 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.041433    5304 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 14:56:40.054910 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.041436    5304 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 14:56:40.054910 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.041438    5304 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 14:56:40.054910 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.041441    5304 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 14:56:40.054910 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.041444    5304 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 14:56:40.054910 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.041447    5304 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 14:56:40.054910 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.041450    5304 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 14:56:40.054910 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.041452    5304 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 14:56:40.054910 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.041455    5304 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 14:56:40.054910 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.041458    5304 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 14:56:40.054910 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.041463    5304 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 14:56:40.055396 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.041466    5304 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 14:56:40.055396 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.041469    5304 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 14:56:40.055396 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.041472    5304 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 14:56:40.055396 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.041474    5304 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 14:56:40.055396 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.041477    5304 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 14:56:40.055396 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.041480    5304 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 14:56:40.055396 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.041482    5304 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 14:56:40.055396 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.041485    5304 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 14:56:40.055396 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.041488    5304 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 14:56:40.055396 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.041490    5304 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 14:56:40.055396 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.041493    5304 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 14:56:40.055396 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.041496    5304 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 14:56:40.055396 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.041499    5304 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 14:56:40.055396 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.041501    5304 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 14:56:40.055396 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.041504    5304 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 14:56:40.055396 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.041507    5304 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 14:56:40.055396 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.041509    5304 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 14:56:40.055396 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.041512    5304 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 14:56:40.055396 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.041516    5304 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 14:56:40.055923 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.041520    5304 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 14:56:40.055923 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.041523    5304 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 14:56:40.055923 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.041526    5304 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 14:56:40.055923 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.041528    5304 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 14:56:40.055923 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.041531    5304 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 14:56:40.055923 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.041533    5304 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 14:56:40.055923 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.041536    5304 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 14:56:40.055923 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.041539    5304 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 14:56:40.055923 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.041541    5304 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 14:56:40.055923 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.041544    5304 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 14:56:40.055923 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.041546    5304 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 14:56:40.055923 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.041549    5304 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 14:56:40.055923 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.041552    5304 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 14:56:40.055923 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.041554    5304 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 14:56:40.055923 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.041557    5304 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 14:56:40.055923 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.041559    5304 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 14:56:40.055923 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.041562    5304 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 14:56:40.055923 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.041564    5304 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 14:56:40.055923 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.041567    5304 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 14:56:40.055923 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.041569    5304 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 14:56:40.056594 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.041572    5304 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 14:56:40.056594 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.041575    5304 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 14:56:40.056594 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.041577    5304 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 14:56:40.056594 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.041579    5304 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 14:56:40.056594 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.041584    5304 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 14:56:40.056594 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.041587    5304 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 14:56:40.056594 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.041589    5304 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 14:56:40.056594 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.041592    5304 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 14:56:40.056594 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.041594    5304 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 14:56:40.056594 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.041597    5304 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 14:56:40.056594 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.041599    5304 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 14:56:40.056594 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.041603    5304 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 14:56:40.056594 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.041606    5304 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 14:56:40.056594 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.041609    5304 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 14:56:40.056594 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.045933    5304 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Nov 02 14:56:40.056594 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.045950    5304 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Nov 02 14:56:40.056594 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.045969    5304 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Nov 02 14:56:40.056594 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.045973    5304 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Nov 02 14:56:40.056594 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.046050    5304 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 14:56:40.056594 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.046055    5304 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 14:56:40.057145 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.046058    5304 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 14:56:40.057145 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.046061    5304 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 14:56:40.057145 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.046064    5304 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 14:56:40.057145 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.046067    5304 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 14:56:40.057145 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.046070    5304 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 14:56:40.057145 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.046073    5304 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 14:56:40.057145 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.046076    5304 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 14:56:40.057145 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.046079    5304 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 14:56:40.057145 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.046082    5304 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 14:56:40.057145 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.046084    5304 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 14:56:40.057145 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.046087    5304 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 14:56:40.057145 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.046090    5304 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 14:56:40.057145 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.046093    5304 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 14:56:40.057145 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.046096    5304 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 14:56:40.057145 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.046099    5304 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 14:56:40.057145 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.046102    5304 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 14:56:40.057145 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.046107    5304 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 14:56:40.057145 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.046111    5304 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 14:56:40.057145 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.046114    5304 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 14:56:40.057661 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.046117    5304 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 14:56:40.057661 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.046120    5304 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 14:56:40.057661 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.046137    5304 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 14:56:40.057661 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.046141    5304 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 14:56:40.057661 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.046143    5304 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 14:56:40.057661 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.046153    5304 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 14:56:40.057661 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.046156    5304 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 14:56:40.057661 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.046166    5304 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 14:56:40.057661 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.046169    5304 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 14:56:40.057661 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.046172    5304 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 14:56:40.057661 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.046174    5304 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 14:56:40.057661 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.046177    5304 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 14:56:40.057661 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.046180    5304 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 14:56:40.057661 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.046183    5304 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 14:56:40.057661 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.046186    5304 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 14:56:40.057661 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.046188    5304 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 14:56:40.057661 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.046191    5304 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 14:56:40.057661 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.046194    5304 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 14:56:40.057661 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.046196    5304 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 14:56:40.058150 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.046199    5304 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 14:56:40.058150 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.046202    5304 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 14:56:40.058150 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.046204    5304 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 14:56:40.058150 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.046207    5304 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 14:56:40.058150 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.046210    5304 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 14:56:40.058150 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.046212    5304 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 14:56:40.058150 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.046215    5304 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 14:56:40.058150 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.046218    5304 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 14:56:40.058150 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.046221    5304 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 14:56:40.058150 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.046223    5304 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 14:56:40.058150 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.046227    5304 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 14:56:40.058150 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.046229    5304 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 14:56:40.058150 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.046232    5304 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 14:56:40.058150 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.046235    5304 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 14:56:40.058150 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.046238    5304 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 14:56:40.058150 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.046241    5304 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 14:56:40.058150 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.046244    5304 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 14:56:40.058150 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.046246    5304 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 14:56:40.058150 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.046249    5304 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 14:56:40.058150 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.046252    5304 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 14:56:40.058649 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.046254    5304 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 14:56:40.058649 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.046265    5304 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 14:56:40.058649 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.046268    5304 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 14:56:40.058649 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.046271    5304 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 14:56:40.058649 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.046274    5304 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 14:56:40.058649 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.046276    5304 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 14:56:40.058649 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.046279    5304 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 14:56:40.058649 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.046282    5304 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 14:56:40.058649 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.046284    5304 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 14:56:40.058649 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.046287    5304 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 14:56:40.058649 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.046289    5304 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 14:56:40.058649 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.046292    5304 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 14:56:40.058649 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.046295    5304 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 14:56:40.058649 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.046297    5304 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 14:56:40.058649 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.046300    5304 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 14:56:40.058649 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.046303    5304 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 14:56:40.058649 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.046305    5304 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 14:56:40.058649 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.046308    5304 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 14:56:40.058649 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.046311    5304 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 14:56:40.059176 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.046313    5304 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 14:56:40.059176 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.046316    5304 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 14:56:40.059176 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.046319    5304 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 14:56:40.059176 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.046321    5304 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 14:56:40.059176 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.046326    5304 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 14:56:40.059176 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.046329    5304 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 14:56:40.059176 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.046333    5304 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 14:56:40.059176 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.046336    5304 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 14:56:40.059176 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.046339    5304 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 14:56:40.059176 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.046342    5304 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 14:56:40.059176 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.046345    5304 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 14:56:40.059176 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.046347    5304 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 14:56:40.059176 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.046350    5304 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 14:56:40.059176 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.046353    5304 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 14:56:40.059176 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.046355    5304 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 14:56:40.059176 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.046364    5304 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 14:56:40.059176 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.046367    5304 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 14:56:40.059176 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.046370    5304 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 14:56:40.059176 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.046373    5304 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 14:56:40.059176 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.046375    5304 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 14:56:40.059671 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.046378    5304 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 14:56:40.059671 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.046381    5304 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 14:56:40.059671 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.046384    5304 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 14:56:40.059671 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.046386    5304 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 14:56:40.059671 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.046389    5304 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 14:56:40.059671 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.046392    5304 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 14:56:40.059671 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.046395    5304 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 14:56:40.059671 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.046423    5304 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 02 14:56:40.059671 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.046635    5304 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 14:56:40.059671 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.046641    5304 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 14:56:40.059671 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.046645    5304 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 14:56:40.059671 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.046647    5304 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 14:56:40.059671 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.046651    5304 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 14:56:40.059671 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.046654    5304 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 14:56:40.059671 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.046656    5304 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 14:56:40.059671 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.046659    5304 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 14:56:40.060066 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.046662    5304 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 14:56:40.060066 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.046665    5304 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 14:56:40.060066 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.046667    5304 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 14:56:40.060066 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.046670    5304 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 14:56:40.060066 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.046673    5304 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 14:56:40.060066 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.046675    5304 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 14:56:40.060066 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.046678    5304 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 14:56:40.060066 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.046680    5304 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 14:56:40.060066 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.046683    5304 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 14:56:40.060066 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.046686    5304 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 14:56:40.060066 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.046688    5304 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 14:56:40.060066 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.046691    5304 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 14:56:40.060066 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.046702    5304 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 14:56:40.060066 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.046705    5304 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 14:56:40.060066 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.046707    5304 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 14:56:40.060066 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.046710    5304 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 14:56:40.060066 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.046713    5304 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 14:56:40.060066 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.046715    5304 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 14:56:40.060066 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.046718    5304 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 14:56:40.060066 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.046721    5304 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 14:56:40.060765 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.046723    5304 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 14:56:40.060765 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.046726    5304 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 14:56:40.060765 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.046729    5304 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 14:56:40.060765 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.046732    5304 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 14:56:40.060765 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.046736    5304 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 14:56:40.060765 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.046740    5304 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 14:56:40.060765 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.046743    5304 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 14:56:40.060765 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.046746    5304 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 14:56:40.060765 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.046749    5304 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 14:56:40.060765 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.046752    5304 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 14:56:40.060765 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.046754    5304 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 14:56:40.060765 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.046757    5304 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 14:56:40.060765 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.046760    5304 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 14:56:40.060765 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.046762    5304 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 14:56:40.060765 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.046765    5304 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 14:56:40.060765 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.046768    5304 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 14:56:40.060765 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.046770    5304 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 14:56:40.060765 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.046773    5304 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 14:56:40.060765 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.046775    5304 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 14:56:40.061251 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.046778    5304 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 14:56:40.061251 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.046780    5304 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 14:56:40.061251 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.046783    5304 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 14:56:40.061251 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.046786    5304 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 14:56:40.061251 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.046788    5304 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 14:56:40.061251 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.046791    5304 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 14:56:40.061251 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.046799    5304 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 14:56:40.061251 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.046802    5304 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 14:56:40.061251 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.046805    5304 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 14:56:40.061251 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.046808    5304 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 14:56:40.061251 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.046810    5304 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 14:56:40.061251 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.046813    5304 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 14:56:40.061251 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.046820    5304 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 14:56:40.061251 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.046823    5304 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 14:56:40.061251 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.046826    5304 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 14:56:40.061251 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.046828    5304 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 14:56:40.061251 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.046831    5304 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 14:56:40.061251 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.046834    5304 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 14:56:40.061251 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.046836    5304 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 14:56:40.061709 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.046839    5304 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 14:56:40.061709 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.046841    5304 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 14:56:40.061709 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.046844    5304 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 14:56:40.061709 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.046847    5304 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 14:56:40.061709 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.046849    5304 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 14:56:40.061709 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.046853    5304 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 14:56:40.061709 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.046857    5304 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 14:56:40.061709 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.046860    5304 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 14:56:40.061709 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.046863    5304 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 14:56:40.061709 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.046866    5304 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 14:56:40.061709 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.046869    5304 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 14:56:40.061709 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.046872    5304 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 14:56:40.061709 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.046874    5304 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 14:56:40.061709 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.046877    5304 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 14:56:40.061709 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.046879    5304 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 14:56:40.061709 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.046882    5304 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 14:56:40.061709 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.046885    5304 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 14:56:40.061709 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.046888    5304 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 14:56:40.061709 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.046891    5304 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 14:56:40.062175 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.046893    5304 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 14:56:40.062175 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.046902    5304 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 14:56:40.062175 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.046905    5304 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 14:56:40.062175 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.046908    5304 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 14:56:40.062175 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.046910    5304 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 14:56:40.062175 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.046913    5304 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 14:56:40.062175 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.046917    5304 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 14:56:40.062175 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.046920    5304 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 14:56:40.062175 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.046925    5304 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 14:56:40.062175 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.046928    5304 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 14:56:40.062175 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.046931    5304 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 14:56:40.062175 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.046933    5304 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 14:56:40.062175 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.046936    5304 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 14:56:40.062175 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.046939    5304 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 14:56:40.062175 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.046942    5304 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 14:56:40.062175 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.046944    5304 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 14:56:40.062175 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.046947    5304 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 14:56:40.062175 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.046950    5304 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 14:56:40.062175 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.046952    5304 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 14:56:40.062175 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.046955    5304 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 14:56:40.062667 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.046958    5304 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 14:56:40.062667 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.046980    5304 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 02 14:56:40.062667 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.047029    5304 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 14:56:40.062667 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.047033    5304 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 14:56:40.062667 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.047036    5304 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 14:56:40.062667 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.047040    5304 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 14:56:40.062667 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.047043    5304 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 14:56:40.062667 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.047046    5304 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 14:56:40.062667 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.047049    5304 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 14:56:40.062667 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.047052    5304 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 14:56:40.062667 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.047054    5304 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 14:56:40.062667 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.047057    5304 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 14:56:40.062667 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.047060    5304 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 14:56:40.062667 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.047063    5304 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 14:56:40.062667 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.047077    5304 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 14:56:40.063036 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.047080    5304 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 14:56:40.063036 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.047083    5304 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 14:56:40.063036 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.047086    5304 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 14:56:40.063036 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.047088    5304 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 14:56:40.063036 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.047093    5304 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 14:56:40.063036 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.047098    5304 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 14:56:40.063036 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.047100    5304 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 14:56:40.063036 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.047103    5304 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 14:56:40.063036 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.047106    5304 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 14:56:40.063036 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.047109    5304 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 14:56:40.063036 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.047111    5304 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 14:56:40.063036 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.047114    5304 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 14:56:40.063036 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.047116    5304 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 14:56:40.063036 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.047119    5304 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 14:56:40.063036 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.047141    5304 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 14:56:40.063036 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.047145    5304 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 14:56:40.063036 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.047147    5304 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 14:56:40.063036 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.047150    5304 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 14:56:40.063036 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.047153    5304 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 14:56:40.063525 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.047156    5304 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 14:56:40.063525 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.047159    5304 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 14:56:40.063525 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.047161    5304 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 14:56:40.063525 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.047164    5304 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 14:56:40.063525 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.047369    5304 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 14:56:40.063525 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.047422    5304 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 14:56:40.063525 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.047426    5304 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 14:56:40.063525 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.047429    5304 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 14:56:40.063525 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.047433    5304 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 14:56:40.063525 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.047437    5304 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 14:56:40.063525 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.047444    5304 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 14:56:40.063525 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.047450    5304 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 14:56:40.063525 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.047455    5304 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 14:56:40.063525 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.047510    5304 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 14:56:40.063525 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.047525    5304 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 14:56:40.063525 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.047543    5304 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 14:56:40.063525 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.047552    5304 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 14:56:40.063525 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.047560    5304 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 14:56:40.063525 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.047573    5304 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 14:56:40.063992 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.047581    5304 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 14:56:40.063992 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.047589    5304 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 14:56:40.063992 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.047596    5304 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 14:56:40.063992 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.047604    5304 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 14:56:40.063992 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.047611    5304 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 14:56:40.063992 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.047618    5304 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 14:56:40.063992 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.047626    5304 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 14:56:40.063992 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.047634    5304 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 14:56:40.063992 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.047642    5304 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 14:56:40.063992 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.047653    5304 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 14:56:40.063992 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.047661    5304 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 14:56:40.063992 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.047668    5304 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 14:56:40.063992 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.047676    5304 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 14:56:40.063992 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.047684    5304 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 14:56:40.063992 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.047691    5304 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 14:56:40.063992 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.047699    5304 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 14:56:40.063992 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.047707    5304 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 14:56:40.063992 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.047715    5304 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 14:56:40.063992 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.047722    5304 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 14:56:40.064503 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.047730    5304 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 14:56:40.064503 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.047737    5304 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 14:56:40.064503 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.047745    5304 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 14:56:40.064503 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.047759    5304 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 14:56:40.064503 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.047767    5304 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 14:56:40.064503 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.047776    5304 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 14:56:40.064503 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.047784    5304 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 14:56:40.064503 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.047792    5304 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 14:56:40.064503 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.047833    5304 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 14:56:40.064503 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.047842    5304 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 14:56:40.064503 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.047850    5304 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 14:56:40.064503 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.047858    5304 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 14:56:40.064503 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.047870    5304 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 14:56:40.064503 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.047878    5304 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 14:56:40.064503 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.047888    5304 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 14:56:40.064503 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.047898    5304 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 14:56:40.064503 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.047912    5304 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 14:56:40.064503 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.047920    5304 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 14:56:40.064503 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.047928    5304 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 14:56:40.064996 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.047937    5304 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 14:56:40.064996 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.047945    5304 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 14:56:40.064996 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.047951    5304 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 14:56:40.064996 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.047960    5304 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 14:56:40.064996 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.047968    5304 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 14:56:40.064996 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.047976    5304 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 14:56:40.064996 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.047985    5304 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 14:56:40.064996 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.047992    5304 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 14:56:40.064996 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.048006    5304 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 14:56:40.064996 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.048012    5304 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 14:56:40.064996 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.048016    5304 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 14:56:40.064996 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.048021    5304 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 14:56:40.064996 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.048026    5304 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 14:56:40.064996 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.048031    5304 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 14:56:40.064996 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.048036    5304 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 14:56:40.064996 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.048040    5304 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 14:56:40.064996 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:56:40.048045    5304 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 14:56:40.064996 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.048202    5304 server.go:957] "Client rotation is on, will bootstrap in background"
Nov 02 14:56:40.064996 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.049547    5304 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary"
Nov 02 14:56:40.064996 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.049605    5304 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem"
Nov 02 14:56:40.065492 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.049793    5304 server.go:1014] "Starting client certificate rotation"
Nov 02 14:56:40.065492 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.049928    5304 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 02 14:56:40.065492 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.049982    5304 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-01 14:50:29 +0000 UTC" deadline="2028-04-20 01:23:36.080874513 +0000 UTC"
Nov 02 14:56:40.065492 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.050011    5304 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12826h26m56.030866035s"
Nov 02 14:56:40.065492 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.050257    5304 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 02 14:56:40.065492 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.052611    5304 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 02 14:56:40.065492 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.054052    5304 log.go:25] "Validated CRI v1 runtime API"
Nov 02 14:56:40.065492 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.055846    5304 log.go:25] "Validated CRI v1 image API"
Nov 02 14:56:40.065492 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.056635    5304 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Nov 02 14:56:40.065492 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.058327    5304 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 b930873d-65fe-4796-a85b-b436842c6376:/dev/nvme0n1p3 cfc27493-9538-44d8-99e4-7a391d34725d:/dev/nvme0n1p4]
Nov 02 14:56:40.065765 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.058344    5304 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/03b691e65cf37f7a94beeee735d801b8dc31a3125b18747c22f89e827fd248ba/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/03b691e65cf37f7a94beeee735d801b8dc31a3125b18747c22f89e827fd248ba/userdata/shm major:0 minor:226 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0aae85bb0b0b3446e320e80d65728fda1f1ac14b0af5de6c99416d8bde25db9d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0aae85bb0b0b3446e320e80d65728fda1f1ac14b0af5de6c99416d8bde25db9d/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2315db45e98cf6c8210934b607b07cfcaf32866bfffd173ccb4d1c2baae1849c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2315db45e98cf6c8210934b607b07cfcaf32866bfffd173ccb4d1c2baae1849c/userdata/shm major:0 minor:95 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2dffcc175ba039b64912a830e8787c67872dcb2ea3a2d115ce4d37a7558454e3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2dffcc175ba039b64912a830e8787c67872dcb2ea3a2d115ce4d37a7558454e3/userdata/shm major:0 minor:202 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/40258544b5fc72c08d5050ca375b814933292f7409af909ee8add2ebafb2e17f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/40258544b5fc72c08d5050ca375b814933292f7409af909ee8add2ebafb2e17f/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6917bc2965fdd531105d8ed16d2dea52c30702bfe6a4fd41de68d832fb3dff14/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6917bc2965fdd531105d8ed16d2dea52c30702bfe6a4fd41de68d832fb3dff14/userdata/shm major:0 minor:276 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/777cd23a55bfad60d07f0c9cc7b69832952228d10ba770ce6b855beb1aa7fb9f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/777cd23a55bfad60d07f0c9cc7b69832952228d10ba770ce6b855beb1aa7fb9f/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/979ce81d18065aab8e148d5e90f911903f8277633576644f10bbfa9486b6cadc/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/979ce81d18065aab8e148d5e90f911903f8277633576644f10bbfa9486b6cadc/userdata/shm major:0 minor:224 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9881d50d11ec0a6732c1726cbb74624ab2bdebf9c73f8b5222237dafe496e451/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9881d50d11ec0a6732c1726cbb74624ab2bdebf9c73f8b5222237dafe496e451/userdata/shm major:0 minor:222 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9d80c77407d3e67f6198e00be9d36a6e7f9c402b652e9d03c548331917a7ac4c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9d80c77407d3e67f6198e00be9d36a6e7f9c402b652e9d03c548331917a7ac4c/userdata/shm major:0 minor:48 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b236aedf11348eb6319968e96631ba9fb50c69aafadf3aa5a5c86ad633dd1234/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b236aedf11348eb6319968e96631ba9fb50c69aafadf3aa5a5c86ad633dd1234/userdata/shm major:0 minor:269 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b7ab68c4d9b36e6507e0795c8bb246a797d2758dfa1a5dc067d56ef0ef5b459d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b7ab68c4d9b36e6507e0795c8bb246a797d2758dfa1a5dc067d56ef0ef5b459d/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/bf709fa05cba77da3f9435c58038e48838b3bfc405abff55963f4d883f2a19ba/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/bf709fa05cba77da3f9435c58038e48838b3bfc405abff55963f4d883f2a19ba/userdata/shm major:0 minor:86 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d1f7259e798f4c778135523c563af014d044be9176e87cba4459ed3b40ffab7d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d1f7259e798f4c778135523c563af014d044be9176e87cba4459ed3b40ffab7d/userdata/shm major:0 minor:82 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d5d6f959273f30a64b18a421bf431362ed4babd9fee6bef83d84d8bf6665d58d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d5d6f959273f30a64b18a421bf431362ed4babd9fee6bef83d84d8bf6665d58d/userdata/shm major:0 minor:270 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/dedf3babecad2f468747d3e72210ea54c17d3ccf467e6671f136263e8ad8c84c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/dedf3babecad2f468747d3e72210ea54c17d3ccf467e6671f136263e8ad8c84c/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e13c510d9ba4b82e51fb2f10c06b32bbf9c8e5ebf6225c60f9db08c58e4c2b4c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e13c510d9ba4b82e51fb2f10c06b32bbf9c8e5ebf6225c60f9db08c58e4c2b4c/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/03b3012d-0bdf-4a78-ad31-c03d1df51ec0/volumes/kubernetes.io~projected/kube-api-access-9vxjj:{mountpoint:/var/lib/kubelet/pods/03b3012d-0bdf-4a78-ad31-c03d1df51ec0/volumes/kubernetes.io~projected/kube-api-access-9vxjj major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/03b3012d-0bdf-4a78-ad31-c03d1df51ec0/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/03b3012d-0bdf-4a78-ad31-c03d1df51ec0/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/03b3012d-0bdf-4a78-ad31-c03d1df51ec0/volumes/kubernetes.io~secret/node-exporter-tls:{mountpoint:/var/lib/kubelet/pods/03b3012d-0bdf-4a78-ad31-c03d1df51ec0/volumes/kubernetes.io~secret/node-exporter-tls major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/080a338e-be02-49fb-bebe-f873e3f4017e/volumes/kubernetes.io~projected/kube-api-access-r67xl:{mountpoint:/var/lib/kubelet/pods/080a338e-be02-49fb-bebe-f873e3f4017e/volumes/kubernetes.io~projected/kube-api-access-r67xl major:0 minor:219 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/080a338e-be02-49fb-bebe-f873e3f4017e/volumes/kubernetes.io~secret/metrics-tls:{mountpoint:/var/lib/kubelet/pods/080a338e-be02-49fb-bebe-f873e3f4017e/volumes/kubernetes.io~secret/metrics-tls major:0 minor:218 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0d6c5872-e278-4463-953d-e1388a001d0d/volumes/kubernetes.io~projected/kube-api-access-hpfk5:{mountpoint:/var/lib/kubelet/pods/0d6c5872-e278-4463-953d-e1388a001d0d/volumes/kubernetes.io~projected/kube-api-access-hpfk5 major:0 minor:273 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/332f4a2d-a241-4c71-8ebb-a99618c0d3dc/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/332f4a2d-a241-4c71-8ebb-a99618c0d3dc/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:268 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/54dc6898-f041-4ed6-9fc9-1b5159d00d27/volumes/kubernetes.io~projected/kube-api-access-8bt8x:{mountpoint:/var/lib/kubelet/pods/54dc6898-f041-4ed6-9fc9-1b5159d00d27/volumes/kubernetes.io~projected/kube-api-access-8bt8x major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/65ce39fc-05c7-433b-81f4-c1b219f1bd71/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/65ce39fc-05c7-433b-81f4-c1b219f1bd71/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/65ce39fc-05c7-433b-81f4-c1b219f1bd71/volumes/kubernetes.io~projected/kube-api-access-ps49c:{mountpoint:/var/lib/kubelet/pods/65ce39fc-05c7-433b-81f4-c1b219f1bd71/volumes/kubernetes.io~projected/kube-api-access-ps49c major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/65ce39fc-05c7-433b-81f4-c1b219f1bd71/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/65ce39fc-05c7-433b-81f4-c1b219f1bd71/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/68b781b2-be96-4c5c-8daa-2238fd0a9623/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/68b781b2-be96-4c5c-8daa-2238fd0a9623/volumes/kubernetes.io~secret/agent-certs major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/702f7864-7db1-40d5-ba62-33cb9477b7da/volumes/kubernetes.io~projected/kube-api-access-kw44n:{mountpoint:/var/lib/kubelet/pods/702f7864-7db1-40d5-ba62-33cb9477b7da/volumes/kubernetes.io~projected/kube-api-access-kw44n major:0 minor:220 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/702f7864-7db1-40d5-ba62-33cb9477b7da/volumes/kubernetes.io~secret/insights-runtime-extractor-tls:{mountpoint:/var/lib/kubelet/pods/702f7864-7db1-40d5-ba62-33cb9477b7da/volumes/kubernetes.io~secret/insights-runtime-extractor-tls major:0 minor:217 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/720ef290-024b-4ed2-8417-92290f360dc9/volumes/kubernetes.io~projected/kube-api-access-f28t5:{mountpoint:/var/lib/kubelet/pods/720ef290-024b-4ed2-8417-92290f360dc9/volumes/kubernetes.io~projected/kube-api-access-f28t5 major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7fe9ba5e-a867-4ac3-8a6f-a316c6f64db4/volumes/kubernetes.io~projected/kube-api-access-8rqfc:{mountpoint:/var/lib/kubelet/pods/7fe9ba5e-a867-4ac3-8a6f-a316c6f64db4/volumes/kubernetes.io~projected/kube-api-access-8rqfc major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/93d7d50e-b320-4217-b183-2bfc921f6c9d/volumes/kubernetes.io~projected/kube-api-access-ndjwt:{mountpoint:/var/lib/kubelet/pods/93d7d50e-b320-4217-b183-2bfc921f6c9d/volumes/kubernetes.io~projected/kube-api-access-ndjwt major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b1715623-afd0-4d0e-9b63-b38499cec1e2/volumes/kubernetes.io~projected/kube-api-access-548bv:{mountpoint:/var/lib/kubelet/pods/b1715623-afd0-4d0e-9b63-b38499cec1e2/volumes/kubernetes.io~projected/kube-api-access-548bv major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/caa8898b-6611-49b8-845d-3b9bdac6691f/volumes/kubernetes.io~projected/kube-api-access-hz5t9:{mountpoint:/var/lib/kubelet/pods/caa8898b-6611-49b8-845d-3b9bdac6691f/volumes/kubernetes.io~projected/kube-api-access-hz5t9 major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/caa8898b-6611-49b8-845d-3b9bdac6691f/volumes/kubernetes.io~secret/metrics-certs:{mountpoint:/var/lib/kubelet/pods/caa8898b-6611-49b8-845d-3b9bdac6691f/volumes/kubernetes.io~secret/metrics-certs major:0 minor:263 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e1da99c3-8294-4e34-9d33-47e04a6182d3/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/e1da99c3-8294-4e34-9d33-47e04a6182d3/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e1da99c3-8294-4e34-9d33-47e04a6182d3/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/e1da99c3-8294-4e34-9d33-47e04a6182d3/volumes/kubernetes.io~empty-dir/tmp major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e1da99c3-8294-4e34-9d33-47e04a6182d3/volumes/kubernetes.io~projected/kube-api-access-prmck:{mountpoint:/var/lib/kubelet/pods/e1da99c3-8294-4e34-9d33-47e04a6182d3/volumes/kubernetes.io~projected/kube-api-access-prmck major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e4983b70-40f4-4e1e-a5da-80194d5d4786/volumes/kubernetes.io~projected/kube-api-access-7758v:{mountpoint:/var/lib/kubelet/pods/e4983b70-40f4-4e1e-a5da-80194d5d4786/volumes/kubernetes.io~projected/kube-api-access-7758v major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e4f9e55c-3232-430c-a123-30d17704cd2e/volumes/kubernetes.io~projected/kube-api-access-ghkdw:{mountpoint:/var/lib/kubelet/pods/e4f9e55c-3232-430c-a123-30d17704cd2e/volumes/kubernetes.io~projected/kube-api-access-ghkdw major:0 minor:221 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e4f9e55c-3232-430c-a123-30d17704cd2e/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/e4f9e55c-3232-430c-a123-30d17704cd2e/volumes/kubernetes.io~secret/cert major:0 minor:213 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/1bf491ce1fde3e90e47e02294acaa3f4ae9d211da7f9bbe89cc0f491158e23b5/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/d688178cd78d8a9fbbc0a3e1107318338573d8fa54de2f413fc944f712538667/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/a0b80285daae9df7a06c8d7ba675b932ead493c7dbba5b6baaf0ff4681271400/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-108:{mountpoint:/var/lib/containers/storage/overlay/4e94aa43970f5129c47e20a355e1c1cae3bc3b364f38b57a4e79158dbe5a6bca/merged major:0 minor:108 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/786333424a716e9064cedc11475ac211be0f2b8bd81ddcf3a243a9b8d9015ae6/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/2512dd143cec82295849ab25669b024198167124f39ed9b771945569344d1c9b/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/7e4a5442b3af04c1f490037ac07a1c2989a627ebd9ae01793f29c188e0eb13bf/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/bfbdf0b6b17cf2819ad495e4a2bc63e3d6a49a21311ab92ffbdc8834b2ac0371/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/fbabe794bac01ada5f1604519d3257d40fa41d59230b9ce5338d5f56c696ac88/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-133:{mountpoint:/var/lib/containers/storage/overlay/27004f514f9ca3e25575313b8db412b57ceec7264dd2a6775c113e76b306e4ec/merged major:0 minor:133 fsType:overlay blockSize:0} overlay_0-139:{mountpoint:/var/lib/containers/storage/overlay/ba2c04a366a821097e807eadb178c45ee6bccc75f06fb6f7cb86c921a58fcf86/merged major:0 minor:139 fsType:overlay blockSize:0} overlay_0-141:{mountpoint:/var/lib/containers/storage/overlay/7c98f7d4721db378d5ea85f8335a709f5548302eff18832b6a94c8c472a71bc8/merged major:0 minor:141 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/21d7fdda311301c35815978043ee4b6b4c0da87f20a449ffc706a767915f86c6/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/5823c012aff944dfa634df8bc2332fdc04f59c4076bd718e727ddb65f3fc5f51/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/ee2546b9dde4219c597aaa1506cc72b6f504f76db2ed6cdf5405cbedbcae8cbf/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/24132168f23d654c361ce39e1ed7fc6f91bb8ad91ba7577e1d5e9553af028782/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/d7ad07e87cd055db3c661f74f80d295f8cfe208229294cb5e23007cfb28c1aff/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/c0e5807dce84d8fa01f5e4b5cfbbc8a57fc1ad5da0293b6558214d52e71015ce/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/4ca92ee00b90ec1b9299e6c4460c20a14114fbe3e52e4399db14fd4f4ee3f496/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/1dff1eec8034d8399baf5780d0467999934f17217aa4540b47d7501c52dd8256/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/69d49f54e14691a6b2afd7a868764eb00f455de1f8e297992a7cb84a1a6e92e5/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-204:{mountpoint:/var/lib/containers/storage/overlay/1c2164cb9baa0d0f521ef34a73261d9e888735f1c8cb5118b8b462e8b83e3f58/merged major:0 minor:204 fsType:overlay blockSize:0} overlay_0-206:{mountpoint:/var/lib/containers/storage/overlay/d1778b899d7c50d1efe82228d61a3cd8e5f428dc163ef16056ae81e1d91f92dd/merged major:0 minor:206 fsType:overlay blockSize:0} overlay_0-208:{mountpoint:/var/lib/containers/storage/overlay/639c5487e283058fab06a15b52d224dbee9f0cfe3d5d9e57c5f0acd829f34820/merged major:0 minor:208 fsType:overlay blockSize:0} overlay_0-228:{mountpoint:/var/lib/containers/storage/overlay/f4736d4bf742b7470efa7d84993ddaf7628858a0079f3081c3bdab7e988cc539/merged major:0 minor:228 fsType:overlay blockSize:0} overlay_0-230:{mountpoint:/var/lib/containers/storage/overlay/062545c9175dd24f70e3fa49ed439fffce17335e4a6caff0f7709fb4c5578225/merged major:0 minor:230 fsType:overlay blockSize:0} overlay_0-232:{mountpoint:/var/lib/containers/storage/overlay/edced3ee4bc2560b06a8af0b6f5045fb7d5e31210fac4a5b1e045f81531c9555/merged major:0 minor:232 fsType:overlay blockSize:0} overlay_0-234:{mountpoint:/var/lib/containers/storage/overlay/90e081603502394474e5c8cfc8a610825bd7694432da3253c2e8961ee5569ea2/merged major:0 minor:234 fsType:overlay blockSize:0} overlay_0-236:{mountpoint:/var/lib/containers/storage/overlay/6c34e5a7f466d3fd6ff526f0f0db0a59a8ee2b2bd57a41ffb625a6b6a95efb05/merged major:0 minor:236 fsType:overlay blockSize:0} overlay_0-242:{mountpoint:/var/lib/containers/storage/overlay/9e40308114a34afd82a1b138dd254bcfd7bfb0a4910cce558008ef56eb79355a/merged major:0 minor:242 fsType:overlay blockSize:0} overlay_0-247:{mountpoint:/var/lib/containers/storage/overlay/5f99c2315c356ac3fc22c7c09c836c22f495b342d42690c3573ddca016cda8f7/merged major:0 minor:247 fsType:overlay blockSize:0} overlay_0-253:{mountpoint:/var/lib/containers/storage/overlay/631bd9161530ddea89abfc0dfa9eb8a2ac95dcd59ec28508fc91b3e53ca95157/merged major:0 minor:253 fsType:overlay blockSize:0} overlay_0-258:{mountpoint:/var/lib/containers/storage/overlay/95e6973cae646760c755e7fabd56a2c548c5558fac6050d075951aef617a25e8/merged major:0 minor:258 fsType:overlay blockSize:0} overlay_0-274:{mountpoint:/var/lib/containers/storage/overlay/566bf78b38bed10233e28abb083f83f19ab251d1ba9281ac648c8391886eff3a/merged major:0 minor:274 fsType:overlay blockSize:0} overlay_0-278:{mountpoint:/var/lib/containers/storage/overlay/9d5352eab8c400ec939f9d136a0899b4ba6c5a3f7a81963a8d65de2483d8882b/merged major:0 minor:278 fsType:overlay blockSize:0} overlay_0-280:{mountpoint:/var/lib/containers/storage/overlay/c437e31e237ca08f0478f1b9313f00dce0db78ede517d26d026942852715e608/merged major:0 minor:280 fsType:overlay blockSize:0} overlay_0-282:{mountpoint:/var/lib/containers/storage/overlay/44c7942db4dcc892ac1baac4d4ff397e122459bec591e658017ea161de2f9f45/merged major:0 minor:282 fsType:overlay blockSize:0} overlay_0-284:{mountpoint:/var/lib/containers/storage/overlay/04a80b8bf2296046551e57cef5f875f2ea5a319e008a8dcdda1dfd687e036078/merged major:0 minor:284 fsType:overlay blockSize:0} overlay_0-290:{mountpoint:/var/lib/containers/storage/overlay/d6718abee833e269ca50d94350884c4bad6e4f082e97ba65c9d527b1c4e85e76/merged major:0 minor:290 fsType:overlay blockSize:0} overlay_0-295:{mountpoint:/var/lib/containers/storage/overlay/43c1eb890f975800cd64c3538d5b64b62a469d4964d80411f8299adc4e411c06/merged major:0 minor:295 fsType:overlay blockSize:0} overlay_0-51:{mountpoint:/var/lib/containers/storage/overlay/cf8bcb61026b347d0c7c93c54985de7c51a345a68b29011074f90ee4ae6436d1/merged major:0 minor:51 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/4246993e65c4607c3a643e74fd054f8e27c5df8f9870f167b675cfec30a9bb39/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/407abc6017bf920e1c6c289b1ca1e955a7e1ccf84dc0532e3e1df7a00aded1ca/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/ce2c6bd4711676597810c96b7b330693882253f62adcf6ede298cf4598697a4d/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/5531ef7bd1903ab6ea81852a293791b5971afe56363b633c54ad1f0e00ad78f3/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-80:{mountpoint:/var/lib/containers/storage/overlay/6e27607fdbc067d97d0f72f6e2a5bd3ac206afb2b1fe5b2ffe57be176088f764/merged major:0 minor:80 fsType:overlay blockSize:0} overlay_0-84:{mountpoint:/var/lib/containers/storage/overlay/799a0e7af920d521dfb1675b5ec5af1998bd1dbfe407be794dd6fcfbcf3c5269/merged major:0 minor:84 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/f941313341d6465a1b5a81bf51d7eb04193ded5a18f98430796720dbe66eb2f2/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/238b58f23a851cefd1a00019d326f440f407996518863b41474f0a3bab605927/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-98:{mountpoint:/var/lib/containers/storage/overlay/0f7dd9eb11a658267a7f6aad4f989fbde4c57458a04851e2903bec716f3c569d/merged major:0 minor:98 fsType:overlay blockSize:0}]
Nov 02 14:56:40.066382 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.066025    5304 manager.go:217] Machine: {Timestamp:2026-11-02 14:56:40.065544694 +0000 UTC m=+0.059369095 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100274 MemoryCapacity:33164328960 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2d3573186e82525e7d99d71f5cc5d2 SystemUUID:ec2d3573-186e-8252-5e7d-99d71f5cc5d2 BootID:58ecc053-6c06-4b52-acf5-b49dc2859d14 Filesystems:[{Device:/run/containers/storage/overlay-containers/0aae85bb0b0b3446e320e80d65728fda1f1ac14b0af5de6c99416d8bde25db9d/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-98 DeviceMajor:0 DeviceMinor:98 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-204 DeviceMajor:0 DeviceMinor:204 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-206 DeviceMajor:0 DeviceMinor:206 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/03b691e65cf37f7a94beeee735d801b8dc31a3125b18747c22f89e827fd248ba/userdata/shm DeviceMajor:0 DeviceMinor:226 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-234 DeviceMajor:0 DeviceMinor:234 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d5d6f959273f30a64b18a421bf431362ed4babd9fee6bef83d84d8bf6665d58d/userdata/shm DeviceMajor:0 DeviceMinor:270 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-284 DeviceMajor:0 DeviceMinor:284 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-295 DeviceMajor:0 DeviceMinor:295 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-108 DeviceMajor:0 DeviceMinor:108 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-141 DeviceMajor:0 DeviceMinor:141 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/03b3012d-0bdf-4a78-ad31-c03d1df51ec0/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:197 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/979ce81d18065aab8e148d5e90f911903f8277633576644f10bbfa9486b6cadc/userdata/shm DeviceMajor:0 DeviceMinor:224 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-247 DeviceMajor:0 DeviceMinor:247 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-290 DeviceMajor:0 DeviceMinor:290 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/93d7d50e-b320-4217-b183-2bfc921f6c9d/volumes/kubernetes.io~projected/kube-api-access-ndjwt DeviceMajor:0 DeviceMinor:59 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/dedf3babecad2f468747d3e72210ea54c17d3ccf467e6671f136263e8ad8c84c/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e4f9e55c-3232-430c-a123-30d17704cd2e/volumes/kubernetes.io~projected/kube-api-access-ghkdw DeviceMajor:0 DeviceMinor:221 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/332f4a2d-a241-4c71-8ebb-a99618c0d3dc/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:268 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-278 DeviceMajor:0 DeviceMinor:278 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/777cd23a55bfad60d07f0c9cc7b69832952228d10ba770ce6b855beb1aa7fb9f/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-242 DeviceMajor:0 DeviceMinor:242 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582164480 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/54dc6898-f041-4ed6-9fc9-1b5159d00d27/volumes/kubernetes.io~projected/kube-api-access-8bt8x DeviceMajor:0 DeviceMinor:66 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/caa8898b-6611-49b8-845d-3b9bdac6691f/volumes/kubernetes.io~projected/kube-api-access-hz5t9 DeviceMajor:0 DeviceMinor:67 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-84 DeviceMajor:0 DeviceMinor:84 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/bf709fa05cba77da3f9435c58038e48838b3bfc405abff55963f4d883f2a19ba/userdata/shm DeviceMajor:0 DeviceMinor:86 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-133 DeviceMajor:0 DeviceMinor:133 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/65ce39fc-05c7-433b-81f4-c1b219f1bd71/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:57 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/e1da99c3-8294-4e34-9d33-47e04a6182d3/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:58 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/03b3012d-0bdf-4a78-ad31-c03d1df51ec0/volumes/kubernetes.io~secret/node-exporter-tls DeviceMajor:0 DeviceMinor:45 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2dffcc175ba039b64912a830e8787c67872dcb2ea3a2d115ce4d37a7558454e3/userdata/shm DeviceMajor:0 DeviceMinor:202 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/080a338e-be02-49fb-bebe-f873e3f4017e/volumes/kubernetes.io~secret/metrics-tls DeviceMajor:0 DeviceMinor:218 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/e1da99c3-8294-4e34-9d33-47e04a6182d3/volumes/kubernetes.io~projected/kube-api-access-prmck DeviceMajor:0 DeviceMinor:62 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-274 DeviceMajor:0 DeviceMinor:274 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-280 DeviceMajor:0 DeviceMinor:280 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-253 DeviceMajor:0 DeviceMinor:253 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/68b781b2-be96-4c5c-8daa-2238fd0a9623/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:55 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-80 DeviceMajor:0 DeviceMinor:80 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-208 DeviceMajor:0 DeviceMinor:208 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632865792 Type:vfs Inodes:819200 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9d80c77407d3e67f6198e00be9d36a6e7f9c402b652e9d03c548331917a7ac4c/userdata/shm DeviceMajor:0 DeviceMinor:48 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/720ef290-024b-4ed2-8417-92290f360dc9/volumes/kubernetes.io~projected/kube-api-access-f28t5 DeviceMajor:0 DeviceMinor:64 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2315db45e98cf6c8210934b607b07cfcaf32866bfffd173ccb4d1c2baae1849c/userdata/shm DeviceMajor:0 DeviceMinor:95 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-232 DeviceMajor:0 DeviceMinor:232 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582164480 Type:vfs Inodes:1048576 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-139 DeviceMajor:0 DeviceMinor:139 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e4f9e55c-3232-430c-a123-30d17704cd2e/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:213 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-258 DeviceMajor:0 DeviceMinor:258 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/65ce39fc-05c7-433b-81f4-c1b219f1bd71/volumes/kubernetes.io~projected/kube-api-access-ps49c DeviceMajor:0 DeviceMinor:63 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/b1715623-afd0-4d0e-9b63-b38499cec1e2/volumes/kubernetes.io~projected/kube-api-access-548bv DeviceMajor:0 DeviceMinor:65 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d1f7259e798f4c778135523c563af014d044be9176e87cba4459ed3b40ffab7d/userdata/shm DeviceMajor:0 DeviceMinor:82 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/702f7864-7db1-40d5-ba62-33cb9477b7da/volumes/kubernetes.io~projected/kube-api-access-kw44n DeviceMajor:0 DeviceMinor:220 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-230 DeviceMajor:0 DeviceMinor:230 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0d6c5872-e278-4463-953d-e1388a001d0d/volumes/kubernetes.io~projected/kube-api-access-hpfk5 DeviceMajor:0 DeviceMinor:273 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6917bc2965fdd531105d8ed16d2dea52c30702bfe6a4fd41de68d832fb3dff14/userdata/shm DeviceMajor:0 DeviceMinor:276 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/65ce39fc-05c7-433b-81f4-c1b219f1bd71/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632865792 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/080a338e-be02-49fb-bebe-f873e3f4017e/volumes/kubernetes.io~projected/kube-api-access-r67xl DeviceMajor:0 DeviceMinor:219 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/var/lib/kubelet/pods/e1da99c3-8294-4e34-9d33-47e04a6182d3/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:56 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b7ab68c4d9b36e6507e0795c8bb246a797d2758dfa1a5dc067d56ef0ef5b459d/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9881d50d11ec0a6732c1726cbb74624ab2bdebf9c73f8b5222237dafe496e451/userdata/shm DeviceMajor:0 DeviceMinor:222 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-236 DeviceMajor:0 DeviceMinor:236 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/caa8898b-6611-49b8-845d-3b9bdac6691f/volumes/kubernetes.io~secret/metrics-certs DeviceMajor:0 DeviceMinor:263 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-282 DeviceMajor:0 DeviceMinor:282 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-51 DeviceMajor:0 DeviceMinor:51 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7fe9ba5e-a867-4ac3-8a6f-a316c6f64db4/volumes/kubernetes.io~projected/kube-api-access-8rqfc DeviceMajor:0 DeviceMinor:61 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/03b3012d-0bdf-4a78-ad31-c03d1df51ec0/volumes/kubernetes.io~projected/kube-api-access-9vxjj DeviceMajor:0 DeviceMinor:201 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/702f7864-7db1-40d5-ba62-33cb9477b7da/volumes/kubernetes.io~secret/insights-runtime-extractor-tls DeviceMajor:0 DeviceMinor:217 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-228 DeviceMajor:0 DeviceMinor:228 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b236aedf11348eb6319968e96631ba9fb50c69aafadf3aa5a5c86ad633dd1234/userdata/shm DeviceMajor:0 DeviceMinor:269 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/var/lib/kubelet/pods/e4983b70-40f4-4e1e-a5da-80194d5d4786/volumes/kubernetes.io~projected/kube-api-access-7758v DeviceMajor:0 DeviceMinor:60 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/40258544b5fc72c08d5050ca375b814933292f7409af909ee8add2ebafb2e17f/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e13c510d9ba4b82e51fb2f10c06b32bbf9c8e5ebf6225c60f9db08c58e4c2b4c/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:03b691e65cf37f7 MacAddress:ca:51:81:19:e2:37 Speed:10000 Mtu:8901} {Name:6917bc2965fdd53 MacAddress:9a:07:2e:58:3b:25 Speed:10000 Mtu:8901} {Name:979ce81d18065aa MacAddress:f2:8a:53:78:29:62 Speed:10000 Mtu:8901} {Name:9881d50d11ec0a6 MacAddress:72:ce:3b:a8:ec:5f Speed:10000 Mtu:8901} {Name:b236aedf11348eb MacAddress:e6:f4:c2:62:c4:22 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:d3:c5:0b:7d Speed:0 Mtu:9001} {Name:br-int MacAddress:be:be:64:cc:a6:1f Speed:0 Mtu:8901} {Name:d5d6f959273f30a MacAddress:c6:a1:fa:48:db:5f Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:ff:d3:c5:0b:7d Speed:0 Mtu:9001} {Name:genev_sys_6081 MacAddress:7e:da:e3:6d:38:25 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:85:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:0e:3d:93:9a:5a:d4 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164328960 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Nov 02 14:56:40.066598 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.066516    5304 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Nov 02 14:56:40.066645 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.066590    5304 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Nov 02 14:56:40.066952 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.066926    5304 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Nov 02 14:56:40.067320 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.066955    5304 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-135-138.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Nov 02 14:56:40.067403 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.067326    5304 topology_manager.go:143] "Creating topology manager with none policy"
Nov 02 14:56:40.067403 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.067338    5304 container_manager_linux.go:311] "Creating device plugin manager"
Nov 02 14:56:40.067403 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.067352    5304 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 02 14:56:40.067403 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.067379    5304 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 02 14:56:40.067560 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.067515    5304 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Nov 02 14:56:40.068025 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.068010    5304 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Nov 02 14:56:40.068203 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.068171    5304 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Nov 02 14:56:40.068235 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.068210    5304 kubelet.go:493] "Attempting to sync node with API server"
Nov 02 14:56:40.068235 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.068226    5304 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Nov 02 14:56:40.068291 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.068236    5304 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Nov 02 14:56:40.068291 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.068246    5304 kubelet.go:405] "Adding apiserver pod source"
Nov 02 14:56:40.068291 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.068254    5304 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Nov 02 14:56:40.068859 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.068846    5304 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Nov 02 14:56:40.068859 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.068858    5304 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 02 14:56:40.069397 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.069380    5304 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Nov 02 14:56:40.071421 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.071409    5304 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Nov 02 14:56:40.071470 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.071422    5304 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 02 14:56:40.071531 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.071521    5304 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem"
Nov 02 14:56:40.071728 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.071717    5304 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Nov 02 14:56:40.071761 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.071746    5304 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Nov 02 14:56:40.071845 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.071836    5304 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Nov 02 14:56:40.071874 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.071851    5304 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Nov 02 14:56:40.071874 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.071857    5304 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Nov 02 14:56:40.071874 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.071863    5304 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Nov 02 14:56:40.071874 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.071869    5304 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Nov 02 14:56:40.071874 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.071874    5304 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Nov 02 14:56:40.072005 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.071881    5304 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Nov 02 14:56:40.072005 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.071886    5304 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Nov 02 14:56:40.072005 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.071894    5304 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Nov 02 14:56:40.072005 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.071899    5304 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Nov 02 14:56:40.072005 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.071913    5304 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Nov 02 14:56:40.072005 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.071923    5304 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Nov 02 14:56:40.072005 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.071950    5304 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Nov 02 14:56:40.072005 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.071955    5304 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Nov 02 14:56:40.075293 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.075281    5304 server.go:1263] "Started kubelet"
Nov 02 14:56:40.075379 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.075353    5304 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Nov 02 14:56:40.075759 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.075431    5304 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Nov 02 14:56:40.075759 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.075526    5304 server_v1.go:49] "podresources" method="list" useActivePods=true
Nov 02 14:56:40.075894 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.075807    5304 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Nov 02 14:56:40.075887 ip-10-0-135-138 systemd[1]: Started Kubernetes Kubelet.
Nov 02 14:56:40.077269 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.077254    5304 server.go:317] "Adding debug handlers to kubelet server"
Nov 02 14:56:40.078455 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.078435    5304 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Nov 02 14:56:40.078606 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.078477    5304 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Nov 02 14:56:40.078606 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.078495    5304 volume_manager.go:309] "The desired_state_of_world populator starts"
Nov 02 14:56:40.078606 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.078510    5304 volume_manager.go:311] "Starting Kubelet Volume Manager"
Nov 02 14:56:40.078606 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.078548    5304 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Nov 02 14:56:40.078606 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.078534    5304 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-01 14:50:31 +0000 UTC" deadline="2028-08-16 03:56:50.117667674 +0000 UTC"
Nov 02 14:56:40.078606 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.078568    5304 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15661h0m10.039102457s"
Nov 02 14:56:40.079786 ip-10-0-135-138 kubenswrapper[5304]: E1102 14:56:40.078625    5304 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-138.ec2.internal\" not found"
Nov 02 14:56:40.079786 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.079697    5304 factory.go:55] Registering systemd factory
Nov 02 14:56:40.079786 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.079717    5304 factory.go:223] Registration of the systemd container factory successfully
Nov 02 14:56:40.080904 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.080291    5304 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="080a338e-be02-49fb-bebe-f873e3f4017e" volumeName="kubernetes.io/configmap/080a338e-be02-49fb-bebe-f873e3f4017e-config-volume" seLinuxMountContext=""
Nov 02 14:56:40.080904 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.080323    5304 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="080a338e-be02-49fb-bebe-f873e3f4017e" volumeName="kubernetes.io/projected/080a338e-be02-49fb-bebe-f873e3f4017e-kube-api-access-r67xl" seLinuxMountContext=""
Nov 02 14:56:40.080904 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.080341    5304 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="65ce39fc-05c7-433b-81f4-c1b219f1bd71" volumeName="kubernetes.io/projected/65ce39fc-05c7-433b-81f4-c1b219f1bd71-kube-api-access-ps49c" seLinuxMountContext=""
Nov 02 14:56:40.080904 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.080354    5304 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="65ce39fc-05c7-433b-81f4-c1b219f1bd71" volumeName="kubernetes.io/configmap/65ce39fc-05c7-433b-81f4-c1b219f1bd71-ovnkube-config" seLinuxMountContext=""
Nov 02 14:56:40.080904 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.080369    5304 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b1715623-afd0-4d0e-9b63-b38499cec1e2" volumeName="kubernetes.io/configmap/b1715623-afd0-4d0e-9b63-b38499cec1e2-cni-sysctl-allowlist" seLinuxMountContext=""
Nov 02 14:56:40.080904 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.080381    5304 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b1715623-afd0-4d0e-9b63-b38499cec1e2" volumeName="kubernetes.io/configmap/b1715623-afd0-4d0e-9b63-b38499cec1e2-cni-binary-copy" seLinuxMountContext=""
Nov 02 14:56:40.080904 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.080394    5304 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="caa8898b-6611-49b8-845d-3b9bdac6691f" volumeName="kubernetes.io/secret/caa8898b-6611-49b8-845d-3b9bdac6691f-metrics-certs" seLinuxMountContext=""
Nov 02 14:56:40.080904 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.080400    5304 factory.go:153] Registering CRI-O factory
Nov 02 14:56:40.080904 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.080405    5304 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e4f9e55c-3232-430c-a123-30d17704cd2e" volumeName="kubernetes.io/projected/e4f9e55c-3232-430c-a123-30d17704cd2e-kube-api-access-ghkdw" seLinuxMountContext=""
Nov 02 14:56:40.080904 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.080415    5304 factory.go:223] Registration of the crio container factory successfully
Nov 02 14:56:40.080904 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.080428    5304 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e4f9e55c-3232-430c-a123-30d17704cd2e" volumeName="kubernetes.io/secret/e4f9e55c-3232-430c-a123-30d17704cd2e-cert" seLinuxMountContext=""
Nov 02 14:56:40.080904 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.080441    5304 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="03b3012d-0bdf-4a78-ad31-c03d1df51ec0" volumeName="kubernetes.io/projected/03b3012d-0bdf-4a78-ad31-c03d1df51ec0-kube-api-access-9vxjj" seLinuxMountContext=""
Nov 02 14:56:40.080904 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.080453    5304 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="080a338e-be02-49fb-bebe-f873e3f4017e" volumeName="kubernetes.io/secret/080a338e-be02-49fb-bebe-f873e3f4017e-metrics-tls" seLinuxMountContext=""
Nov 02 14:56:40.080904 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.080464    5304 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="65ce39fc-05c7-433b-81f4-c1b219f1bd71" volumeName="kubernetes.io/configmap/65ce39fc-05c7-433b-81f4-c1b219f1bd71-env-overrides" seLinuxMountContext=""
Nov 02 14:56:40.080904 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.080475    5304 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="65ce39fc-05c7-433b-81f4-c1b219f1bd71" volumeName="kubernetes.io/secret/65ce39fc-05c7-433b-81f4-c1b219f1bd71-ovn-node-metrics-cert" seLinuxMountContext=""
Nov 02 14:56:40.080904 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.080488    5304 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="702f7864-7db1-40d5-ba62-33cb9477b7da" volumeName="kubernetes.io/empty-dir/702f7864-7db1-40d5-ba62-33cb9477b7da-data-volume" seLinuxMountContext=""
Nov 02 14:56:40.080904 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.080506    5304 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Nov 02 14:56:40.080904 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.080506    5304 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="93d7d50e-b320-4217-b183-2bfc921f6c9d" volumeName="kubernetes.io/configmap/93d7d50e-b320-4217-b183-2bfc921f6c9d-multus-daemon-config" seLinuxMountContext=""
Nov 02 14:56:40.080904 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.080538    5304 factory.go:103] Registering Raw factory
Nov 02 14:56:40.080904 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.080530    5304 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="93d7d50e-b320-4217-b183-2bfc921f6c9d" volumeName="kubernetes.io/projected/93d7d50e-b320-4217-b183-2bfc921f6c9d-kube-api-access-ndjwt" seLinuxMountContext=""
Nov 02 14:56:40.080904 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.080557    5304 manager.go:1199] Started watching for new ooms in manager
Nov 02 14:56:40.080904 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.080556    5304 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7fe9ba5e-a867-4ac3-8a6f-a316c6f64db4" volumeName="kubernetes.io/projected/7fe9ba5e-a867-4ac3-8a6f-a316c6f64db4-kube-api-access-8rqfc" seLinuxMountContext=""
Nov 02 14:56:40.080904 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.080570    5304 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="03b3012d-0bdf-4a78-ad31-c03d1df51ec0" volumeName="kubernetes.io/secret/03b3012d-0bdf-4a78-ad31-c03d1df51ec0-node-exporter-kube-rbac-proxy-config" seLinuxMountContext=""
Nov 02 14:56:40.080904 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.080582    5304 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0d6c5872-e278-4463-953d-e1388a001d0d" volumeName="kubernetes.io/projected/0d6c5872-e278-4463-953d-e1388a001d0d-kube-api-access-hpfk5" seLinuxMountContext=""
Nov 02 14:56:40.080904 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.080594    5304 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="332f4a2d-a241-4c71-8ebb-a99618c0d3dc" volumeName="kubernetes.io/secret/332f4a2d-a241-4c71-8ebb-a99618c0d3dc-original-pull-secret" seLinuxMountContext=""
Nov 02 14:56:40.080904 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.080604    5304 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="702f7864-7db1-40d5-ba62-33cb9477b7da" volumeName="kubernetes.io/projected/702f7864-7db1-40d5-ba62-33cb9477b7da-kube-api-access-kw44n" seLinuxMountContext=""
Nov 02 14:56:40.080904 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.080615    5304 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="93d7d50e-b320-4217-b183-2bfc921f6c9d" volumeName="kubernetes.io/configmap/93d7d50e-b320-4217-b183-2bfc921f6c9d-cni-binary-copy" seLinuxMountContext=""
Nov 02 14:56:40.080904 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.080624    5304 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b1715623-afd0-4d0e-9b63-b38499cec1e2" volumeName="kubernetes.io/configmap/b1715623-afd0-4d0e-9b63-b38499cec1e2-whereabouts-flatfile-configmap" seLinuxMountContext=""
Nov 02 14:56:40.080904 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.080635    5304 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e1da99c3-8294-4e34-9d33-47e04a6182d3" volumeName="kubernetes.io/empty-dir/e1da99c3-8294-4e34-9d33-47e04a6182d3-etc-tuned" seLinuxMountContext=""
Nov 02 14:56:40.080904 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.080644    5304 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="03b3012d-0bdf-4a78-ad31-c03d1df51ec0" volumeName="kubernetes.io/configmap/03b3012d-0bdf-4a78-ad31-c03d1df51ec0-node-exporter-accelerators-collector-config" seLinuxMountContext=""
Nov 02 14:56:40.080904 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.080654    5304 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="54dc6898-f041-4ed6-9fc9-1b5159d00d27" volumeName="kubernetes.io/projected/54dc6898-f041-4ed6-9fc9-1b5159d00d27-kube-api-access-8bt8x" seLinuxMountContext=""
Nov 02 14:56:40.080904 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.080662    5304 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="702f7864-7db1-40d5-ba62-33cb9477b7da" volumeName="kubernetes.io/secret/702f7864-7db1-40d5-ba62-33cb9477b7da-insights-runtime-extractor-tls" seLinuxMountContext=""
Nov 02 14:56:40.080904 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.080673    5304 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="720ef290-024b-4ed2-8417-92290f360dc9" volumeName="kubernetes.io/empty-dir/720ef290-024b-4ed2-8417-92290f360dc9-tmp-dir" seLinuxMountContext=""
Nov 02 14:56:40.080904 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.080687    5304 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="caa8898b-6611-49b8-845d-3b9bdac6691f" volumeName="kubernetes.io/projected/caa8898b-6611-49b8-845d-3b9bdac6691f-kube-api-access-hz5t9" seLinuxMountContext=""
Nov 02 14:56:40.080904 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.080700    5304 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e1da99c3-8294-4e34-9d33-47e04a6182d3" volumeName="kubernetes.io/projected/e1da99c3-8294-4e34-9d33-47e04a6182d3-kube-api-access-prmck" seLinuxMountContext=""
Nov 02 14:56:40.080904 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.080712    5304 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="080a338e-be02-49fb-bebe-f873e3f4017e" volumeName="kubernetes.io/empty-dir/080a338e-be02-49fb-bebe-f873e3f4017e-tmp-dir" seLinuxMountContext=""
Nov 02 14:56:40.082765 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.080727    5304 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="68b781b2-be96-4c5c-8daa-2238fd0a9623" volumeName="kubernetes.io/secret/68b781b2-be96-4c5c-8daa-2238fd0a9623-agent-certs" seLinuxMountContext=""
Nov 02 14:56:40.082765 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.081397    5304 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="702f7864-7db1-40d5-ba62-33cb9477b7da" volumeName="kubernetes.io/configmap/702f7864-7db1-40d5-ba62-33cb9477b7da-kube-rbac-proxy-cm" seLinuxMountContext=""
Nov 02 14:56:40.082765 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.081410    5304 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7fe9ba5e-a867-4ac3-8a6f-a316c6f64db4" volumeName="kubernetes.io/configmap/7fe9ba5e-a867-4ac3-8a6f-a316c6f64db4-iptables-alerter-script" seLinuxMountContext=""
Nov 02 14:56:40.082765 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.081438    5304 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b1715623-afd0-4d0e-9b63-b38499cec1e2" volumeName="kubernetes.io/projected/b1715623-afd0-4d0e-9b63-b38499cec1e2-kube-api-access-548bv" seLinuxMountContext=""
Nov 02 14:56:40.082765 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.081452    5304 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e4983b70-40f4-4e1e-a5da-80194d5d4786" volumeName="kubernetes.io/configmap/e4983b70-40f4-4e1e-a5da-80194d5d4786-serviceca" seLinuxMountContext=""
Nov 02 14:56:40.082765 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.081509    5304 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="03b3012d-0bdf-4a78-ad31-c03d1df51ec0" volumeName="kubernetes.io/empty-dir/03b3012d-0bdf-4a78-ad31-c03d1df51ec0-node-exporter-textfile" seLinuxMountContext=""
Nov 02 14:56:40.082765 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.081519    5304 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="03b3012d-0bdf-4a78-ad31-c03d1df51ec0" volumeName="kubernetes.io/secret/03b3012d-0bdf-4a78-ad31-c03d1df51ec0-node-exporter-tls" seLinuxMountContext=""
Nov 02 14:56:40.082765 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.081527    5304 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="65ce39fc-05c7-433b-81f4-c1b219f1bd71" volumeName="kubernetes.io/configmap/65ce39fc-05c7-433b-81f4-c1b219f1bd71-ovnkube-script-lib" seLinuxMountContext=""
Nov 02 14:56:40.082765 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.081543    5304 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="68b781b2-be96-4c5c-8daa-2238fd0a9623" volumeName="kubernetes.io/configmap/68b781b2-be96-4c5c-8daa-2238fd0a9623-konnectivity-ca" seLinuxMountContext=""
Nov 02 14:56:40.082765 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.081555    5304 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="720ef290-024b-4ed2-8417-92290f360dc9" volumeName="kubernetes.io/projected/720ef290-024b-4ed2-8417-92290f360dc9-kube-api-access-f28t5" seLinuxMountContext=""
Nov 02 14:56:40.082765 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.081567    5304 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e1da99c3-8294-4e34-9d33-47e04a6182d3" volumeName="kubernetes.io/empty-dir/e1da99c3-8294-4e34-9d33-47e04a6182d3-tmp" seLinuxMountContext=""
Nov 02 14:56:40.082765 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.081576    5304 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e4983b70-40f4-4e1e-a5da-80194d5d4786" volumeName="kubernetes.io/projected/e4983b70-40f4-4e1e-a5da-80194d5d4786-kube-api-access-7758v" seLinuxMountContext=""
Nov 02 14:56:40.082765 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.081585    5304 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="03b3012d-0bdf-4a78-ad31-c03d1df51ec0" volumeName="kubernetes.io/configmap/03b3012d-0bdf-4a78-ad31-c03d1df51ec0-metrics-client-ca" seLinuxMountContext=""
Nov 02 14:56:40.082765 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.081593    5304 reconstruct.go:97] "Volume reconstruction finished"
Nov 02 14:56:40.082765 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.081598    5304 reconciler.go:29] "Reconciler: start to sync state"
Nov 02 14:56:40.082765 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.081969    5304 manager.go:319] Starting recovery of all containers
Nov 02 14:56:40.095365 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.095179    5304 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Nov 02 14:56:40.110602 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.110569    5304 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Nov 02 14:56:40.110602 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.110596    5304 status_manager.go:249] "Starting to sync pod status with apiserver"
Nov 02 14:56:40.110797 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.110616    5304 kubelet.go:2521] "Starting kubelet main sync loop"
Nov 02 14:56:40.110797 ip-10-0-135-138 kubenswrapper[5304]: E1102 14:56:40.110658    5304 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Nov 02 14:56:40.117863 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.117826    5304 generic.go:356] "Generic (PLEG): container finished" podID="b1715623-afd0-4d0e-9b63-b38499cec1e2" containerID="5d28dbe3f4717bb50c9f5a7d9e249c9e881f651f7ce67795bcfe4729c7246fb7" exitCode=0
Nov 02 14:56:40.117863 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.117859    5304 generic.go:356] "Generic (PLEG): container finished" podID="b1715623-afd0-4d0e-9b63-b38499cec1e2" containerID="24146e9d6d5a628b32ab6acd6a6ab9c79d72dfd83262baf717e2014333fbae11" exitCode=0
Nov 02 14:56:40.118025 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.117870    5304 generic.go:356] "Generic (PLEG): container finished" podID="b1715623-afd0-4d0e-9b63-b38499cec1e2" containerID="d9d301f4d4552c2692a103e04634e1ff319be58f054f325e2e270644e0710c13" exitCode=0
Nov 02 14:56:40.118025 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.117879    5304 generic.go:356] "Generic (PLEG): container finished" podID="b1715623-afd0-4d0e-9b63-b38499cec1e2" containerID="43c4fdb0aaa0f180b3f65b969d1924ef563707eed1d6e01247a4c01a41dda65b" exitCode=0
Nov 02 14:56:40.118025 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.117887    5304 generic.go:356] "Generic (PLEG): container finished" podID="b1715623-afd0-4d0e-9b63-b38499cec1e2" containerID="a1be58fa68a97dcb5a116a93e4b895bad3e7a128ad32668f5fd10fe450279f62" exitCode=0
Nov 02 14:56:40.118025 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.117895    5304 generic.go:356] "Generic (PLEG): container finished" podID="b1715623-afd0-4d0e-9b63-b38499cec1e2" containerID="f62b7e1913bd0bd851d8ec494f42b43881661a821c8e5291037ff952aa42cef8" exitCode=0
Nov 02 14:56:40.128145 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.128100    5304 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 02 14:56:40.128803 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.128783    5304 reconstruct.go:205] "DevicePaths of reconstructed volumes updated"
Nov 02 14:56:40.131977 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.131954    5304 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 02 14:56:40.140526 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.140475    5304 generic.go:356] "Generic (PLEG): container finished" podID="a98e99ec11875ba668469a960777e8a7" containerID="12796333caa3b4bb02b5aaa645b4fe31c754e02b6d8f2ca0346e657eaeb5ab7b" exitCode=0
Nov 02 14:56:40.143530 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.143508    5304 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 02 14:56:40.149379 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.149352    5304 generic.go:356] "Generic (PLEG): container finished" podID="03b3012d-0bdf-4a78-ad31-c03d1df51ec0" containerID="799bb67abac0a19dcbe9f1b21ded87c530dea52098d262f901b1c05af88c84fa" exitCode=0
Nov 02 14:56:40.152504 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.152482    5304 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 02 14:56:40.162004 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.161984    5304 manager.go:324] Recovery completed
Nov 02 14:56:40.177726 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.177705    5304 cpu_manager.go:226] "Starting" policy="none"
Nov 02 14:56:40.177726 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.177719    5304 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Nov 02 14:56:40.177870 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.177737    5304 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Nov 02 14:56:40.177870 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.177852    5304 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet=""
Nov 02 14:56:40.177935 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.177863    5304 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={}
Nov 02 14:56:40.177935 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.177881    5304 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint"
Nov 02 14:56:40.177935 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.177886    5304 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet=""
Nov 02 14:56:40.177935 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.177891    5304 policy_none.go:50] "Start"
Nov 02 14:56:40.177935 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.177901    5304 memory_manager.go:187] "Starting memorymanager" policy="None"
Nov 02 14:56:40.177935 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.177912    5304 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Nov 02 14:56:40.178080 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.178036    5304 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint"
Nov 02 14:56:40.178080 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.178043    5304 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint"
Nov 02 14:56:40.178080 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.178048    5304 policy_none.go:44] "Start"
Nov 02 14:56:40.182056 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.182027    5304 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Nov 02 14:56:40.184569 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.184519    5304 manager.go:341] "Starting Device Plugin manager"
Nov 02 14:56:40.185286 ip-10-0-135-138 kubenswrapper[5304]: E1102 14:56:40.184543    5304 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Nov 02 14:56:40.187705 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.187643    5304 server.go:87] "Starting device plugin registration server"
Nov 02 14:56:40.188190 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.188177    5304 eviction_manager.go:194] "Eviction manager: starting control loop"
Nov 02 14:56:40.188269 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.188191    5304 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Nov 02 14:56:40.188321 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.188279    5304 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Nov 02 14:56:40.188426 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.188387    5304 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Nov 02 14:56:40.188481 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.188438    5304 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Nov 02 14:56:40.188481 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.188445    5304 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Nov 02 14:56:40.188604 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.188525    5304 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-02T14:56:40.188416815Z","UUID":"71e1392e-826f-4169-b123-806a3d1c051e","Handler":null,"Name":"","Endpoint":""}
Nov 02 14:56:40.189914 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.189893    5304 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Nov 02 14:56:40.190031 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.189932    5304 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Nov 02 14:56:40.190112 ip-10-0-135-138 kubenswrapper[5304]: E1102 14:56:40.190096    5304 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Nov 02 14:56:40.210770 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.210723    5304 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-138.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-135-138.ec2.internal"]
Nov 02 14:56:40.211226 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.211195    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-138.ec2.internal" event={"ID":"9a8741d4cd8a99bb77c79c43a5144068","Type":"ContainerStarted","Data":"53cf9b2218c1a8bc9581d9f9c00cd11faa4cd2b7637385f17dc66335af2e8fae"}
Nov 02 14:56:40.211338 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.211230    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-138.ec2.internal" event={"ID":"9a8741d4cd8a99bb77c79c43a5144068","Type":"ContainerStarted","Data":"777cd23a55bfad60d07f0c9cc7b69832952228d10ba770ce6b855beb1aa7fb9f"}
Nov 02 14:56:40.211338 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.211280    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-138.ec2.internal" event={"ID":"a98e99ec11875ba668469a960777e8a7","Type":"ContainerStarted","Data":"d3908d1d68fe643637975c994cda57abe46a3172eaa4bd89d5123ae4844c17b4"}
Nov 02 14:56:40.211338 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.211295    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-138.ec2.internal" event={"ID":"a98e99ec11875ba668469a960777e8a7","Type":"ContainerDied","Data":"12796333caa3b4bb02b5aaa645b4fe31c754e02b6d8f2ca0346e657eaeb5ab7b"}
Nov 02 14:56:40.211338 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.211309    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-138.ec2.internal" event={"ID":"a98e99ec11875ba668469a960777e8a7","Type":"ContainerStarted","Data":"9d80c77407d3e67f6198e00be9d36a6e7f9c402b652e9d03c548331917a7ac4c"}
Nov 02 14:56:40.211338 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.211322    5304 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-138.ec2.internal"
Nov 02 14:56:40.211574 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.211424    5304 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-138.ec2.internal"
Nov 02 14:56:40.239521 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.239495    5304 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 02 14:56:40.239633 ip-10-0-135-138 kubenswrapper[5304]: E1102 14:56:40.239549    5304 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-135-138.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-138.ec2.internal"
Nov 02 14:56:40.240018 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.240003    5304 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 02 14:56:40.240066 ip-10-0-135-138 kubenswrapper[5304]: E1102 14:56:40.240030    5304 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-135-138.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-138.ec2.internal"
Nov 02 14:56:40.289182 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.289122    5304 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 14:56:40.293058 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.293037    5304 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-138.ec2.internal" event="NodeHasSufficientMemory"
Nov 02 14:56:40.293189 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.293070    5304 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-138.ec2.internal" event="NodeHasNoDiskPressure"
Nov 02 14:56:40.293189 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.293085    5304 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-138.ec2.internal" event="NodeHasSufficientPID"
Nov 02 14:56:40.293189 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.293141    5304 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-135-138.ec2.internal"
Nov 02 14:56:40.327163 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.327106    5304 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-135-138.ec2.internal"
Nov 02 14:56:40.327278 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.327204    5304 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-135-138.ec2.internal"
Nov 02 14:56:40.329925 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.329905    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a98e99ec11875ba668469a960777e8a7-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-138.ec2.internal\" (UID: \"a98e99ec11875ba668469a960777e8a7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-138.ec2.internal"
Nov 02 14:56:40.330028 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.329933    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a98e99ec11875ba668469a960777e8a7-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-138.ec2.internal\" (UID: \"a98e99ec11875ba668469a960777e8a7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-138.ec2.internal"
Nov 02 14:56:40.330028 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.329957    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/9a8741d4cd8a99bb77c79c43a5144068-config\") pod \"kube-apiserver-proxy-ip-10-0-135-138.ec2.internal\" (UID: \"9a8741d4cd8a99bb77c79c43a5144068\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-138.ec2.internal"
Nov 02 14:56:40.430780 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.430739    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a98e99ec11875ba668469a960777e8a7-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-138.ec2.internal\" (UID: \"a98e99ec11875ba668469a960777e8a7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-138.ec2.internal"
Nov 02 14:56:40.430780 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.430783    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/9a8741d4cd8a99bb77c79c43a5144068-config\") pod \"kube-apiserver-proxy-ip-10-0-135-138.ec2.internal\" (UID: \"9a8741d4cd8a99bb77c79c43a5144068\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-138.ec2.internal"
Nov 02 14:56:40.431024 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.430797    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a98e99ec11875ba668469a960777e8a7-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-138.ec2.internal\" (UID: \"a98e99ec11875ba668469a960777e8a7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-138.ec2.internal"
Nov 02 14:56:40.431024 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.430873    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/9a8741d4cd8a99bb77c79c43a5144068-config\") pod \"kube-apiserver-proxy-ip-10-0-135-138.ec2.internal\" (UID: \"9a8741d4cd8a99bb77c79c43a5144068\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-138.ec2.internal"
Nov 02 14:56:40.431024 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.430910    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a98e99ec11875ba668469a960777e8a7-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-138.ec2.internal\" (UID: \"a98e99ec11875ba668469a960777e8a7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-138.ec2.internal"
Nov 02 14:56:40.431024 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:40.430873    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a98e99ec11875ba668469a960777e8a7-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-138.ec2.internal\" (UID: \"a98e99ec11875ba668469a960777e8a7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-138.ec2.internal"
Nov 02 14:56:41.069037 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.069004    5304 apiserver.go:52] "Watching apiserver"
Nov 02 14:56:41.108016 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.107981    5304 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Nov 02 14:56:41.108457 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.108436    5304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-multus/network-metrics-daemon-484ld","openshift-dns/dns-default-kfnkt","openshift-ingress-canary/ingress-canary-fw8qc","openshift-monitoring/node-exporter-gpjvx","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6cj82","kube-system/kube-apiserver-proxy-ip-10-0-135-138.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-138.ec2.internal","openshift-multus/multus-7slzh","openshift-image-registry/node-ca-q9nx7","kube-system/konnectivity-agent-d5t6w","openshift-dns/node-resolver-jpz27","openshift-network-diagnostics/network-check-target-wns8r","openshift-network-operator/iptables-alerter-pl8gn","openshift-cluster-node-tuning-operator/tuned-sq4l4","openshift-insights/insights-runtime-extractor-k6fjk","openshift-ovn-kubernetes/ovnkube-node-f8btv","kube-system/global-pull-secret-syncer-vl6zb","openshift-multus/multus-additional-cni-plugins-fbnnl"]
Nov 02 14:56:41.123330 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.123307    5304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-x78pf\""
Nov 02 14:56:41.123863 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.123847    5304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-zhczs\""
Nov 02 14:56:41.123947 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.123875    5304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Nov 02 14:56:41.124211 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.124154    5304 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Nov 02 14:56:41.124211 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.124168    5304 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Nov 02 14:56:41.124211 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.124181    5304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Nov 02 14:56:41.124211 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.124209    5304 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Nov 02 14:56:41.124571 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.124220    5304 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Nov 02 14:56:41.124571 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.124228    5304 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Nov 02 14:56:41.124571 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.124271    5304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Nov 02 14:56:41.124571 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.124278    5304 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Nov 02 14:56:41.124571 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.124277    5304 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Nov 02 14:56:41.124571 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.124331    5304 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Nov 02 14:56:41.124571 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.124418    5304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-btkxl\""
Nov 02 14:56:41.124571 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.124462    5304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-7pvkf\""
Nov 02 14:56:41.124571 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.124489    5304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-62zsq\""
Nov 02 14:56:41.138004 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.137976    5304 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Nov 02 14:56:41.138453 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.138435    5304 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Nov 02 14:56:41.138626 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.138613    5304 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Nov 02 14:56:41.138751 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.138732    5304 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Nov 02 14:56:41.138829 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.138818    5304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-25j7m\""
Nov 02 14:56:41.138946 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.138931    5304 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Nov 02 14:56:41.139006 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.138980    5304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Nov 02 14:56:41.139050 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.139039    5304 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Nov 02 14:56:41.139176 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.139162    5304 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Nov 02 14:56:41.139306 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.139278    5304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Nov 02 14:56:41.139365 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.139354    5304 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Nov 02 14:56:41.139409 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.138886    5304 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Nov 02 14:56:41.139481 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.139468    5304 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Nov 02 14:56:41.139531 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.139504    5304 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Nov 02 14:56:41.139531 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.139512    5304 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Nov 02 14:56:41.139628 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.139563    5304 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Nov 02 14:56:41.139628 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.139581    5304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-st89p\""
Nov 02 14:56:41.139628 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.139598    5304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Nov 02 14:56:41.139812 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.138478    5304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-nts94\""
Nov 02 14:56:41.139812 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.138621    5304 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Nov 02 14:56:41.139812 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.139670    5304 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Nov 02 14:56:41.139812 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.139210    5304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Nov 02 14:56:41.139812 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.139680    5304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Nov 02 14:56:41.139812 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.139694    5304 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Nov 02 14:56:41.139812 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.138847    5304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-2fmch\""
Nov 02 14:56:41.139812 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.139750    5304 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Nov 02 14:56:41.139812 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.139728    5304 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Nov 02 14:56:41.139812 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.139506    5304 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Nov 02 14:56:41.139812 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.139783    5304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-pj6nr\""
Nov 02 14:56:41.139812 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.139793    5304 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Nov 02 14:56:41.139812 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.139796    5304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Nov 02 14:56:41.139812 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.139821    5304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Nov 02 14:56:41.140359 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.139816    5304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-jmg9z\""
Nov 02 14:56:41.140359 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.139834    5304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-v7sjw\""
Nov 02 14:56:41.140359 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.139862    5304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-kws2w\""
Nov 02 14:56:41.140359 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.139866    5304 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Nov 02 14:56:41.140359 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.139907    5304 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Nov 02 14:56:41.140359 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.139909    5304 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Nov 02 14:56:41.140359 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.139916    5304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-nnwvk\""
Nov 02 14:56:41.140359 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.139918    5304 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Nov 02 14:56:41.140359 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.139949    5304 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Nov 02 14:56:41.140359 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.140019    5304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-dddv8\""
Nov 02 14:56:41.140633 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.140094    5304 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Nov 02 14:56:41.147807 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.147790    5304 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Nov 02 14:56:41.147868 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.147853    5304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-ssfbf\""
Nov 02 14:56:41.147899 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.147870    5304 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Nov 02 14:56:41.153601 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.153579    5304 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-138.ec2.internal"
Nov 02 14:56:41.153675 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.153630    5304 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-138.ec2.internal"
Nov 02 14:56:41.174938 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.174918    5304 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 02 14:56:41.175024 ip-10-0-135-138 kubenswrapper[5304]: E1102 14:56:41.174956    5304 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-135-138.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-138.ec2.internal"
Nov 02 14:56:41.175544 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.175531    5304 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 02 14:56:41.175577 ip-10-0-135-138 kubenswrapper[5304]: E1102 14:56:41.175563    5304 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-135-138.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-138.ec2.internal"
Nov 02 14:56:41.179527 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.179513    5304 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Nov 02 14:56:41.235622 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.235595    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/93d7d50e-b320-4217-b183-2bfc921f6c9d-multus-cni-dir\") pod \"multus-7slzh\" (UID: \"93d7d50e-b320-4217-b183-2bfc921f6c9d\") " pod="openshift-multus/multus-7slzh"
Nov 02 14:56:41.235622 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.235623    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/93d7d50e-b320-4217-b183-2bfc921f6c9d-host-var-lib-cni-multus\") pod \"multus-7slzh\" (UID: \"93d7d50e-b320-4217-b183-2bfc921f6c9d\") " pod="openshift-multus/multus-7slzh"
Nov 02 14:56:41.235774 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.235640    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e1da99c3-8294-4e34-9d33-47e04a6182d3-host\") pod \"tuned-sq4l4\" (UID: \"e1da99c3-8294-4e34-9d33-47e04a6182d3\") " pod="openshift-cluster-node-tuning-operator/tuned-sq4l4"
Nov 02 14:56:41.235774 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.235665    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/65ce39fc-05c7-433b-81f4-c1b219f1bd71-ovn-node-metrics-cert\") pod \"ovnkube-node-f8btv\" (UID: \"65ce39fc-05c7-433b-81f4-c1b219f1bd71\") " pod="openshift-ovn-kubernetes/ovnkube-node-f8btv"
Nov 02 14:56:41.235774 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.235744    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/68b781b2-be96-4c5c-8daa-2238fd0a9623-agent-certs\") pod \"konnectivity-agent-d5t6w\" (UID: \"68b781b2-be96-4c5c-8daa-2238fd0a9623\") " pod="kube-system/konnectivity-agent-d5t6w"
Nov 02 14:56:41.235878 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.235771    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b1715623-afd0-4d0e-9b63-b38499cec1e2-cnibin\") pod \"multus-additional-cni-plugins-fbnnl\" (UID: \"b1715623-afd0-4d0e-9b63-b38499cec1e2\") " pod="openshift-multus/multus-additional-cni-plugins-fbnnl"
Nov 02 14:56:41.235922 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.235884    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/080a338e-be02-49fb-bebe-f873e3f4017e-metrics-tls\") pod \"dns-default-kfnkt\" (UID: \"080a338e-be02-49fb-bebe-f873e3f4017e\") " pod="openshift-dns/dns-default-kfnkt"
Nov 02 14:56:41.235958 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.235924    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/93d7d50e-b320-4217-b183-2bfc921f6c9d-host-run-multus-certs\") pod \"multus-7slzh\" (UID: \"93d7d50e-b320-4217-b183-2bfc921f6c9d\") " pod="openshift-multus/multus-7slzh"
Nov 02 14:56:41.235996 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.235957    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/03b3012d-0bdf-4a78-ad31-c03d1df51ec0-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-gpjvx\" (UID: \"03b3012d-0bdf-4a78-ad31-c03d1df51ec0\") " pod="openshift-monitoring/node-exporter-gpjvx"
Nov 02 14:56:41.235996 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.235983    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/65ce39fc-05c7-433b-81f4-c1b219f1bd71-ovn-node-metrics-cert\") pod \"ovnkube-node-f8btv\" (UID: \"65ce39fc-05c7-433b-81f4-c1b219f1bd71\") " pod="openshift-ovn-kubernetes/ovnkube-node-f8btv"
Nov 02 14:56:41.236052 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.236028    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/68b781b2-be96-4c5c-8daa-2238fd0a9623-agent-certs\") pod \"konnectivity-agent-d5t6w\" (UID: \"68b781b2-be96-4c5c-8daa-2238fd0a9623\") " pod="kube-system/konnectivity-agent-d5t6w"
Nov 02 14:56:41.236092 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.236075    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ghkdw\" (UniqueName: \"kubernetes.io/projected/e4f9e55c-3232-430c-a123-30d17704cd2e-kube-api-access-ghkdw\") pod \"ingress-canary-fw8qc\" (UID: \"e4f9e55c-3232-430c-a123-30d17704cd2e\") " pod="openshift-ingress-canary/ingress-canary-fw8qc"
Nov 02 14:56:41.236195 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.236097    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/65ce39fc-05c7-433b-81f4-c1b219f1bd71-systemd-units\") pod \"ovnkube-node-f8btv\" (UID: \"65ce39fc-05c7-433b-81f4-c1b219f1bd71\") " pod="openshift-ovn-kubernetes/ovnkube-node-f8btv"
Nov 02 14:56:41.236195 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.236158    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/b1715623-afd0-4d0e-9b63-b38499cec1e2-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-fbnnl\" (UID: \"b1715623-afd0-4d0e-9b63-b38499cec1e2\") " pod="openshift-multus/multus-additional-cni-plugins-fbnnl"
Nov 02 14:56:41.236195 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.236181    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/080a338e-be02-49fb-bebe-f873e3f4017e-metrics-tls\") pod \"dns-default-kfnkt\" (UID: \"080a338e-be02-49fb-bebe-f873e3f4017e\") " pod="openshift-dns/dns-default-kfnkt"
Nov 02 14:56:41.236290 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.236276    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-548bv\" (UniqueName: \"kubernetes.io/projected/b1715623-afd0-4d0e-9b63-b38499cec1e2-kube-api-access-548bv\") pod \"multus-additional-cni-plugins-fbnnl\" (UID: \"b1715623-afd0-4d0e-9b63-b38499cec1e2\") " pod="openshift-multus/multus-additional-cni-plugins-fbnnl"
Nov 02 14:56:41.236327 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.236310    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/080a338e-be02-49fb-bebe-f873e3f4017e-config-volume\") pod \"dns-default-kfnkt\" (UID: \"080a338e-be02-49fb-bebe-f873e3f4017e\") " pod="openshift-dns/dns-default-kfnkt"
Nov 02 14:56:41.236327 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.236318    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/03b3012d-0bdf-4a78-ad31-c03d1df51ec0-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-gpjvx\" (UID: \"03b3012d-0bdf-4a78-ad31-c03d1df51ec0\") " pod="openshift-monitoring/node-exporter-gpjvx"
Nov 02 14:56:41.236395 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.236329    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/93d7d50e-b320-4217-b183-2bfc921f6c9d-cnibin\") pod \"multus-7slzh\" (UID: \"93d7d50e-b320-4217-b183-2bfc921f6c9d\") " pod="openshift-multus/multus-7slzh"
Nov 02 14:56:41.236435 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.236401    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/93d7d50e-b320-4217-b183-2bfc921f6c9d-host-run-k8s-cni-cncf-io\") pod \"multus-7slzh\" (UID: \"93d7d50e-b320-4217-b183-2bfc921f6c9d\") " pod="openshift-multus/multus-7slzh"
Nov 02 14:56:41.236435 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.236416    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/b1715623-afd0-4d0e-9b63-b38499cec1e2-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-fbnnl\" (UID: \"b1715623-afd0-4d0e-9b63-b38499cec1e2\") " pod="openshift-multus/multus-additional-cni-plugins-fbnnl"
Nov 02 14:56:41.236505 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.236451    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/93d7d50e-b320-4217-b183-2bfc921f6c9d-host-var-lib-kubelet\") pod \"multus-7slzh\" (UID: \"93d7d50e-b320-4217-b183-2bfc921f6c9d\") " pod="openshift-multus/multus-7slzh"
Nov 02 14:56:41.236535 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.236512    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/080a338e-be02-49fb-bebe-f873e3f4017e-config-volume\") pod \"dns-default-kfnkt\" (UID: \"080a338e-be02-49fb-bebe-f873e3f4017e\") " pod="openshift-dns/dns-default-kfnkt"
Nov 02 14:56:41.236569 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.236545    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/03b3012d-0bdf-4a78-ad31-c03d1df51ec0-node-exporter-wtmp\") pod \"node-exporter-gpjvx\" (UID: \"03b3012d-0bdf-4a78-ad31-c03d1df51ec0\") " pod="openshift-monitoring/node-exporter-gpjvx"
Nov 02 14:56:41.236598 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.236585    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e4f9e55c-3232-430c-a123-30d17704cd2e-cert\") pod \"ingress-canary-fw8qc\" (UID: \"e4f9e55c-3232-430c-a123-30d17704cd2e\") " pod="openshift-ingress-canary/ingress-canary-fw8qc"
Nov 02 14:56:41.236635 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.236612    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f28t5\" (UniqueName: \"kubernetes.io/projected/720ef290-024b-4ed2-8417-92290f360dc9-kube-api-access-f28t5\") pod \"node-resolver-jpz27\" (UID: \"720ef290-024b-4ed2-8417-92290f360dc9\") " pod="openshift-dns/node-resolver-jpz27"
Nov 02 14:56:41.236686 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.236672    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7758v\" (UniqueName: \"kubernetes.io/projected/e4983b70-40f4-4e1e-a5da-80194d5d4786-kube-api-access-7758v\") pod \"node-ca-q9nx7\" (UID: \"e4983b70-40f4-4e1e-a5da-80194d5d4786\") " pod="openshift-image-registry/node-ca-q9nx7"
Nov 02 14:56:41.236743 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.236721    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b1715623-afd0-4d0e-9b63-b38499cec1e2-system-cni-dir\") pod \"multus-additional-cni-plugins-fbnnl\" (UID: \"b1715623-afd0-4d0e-9b63-b38499cec1e2\") " pod="openshift-multus/multus-additional-cni-plugins-fbnnl"
Nov 02 14:56:41.236838 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.236820    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e4f9e55c-3232-430c-a123-30d17704cd2e-cert\") pod \"ingress-canary-fw8qc\" (UID: \"e4f9e55c-3232-430c-a123-30d17704cd2e\") " pod="openshift-ingress-canary/ingress-canary-fw8qc"
Nov 02 14:56:41.236877 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.236866    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/93d7d50e-b320-4217-b183-2bfc921f6c9d-host-run-netns\") pod \"multus-7slzh\" (UID: \"93d7d50e-b320-4217-b183-2bfc921f6c9d\") " pod="openshift-multus/multus-7slzh"
Nov 02 14:56:41.236907 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.236900    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/7fe9ba5e-a867-4ac3-8a6f-a316c6f64db4-iptables-alerter-script\") pod \"iptables-alerter-pl8gn\" (UID: \"7fe9ba5e-a867-4ac3-8a6f-a316c6f64db4\") " pod="openshift-network-operator/iptables-alerter-pl8gn"
Nov 02 14:56:41.236948 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.236917    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7fe9ba5e-a867-4ac3-8a6f-a316c6f64db4-host-slash\") pod \"iptables-alerter-pl8gn\" (UID: \"7fe9ba5e-a867-4ac3-8a6f-a316c6f64db4\") " pod="openshift-network-operator/iptables-alerter-pl8gn"
Nov 02 14:56:41.236948 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.236932    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/65ce39fc-05c7-433b-81f4-c1b219f1bd71-run-openvswitch\") pod \"ovnkube-node-f8btv\" (UID: \"65ce39fc-05c7-433b-81f4-c1b219f1bd71\") " pod="openshift-ovn-kubernetes/ovnkube-node-f8btv"
Nov 02 14:56:41.237010 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.236953    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/65ce39fc-05c7-433b-81f4-c1b219f1bd71-run-ovn\") pod \"ovnkube-node-f8btv\" (UID: \"65ce39fc-05c7-433b-81f4-c1b219f1bd71\") " pod="openshift-ovn-kubernetes/ovnkube-node-f8btv"
Nov 02 14:56:41.237010 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.236995    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b1715623-afd0-4d0e-9b63-b38499cec1e2-cni-binary-copy\") pod \"multus-additional-cni-plugins-fbnnl\" (UID: \"b1715623-afd0-4d0e-9b63-b38499cec1e2\") " pod="openshift-multus/multus-additional-cni-plugins-fbnnl"
Nov 02 14:56:41.237069 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.237026    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/b1715623-afd0-4d0e-9b63-b38499cec1e2-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-fbnnl\" (UID: \"b1715623-afd0-4d0e-9b63-b38499cec1e2\") " pod="openshift-multus/multus-additional-cni-plugins-fbnnl"
Nov 02 14:56:41.237069 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.237041    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/7fe9ba5e-a867-4ac3-8a6f-a316c6f64db4-iptables-alerter-script\") pod \"iptables-alerter-pl8gn\" (UID: \"7fe9ba5e-a867-4ac3-8a6f-a316c6f64db4\") " pod="openshift-network-operator/iptables-alerter-pl8gn"
Nov 02 14:56:41.237069 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.237050    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/54dc6898-f041-4ed6-9fc9-1b5159d00d27-socket-dir\") pod \"aws-ebs-csi-driver-node-6cj82\" (UID: \"54dc6898-f041-4ed6-9fc9-1b5159d00d27\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6cj82"
Nov 02 14:56:41.237181 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.237079    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9vxjj\" (UniqueName: \"kubernetes.io/projected/03b3012d-0bdf-4a78-ad31-c03d1df51ec0-kube-api-access-9vxjj\") pod \"node-exporter-gpjvx\" (UID: \"03b3012d-0bdf-4a78-ad31-c03d1df51ec0\") " pod="openshift-monitoring/node-exporter-gpjvx"
Nov 02 14:56:41.237213 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.237182    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/702f7864-7db1-40d5-ba62-33cb9477b7da-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-k6fjk\" (UID: \"702f7864-7db1-40d5-ba62-33cb9477b7da\") " pod="openshift-insights/insights-runtime-extractor-k6fjk"
Nov 02 14:56:41.237213 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.237202    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/54dc6898-f041-4ed6-9fc9-1b5159d00d27-registration-dir\") pod \"aws-ebs-csi-driver-node-6cj82\" (UID: \"54dc6898-f041-4ed6-9fc9-1b5159d00d27\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6cj82"
Nov 02 14:56:41.237279 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.237219    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/93d7d50e-b320-4217-b183-2bfc921f6c9d-multus-conf-dir\") pod \"multus-7slzh\" (UID: \"93d7d50e-b320-4217-b183-2bfc921f6c9d\") " pod="openshift-multus/multus-7slzh"
Nov 02 14:56:41.237279 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.237242    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/93d7d50e-b320-4217-b183-2bfc921f6c9d-multus-daemon-config\") pod \"multus-7slzh\" (UID: \"93d7d50e-b320-4217-b183-2bfc921f6c9d\") " pod="openshift-multus/multus-7slzh"
Nov 02 14:56:41.237279 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.237265    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/b1715623-afd0-4d0e-9b63-b38499cec1e2-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-fbnnl\" (UID: \"b1715623-afd0-4d0e-9b63-b38499cec1e2\") " pod="openshift-multus/multus-additional-cni-plugins-fbnnl"
Nov 02 14:56:41.237279 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.237275    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b1715623-afd0-4d0e-9b63-b38499cec1e2-cni-binary-copy\") pod \"multus-additional-cni-plugins-fbnnl\" (UID: \"b1715623-afd0-4d0e-9b63-b38499cec1e2\") " pod="openshift-multus/multus-additional-cni-plugins-fbnnl"
Nov 02 14:56:41.237396 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.237275    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e1da99c3-8294-4e34-9d33-47e04a6182d3-etc-sysconfig\") pod \"tuned-sq4l4\" (UID: \"e1da99c3-8294-4e34-9d33-47e04a6182d3\") " pod="openshift-cluster-node-tuning-operator/tuned-sq4l4"
Nov 02 14:56:41.237396 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.237364    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e1da99c3-8294-4e34-9d33-47e04a6182d3-tmp\") pod \"tuned-sq4l4\" (UID: \"e1da99c3-8294-4e34-9d33-47e04a6182d3\") " pod="openshift-cluster-node-tuning-operator/tuned-sq4l4"
Nov 02 14:56:41.237462 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.237395    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/03b3012d-0bdf-4a78-ad31-c03d1df51ec0-metrics-client-ca\") pod \"node-exporter-gpjvx\" (UID: \"03b3012d-0bdf-4a78-ad31-c03d1df51ec0\") " pod="openshift-monitoring/node-exporter-gpjvx"
Nov 02 14:56:41.237462 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.237424    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/93d7d50e-b320-4217-b183-2bfc921f6c9d-multus-daemon-config\") pod \"multus-7slzh\" (UID: \"93d7d50e-b320-4217-b183-2bfc921f6c9d\") " pod="openshift-multus/multus-7slzh"
Nov 02 14:56:41.237462 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.237426    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hz5t9\" (UniqueName: \"kubernetes.io/projected/caa8898b-6611-49b8-845d-3b9bdac6691f-kube-api-access-hz5t9\") pod \"network-metrics-daemon-484ld\" (UID: \"caa8898b-6611-49b8-845d-3b9bdac6691f\") " pod="openshift-multus/network-metrics-daemon-484ld"
Nov 02 14:56:41.237462 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.237448    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/702f7864-7db1-40d5-ba62-33cb9477b7da-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-k6fjk\" (UID: \"702f7864-7db1-40d5-ba62-33cb9477b7da\") " pod="openshift-insights/insights-runtime-extractor-k6fjk"
Nov 02 14:56:41.237462 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.237458    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e1da99c3-8294-4e34-9d33-47e04a6182d3-run\") pod \"tuned-sq4l4\" (UID: \"e1da99c3-8294-4e34-9d33-47e04a6182d3\") " pod="openshift-cluster-node-tuning-operator/tuned-sq4l4"
Nov 02 14:56:41.237611 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.237427    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e1da99c3-8294-4e34-9d33-47e04a6182d3-tmp\") pod \"tuned-sq4l4\" (UID: \"e1da99c3-8294-4e34-9d33-47e04a6182d3\") " pod="openshift-cluster-node-tuning-operator/tuned-sq4l4"
Nov 02 14:56:41.237611 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.237490    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-prmck\" (UniqueName: \"kubernetes.io/projected/e1da99c3-8294-4e34-9d33-47e04a6182d3-kube-api-access-prmck\") pod \"tuned-sq4l4\" (UID: \"e1da99c3-8294-4e34-9d33-47e04a6182d3\") " pod="openshift-cluster-node-tuning-operator/tuned-sq4l4"
Nov 02 14:56:41.237611 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.237521    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/65ce39fc-05c7-433b-81f4-c1b219f1bd71-host-kubelet\") pod \"ovnkube-node-f8btv\" (UID: \"65ce39fc-05c7-433b-81f4-c1b219f1bd71\") " pod="openshift-ovn-kubernetes/ovnkube-node-f8btv"
Nov 02 14:56:41.237611 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.237549    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/03b3012d-0bdf-4a78-ad31-c03d1df51ec0-metrics-client-ca\") pod \"node-exporter-gpjvx\" (UID: \"03b3012d-0bdf-4a78-ad31-c03d1df51ec0\") " pod="openshift-monitoring/node-exporter-gpjvx"
Nov 02 14:56:41.237611 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.237556    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/65ce39fc-05c7-433b-81f4-c1b219f1bd71-host-run-netns\") pod \"ovnkube-node-f8btv\" (UID: \"65ce39fc-05c7-433b-81f4-c1b219f1bd71\") " pod="openshift-ovn-kubernetes/ovnkube-node-f8btv"
Nov 02 14:56:41.237611 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.237579    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/65ce39fc-05c7-433b-81f4-c1b219f1bd71-run-systemd\") pod \"ovnkube-node-f8btv\" (UID: \"65ce39fc-05c7-433b-81f4-c1b219f1bd71\") " pod="openshift-ovn-kubernetes/ovnkube-node-f8btv"
Nov 02 14:56:41.237785 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.237618    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/65ce39fc-05c7-433b-81f4-c1b219f1bd71-env-overrides\") pod \"ovnkube-node-f8btv\" (UID: \"65ce39fc-05c7-433b-81f4-c1b219f1bd71\") " pod="openshift-ovn-kubernetes/ovnkube-node-f8btv"
Nov 02 14:56:41.237785 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.237665    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/65ce39fc-05c7-433b-81f4-c1b219f1bd71-ovnkube-script-lib\") pod \"ovnkube-node-f8btv\" (UID: \"65ce39fc-05c7-433b-81f4-c1b219f1bd71\") " pod="openshift-ovn-kubernetes/ovnkube-node-f8btv"
Nov 02 14:56:41.237785 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.237756    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/65ce39fc-05c7-433b-81f4-c1b219f1bd71-env-overrides\") pod \"ovnkube-node-f8btv\" (UID: \"65ce39fc-05c7-433b-81f4-c1b219f1bd71\") " pod="openshift-ovn-kubernetes/ovnkube-node-f8btv"
Nov 02 14:56:41.237785 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.237779    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/65ce39fc-05c7-433b-81f4-c1b219f1bd71-node-log\") pod \"ovnkube-node-f8btv\" (UID: \"65ce39fc-05c7-433b-81f4-c1b219f1bd71\") " pod="openshift-ovn-kubernetes/ovnkube-node-f8btv"
Nov 02 14:56:41.237909 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.237819    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/720ef290-024b-4ed2-8417-92290f360dc9-tmp-dir\") pod \"node-resolver-jpz27\" (UID: \"720ef290-024b-4ed2-8417-92290f360dc9\") " pod="openshift-dns/node-resolver-jpz27"
Nov 02 14:56:41.237909 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.237826    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/65ce39fc-05c7-433b-81f4-c1b219f1bd71-ovnkube-script-lib\") pod \"ovnkube-node-f8btv\" (UID: \"65ce39fc-05c7-433b-81f4-c1b219f1bd71\") " pod="openshift-ovn-kubernetes/ovnkube-node-f8btv"
Nov 02 14:56:41.237909 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.237846    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/332f4a2d-a241-4c71-8ebb-a99618c0d3dc-dbus\") pod \"global-pull-secret-syncer-vl6zb\" (UID: \"332f4a2d-a241-4c71-8ebb-a99618c0d3dc\") " pod="kube-system/global-pull-secret-syncer-vl6zb"
Nov 02 14:56:41.237909 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.237879    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/720ef290-024b-4ed2-8417-92290f360dc9-tmp-dir\") pod \"node-resolver-jpz27\" (UID: \"720ef290-024b-4ed2-8417-92290f360dc9\") " pod="openshift-dns/node-resolver-jpz27"
Nov 02 14:56:41.237909 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.237905    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/03b3012d-0bdf-4a78-ad31-c03d1df51ec0-sys\") pod \"node-exporter-gpjvx\" (UID: \"03b3012d-0bdf-4a78-ad31-c03d1df51ec0\") " pod="openshift-monitoring/node-exporter-gpjvx"
Nov 02 14:56:41.238059 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.237920    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/65ce39fc-05c7-433b-81f4-c1b219f1bd71-host-slash\") pod \"ovnkube-node-f8btv\" (UID: \"65ce39fc-05c7-433b-81f4-c1b219f1bd71\") " pod="openshift-ovn-kubernetes/ovnkube-node-f8btv"
Nov 02 14:56:41.238059 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.237944    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/caa8898b-6611-49b8-845d-3b9bdac6691f-metrics-certs\") pod \"network-metrics-daemon-484ld\" (UID: \"caa8898b-6611-49b8-845d-3b9bdac6691f\") " pod="openshift-multus/network-metrics-daemon-484ld"
Nov 02 14:56:41.238059 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.237970    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e1da99c3-8294-4e34-9d33-47e04a6182d3-etc-sysctl-conf\") pod \"tuned-sq4l4\" (UID: \"e1da99c3-8294-4e34-9d33-47e04a6182d3\") " pod="openshift-cluster-node-tuning-operator/tuned-sq4l4"
Nov 02 14:56:41.238059 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.237987    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e1da99c3-8294-4e34-9d33-47e04a6182d3-lib-modules\") pod \"tuned-sq4l4\" (UID: \"e1da99c3-8294-4e34-9d33-47e04a6182d3\") " pod="openshift-cluster-node-tuning-operator/tuned-sq4l4"
Nov 02 14:56:41.238195 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.238061    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e1da99c3-8294-4e34-9d33-47e04a6182d3-etc-tuned\") pod \"tuned-sq4l4\" (UID: \"e1da99c3-8294-4e34-9d33-47e04a6182d3\") " pod="openshift-cluster-node-tuning-operator/tuned-sq4l4"
Nov 02 14:56:41.238195 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.238101    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kw44n\" (UniqueName: \"kubernetes.io/projected/702f7864-7db1-40d5-ba62-33cb9477b7da-kube-api-access-kw44n\") pod \"insights-runtime-extractor-k6fjk\" (UID: \"702f7864-7db1-40d5-ba62-33cb9477b7da\") " pod="openshift-insights/insights-runtime-extractor-k6fjk"
Nov 02 14:56:41.238195 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.238145    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/93d7d50e-b320-4217-b183-2bfc921f6c9d-system-cni-dir\") pod \"multus-7slzh\" (UID: \"93d7d50e-b320-4217-b183-2bfc921f6c9d\") " pod="openshift-multus/multus-7slzh"
Nov 02 14:56:41.238195 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.238155    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/caa8898b-6611-49b8-845d-3b9bdac6691f-metrics-certs\") pod \"network-metrics-daemon-484ld\" (UID: \"caa8898b-6611-49b8-845d-3b9bdac6691f\") " pod="openshift-multus/network-metrics-daemon-484ld"
Nov 02 14:56:41.238195 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.238160    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e1da99c3-8294-4e34-9d33-47e04a6182d3-sys\") pod \"tuned-sq4l4\" (UID: \"e1da99c3-8294-4e34-9d33-47e04a6182d3\") " pod="openshift-cluster-node-tuning-operator/tuned-sq4l4"
Nov 02 14:56:41.238195 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.238186    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/03b3012d-0bdf-4a78-ad31-c03d1df51ec0-root\") pod \"node-exporter-gpjvx\" (UID: \"03b3012d-0bdf-4a78-ad31-c03d1df51ec0\") " pod="openshift-monitoring/node-exporter-gpjvx"
Nov 02 14:56:41.238418 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.238201    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e1da99c3-8294-4e34-9d33-47e04a6182d3-etc-tuned\") pod \"tuned-sq4l4\" (UID: \"e1da99c3-8294-4e34-9d33-47e04a6182d3\") " pod="openshift-cluster-node-tuning-operator/tuned-sq4l4"
Nov 02 14:56:41.238418 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.238206    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/65ce39fc-05c7-433b-81f4-c1b219f1bd71-host-cni-netd\") pod \"ovnkube-node-f8btv\" (UID: \"65ce39fc-05c7-433b-81f4-c1b219f1bd71\") " pod="openshift-ovn-kubernetes/ovnkube-node-f8btv"
Nov 02 14:56:41.238418 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.238233    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e4983b70-40f4-4e1e-a5da-80194d5d4786-serviceca\") pod \"node-ca-q9nx7\" (UID: \"e4983b70-40f4-4e1e-a5da-80194d5d4786\") " pod="openshift-image-registry/node-ca-q9nx7"
Nov 02 14:56:41.238418 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.238336    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/702f7864-7db1-40d5-ba62-33cb9477b7da-data-volume\") pod \"insights-runtime-extractor-k6fjk\" (UID: \"702f7864-7db1-40d5-ba62-33cb9477b7da\") " pod="openshift-insights/insights-runtime-extractor-k6fjk"
Nov 02 14:56:41.238418 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.238383    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/54dc6898-f041-4ed6-9fc9-1b5159d00d27-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6cj82\" (UID: \"54dc6898-f041-4ed6-9fc9-1b5159d00d27\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6cj82"
Nov 02 14:56:41.238418 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.238412    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/702f7864-7db1-40d5-ba62-33cb9477b7da-data-volume\") pod \"insights-runtime-extractor-k6fjk\" (UID: \"702f7864-7db1-40d5-ba62-33cb9477b7da\") " pod="openshift-insights/insights-runtime-extractor-k6fjk"
Nov 02 14:56:41.238618 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.238420    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/54dc6898-f041-4ed6-9fc9-1b5159d00d27-etc-selinux\") pod \"aws-ebs-csi-driver-node-6cj82\" (UID: \"54dc6898-f041-4ed6-9fc9-1b5159d00d27\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6cj82"
Nov 02 14:56:41.238618 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.238446    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/93d7d50e-b320-4217-b183-2bfc921f6c9d-cni-binary-copy\") pod \"multus-7slzh\" (UID: \"93d7d50e-b320-4217-b183-2bfc921f6c9d\") " pod="openshift-multus/multus-7slzh"
Nov 02 14:56:41.238618 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.238471    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hpfk5\" (UniqueName: \"kubernetes.io/projected/0d6c5872-e278-4463-953d-e1388a001d0d-kube-api-access-hpfk5\") pod \"network-check-target-wns8r\" (UID: \"0d6c5872-e278-4463-953d-e1388a001d0d\") " pod="openshift-network-diagnostics/network-check-target-wns8r"
Nov 02 14:56:41.238618 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.238500    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/03b3012d-0bdf-4a78-ad31-c03d1df51ec0-node-exporter-tls\") pod \"node-exporter-gpjvx\" (UID: \"03b3012d-0bdf-4a78-ad31-c03d1df51ec0\") " pod="openshift-monitoring/node-exporter-gpjvx"
Nov 02 14:56:41.238618 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.238541    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/03b3012d-0bdf-4a78-ad31-c03d1df51ec0-node-exporter-accelerators-collector-config\") pod \"node-exporter-gpjvx\" (UID: \"03b3012d-0bdf-4a78-ad31-c03d1df51ec0\") " pod="openshift-monitoring/node-exporter-gpjvx"
Nov 02 14:56:41.238618 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.238612    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/65ce39fc-05c7-433b-81f4-c1b219f1bd71-var-lib-openvswitch\") pod \"ovnkube-node-f8btv\" (UID: \"65ce39fc-05c7-433b-81f4-c1b219f1bd71\") " pod="openshift-ovn-kubernetes/ovnkube-node-f8btv"
Nov 02 14:56:41.238868 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.238637    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/65ce39fc-05c7-433b-81f4-c1b219f1bd71-etc-openvswitch\") pod \"ovnkube-node-f8btv\" (UID: \"65ce39fc-05c7-433b-81f4-c1b219f1bd71\") " pod="openshift-ovn-kubernetes/ovnkube-node-f8btv"
Nov 02 14:56:41.238868 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.238643    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/93d7d50e-b320-4217-b183-2bfc921f6c9d-cni-binary-copy\") pod \"multus-7slzh\" (UID: \"93d7d50e-b320-4217-b183-2bfc921f6c9d\") " pod="openshift-multus/multus-7slzh"
Nov 02 14:56:41.238868 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.238666    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e4983b70-40f4-4e1e-a5da-80194d5d4786-host\") pod \"node-ca-q9nx7\" (UID: \"e4983b70-40f4-4e1e-a5da-80194d5d4786\") " pod="openshift-image-registry/node-ca-q9nx7"
Nov 02 14:56:41.238868 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.238692    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b1715623-afd0-4d0e-9b63-b38499cec1e2-tuning-conf-dir\") pod \"multus-additional-cni-plugins-fbnnl\" (UID: \"b1715623-afd0-4d0e-9b63-b38499cec1e2\") " pod="openshift-multus/multus-additional-cni-plugins-fbnnl"
Nov 02 14:56:41.238868 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.238718    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/93d7d50e-b320-4217-b183-2bfc921f6c9d-multus-socket-dir-parent\") pod \"multus-7slzh\" (UID: \"93d7d50e-b320-4217-b183-2bfc921f6c9d\") " pod="openshift-multus/multus-7slzh"
Nov 02 14:56:41.238868 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.238722    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e4983b70-40f4-4e1e-a5da-80194d5d4786-serviceca\") pod \"node-ca-q9nx7\" (UID: \"e4983b70-40f4-4e1e-a5da-80194d5d4786\") " pod="openshift-image-registry/node-ca-q9nx7"
Nov 02 14:56:41.238868 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.238741    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/93d7d50e-b320-4217-b183-2bfc921f6c9d-etc-kubernetes\") pod \"multus-7slzh\" (UID: \"93d7d50e-b320-4217-b183-2bfc921f6c9d\") " pod="openshift-multus/multus-7slzh"
Nov 02 14:56:41.238868 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.238822    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/03b3012d-0bdf-4a78-ad31-c03d1df51ec0-node-exporter-accelerators-collector-config\") pod \"node-exporter-gpjvx\" (UID: \"03b3012d-0bdf-4a78-ad31-c03d1df51ec0\") " pod="openshift-monitoring/node-exporter-gpjvx"
Nov 02 14:56:41.238868 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.238850    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e1da99c3-8294-4e34-9d33-47e04a6182d3-etc-kubernetes\") pod \"tuned-sq4l4\" (UID: \"e1da99c3-8294-4e34-9d33-47e04a6182d3\") " pod="openshift-cluster-node-tuning-operator/tuned-sq4l4"
Nov 02 14:56:41.239266 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.238885    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8rqfc\" (UniqueName: \"kubernetes.io/projected/7fe9ba5e-a867-4ac3-8a6f-a316c6f64db4-kube-api-access-8rqfc\") pod \"iptables-alerter-pl8gn\" (UID: \"7fe9ba5e-a867-4ac3-8a6f-a316c6f64db4\") " pod="openshift-network-operator/iptables-alerter-pl8gn"
Nov 02 14:56:41.239266 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.238905    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/65ce39fc-05c7-433b-81f4-c1b219f1bd71-host-run-ovn-kubernetes\") pod \"ovnkube-node-f8btv\" (UID: \"65ce39fc-05c7-433b-81f4-c1b219f1bd71\") " pod="openshift-ovn-kubernetes/ovnkube-node-f8btv"
Nov 02 14:56:41.239266 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.238944    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/332f4a2d-a241-4c71-8ebb-a99618c0d3dc-kubelet-config\") pod \"global-pull-secret-syncer-vl6zb\" (UID: \"332f4a2d-a241-4c71-8ebb-a99618c0d3dc\") " pod="kube-system/global-pull-secret-syncer-vl6zb"
Nov 02 14:56:41.239266 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.238964    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/68b781b2-be96-4c5c-8daa-2238fd0a9623-konnectivity-ca\") pod \"konnectivity-agent-d5t6w\" (UID: \"68b781b2-be96-4c5c-8daa-2238fd0a9623\") " pod="kube-system/konnectivity-agent-d5t6w"
Nov 02 14:56:41.239266 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.238997    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/702f7864-7db1-40d5-ba62-33cb9477b7da-crio-socket\") pod \"insights-runtime-extractor-k6fjk\" (UID: \"702f7864-7db1-40d5-ba62-33cb9477b7da\") " pod="openshift-insights/insights-runtime-extractor-k6fjk"
Nov 02 14:56:41.239266 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.238944    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/03b3012d-0bdf-4a78-ad31-c03d1df51ec0-node-exporter-tls\") pod \"node-exporter-gpjvx\" (UID: \"03b3012d-0bdf-4a78-ad31-c03d1df51ec0\") " pod="openshift-monitoring/node-exporter-gpjvx"
Nov 02 14:56:41.239266 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.239015    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/93d7d50e-b320-4217-b183-2bfc921f6c9d-hostroot\") pod \"multus-7slzh\" (UID: \"93d7d50e-b320-4217-b183-2bfc921f6c9d\") " pod="openshift-multus/multus-7slzh"
Nov 02 14:56:41.239266 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.239050    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/080a338e-be02-49fb-bebe-f873e3f4017e-tmp-dir\") pod \"dns-default-kfnkt\" (UID: \"080a338e-be02-49fb-bebe-f873e3f4017e\") " pod="openshift-dns/dns-default-kfnkt"
Nov 02 14:56:41.239266 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.239079    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/702f7864-7db1-40d5-ba62-33cb9477b7da-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-k6fjk\" (UID: \"702f7864-7db1-40d5-ba62-33cb9477b7da\") " pod="openshift-insights/insights-runtime-extractor-k6fjk"
Nov 02 14:56:41.239266 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.239104    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/54dc6898-f041-4ed6-9fc9-1b5159d00d27-sys-fs\") pod \"aws-ebs-csi-driver-node-6cj82\" (UID: \"54dc6898-f041-4ed6-9fc9-1b5159d00d27\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6cj82"
Nov 02 14:56:41.239266 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.239146    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8bt8x\" (UniqueName: \"kubernetes.io/projected/54dc6898-f041-4ed6-9fc9-1b5159d00d27-kube-api-access-8bt8x\") pod \"aws-ebs-csi-driver-node-6cj82\" (UID: \"54dc6898-f041-4ed6-9fc9-1b5159d00d27\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6cj82"
Nov 02 14:56:41.239266 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.239166    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e1da99c3-8294-4e34-9d33-47e04a6182d3-etc-sysctl-d\") pod \"tuned-sq4l4\" (UID: \"e1da99c3-8294-4e34-9d33-47e04a6182d3\") " pod="openshift-cluster-node-tuning-operator/tuned-sq4l4"
Nov 02 14:56:41.239266 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.239203    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e1da99c3-8294-4e34-9d33-47e04a6182d3-etc-systemd\") pod \"tuned-sq4l4\" (UID: \"e1da99c3-8294-4e34-9d33-47e04a6182d3\") " pod="openshift-cluster-node-tuning-operator/tuned-sq4l4"
Nov 02 14:56:41.239266 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.239211    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/080a338e-be02-49fb-bebe-f873e3f4017e-tmp-dir\") pod \"dns-default-kfnkt\" (UID: \"080a338e-be02-49fb-bebe-f873e3f4017e\") " pod="openshift-dns/dns-default-kfnkt"
Nov 02 14:56:41.239266 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.239227    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e1da99c3-8294-4e34-9d33-47e04a6182d3-var-lib-kubelet\") pod \"tuned-sq4l4\" (UID: \"e1da99c3-8294-4e34-9d33-47e04a6182d3\") " pod="openshift-cluster-node-tuning-operator/tuned-sq4l4"
Nov 02 14:56:41.240137 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.239277    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/65ce39fc-05c7-433b-81f4-c1b219f1bd71-log-socket\") pod \"ovnkube-node-f8btv\" (UID: \"65ce39fc-05c7-433b-81f4-c1b219f1bd71\") " pod="openshift-ovn-kubernetes/ovnkube-node-f8btv"
Nov 02 14:56:41.240137 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.239312    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/65ce39fc-05c7-433b-81f4-c1b219f1bd71-host-cni-bin\") pod \"ovnkube-node-f8btv\" (UID: \"65ce39fc-05c7-433b-81f4-c1b219f1bd71\") " pod="openshift-ovn-kubernetes/ovnkube-node-f8btv"
Nov 02 14:56:41.240137 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.239296    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/68b781b2-be96-4c5c-8daa-2238fd0a9623-konnectivity-ca\") pod \"konnectivity-agent-d5t6w\" (UID: \"68b781b2-be96-4c5c-8daa-2238fd0a9623\") " pod="kube-system/konnectivity-agent-d5t6w"
Nov 02 14:56:41.240137 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.239335    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/702f7864-7db1-40d5-ba62-33cb9477b7da-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-k6fjk\" (UID: \"702f7864-7db1-40d5-ba62-33cb9477b7da\") " pod="openshift-insights/insights-runtime-extractor-k6fjk"
Nov 02 14:56:41.240137 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.239335    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/65ce39fc-05c7-433b-81f4-c1b219f1bd71-ovnkube-config\") pod \"ovnkube-node-f8btv\" (UID: \"65ce39fc-05c7-433b-81f4-c1b219f1bd71\") " pod="openshift-ovn-kubernetes/ovnkube-node-f8btv"
Nov 02 14:56:41.240137 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.239401    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ps49c\" (UniqueName: \"kubernetes.io/projected/65ce39fc-05c7-433b-81f4-c1b219f1bd71-kube-api-access-ps49c\") pod \"ovnkube-node-f8btv\" (UID: \"65ce39fc-05c7-433b-81f4-c1b219f1bd71\") " pod="openshift-ovn-kubernetes/ovnkube-node-f8btv"
Nov 02 14:56:41.240137 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.239441    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e1da99c3-8294-4e34-9d33-47e04a6182d3-etc-modprobe-d\") pod \"tuned-sq4l4\" (UID: \"e1da99c3-8294-4e34-9d33-47e04a6182d3\") " pod="openshift-cluster-node-tuning-operator/tuned-sq4l4"
Nov 02 14:56:41.240137 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.239535    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/65ce39fc-05c7-433b-81f4-c1b219f1bd71-ovnkube-config\") pod \"ovnkube-node-f8btv\" (UID: \"65ce39fc-05c7-433b-81f4-c1b219f1bd71\") " pod="openshift-ovn-kubernetes/ovnkube-node-f8btv"
Nov 02 14:56:41.240137 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.239555    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/03b3012d-0bdf-4a78-ad31-c03d1df51ec0-node-exporter-textfile\") pod \"node-exporter-gpjvx\" (UID: \"03b3012d-0bdf-4a78-ad31-c03d1df51ec0\") " pod="openshift-monitoring/node-exporter-gpjvx"
Nov 02 14:56:41.240137 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.239583    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/65ce39fc-05c7-433b-81f4-c1b219f1bd71-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-f8btv\" (UID: \"65ce39fc-05c7-433b-81f4-c1b219f1bd71\") " pod="openshift-ovn-kubernetes/ovnkube-node-f8btv"
Nov 02 14:56:41.240137 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.239604    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/720ef290-024b-4ed2-8417-92290f360dc9-hosts-file\") pod \"node-resolver-jpz27\" (UID: \"720ef290-024b-4ed2-8417-92290f360dc9\") " pod="openshift-dns/node-resolver-jpz27"
Nov 02 14:56:41.240137 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.239629    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/03b3012d-0bdf-4a78-ad31-c03d1df51ec0-node-exporter-textfile\") pod \"node-exporter-gpjvx\" (UID: \"03b3012d-0bdf-4a78-ad31-c03d1df51ec0\") " pod="openshift-monitoring/node-exporter-gpjvx"
Nov 02 14:56:41.240137 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.239649    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/332f4a2d-a241-4c71-8ebb-a99618c0d3dc-original-pull-secret\") pod \"global-pull-secret-syncer-vl6zb\" (UID: \"332f4a2d-a241-4c71-8ebb-a99618c0d3dc\") " pod="kube-system/global-pull-secret-syncer-vl6zb"
Nov 02 14:56:41.240137 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.239669    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r67xl\" (UniqueName: \"kubernetes.io/projected/080a338e-be02-49fb-bebe-f873e3f4017e-kube-api-access-r67xl\") pod \"dns-default-kfnkt\" (UID: \"080a338e-be02-49fb-bebe-f873e3f4017e\") " pod="openshift-dns/dns-default-kfnkt"
Nov 02 14:56:41.240137 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.239694    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/54dc6898-f041-4ed6-9fc9-1b5159d00d27-device-dir\") pod \"aws-ebs-csi-driver-node-6cj82\" (UID: \"54dc6898-f041-4ed6-9fc9-1b5159d00d27\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6cj82"
Nov 02 14:56:41.240137 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.239709    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/93d7d50e-b320-4217-b183-2bfc921f6c9d-host-var-lib-cni-bin\") pod \"multus-7slzh\" (UID: \"93d7d50e-b320-4217-b183-2bfc921f6c9d\") " pod="openshift-multus/multus-7slzh"
Nov 02 14:56:41.240137 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.239726    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ndjwt\" (UniqueName: \"kubernetes.io/projected/93d7d50e-b320-4217-b183-2bfc921f6c9d-kube-api-access-ndjwt\") pod \"multus-7slzh\" (UID: \"93d7d50e-b320-4217-b183-2bfc921f6c9d\") " pod="openshift-multus/multus-7slzh"
Nov 02 14:56:41.240137 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.239980    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/332f4a2d-a241-4c71-8ebb-a99618c0d3dc-original-pull-secret\") pod \"global-pull-secret-syncer-vl6zb\" (UID: \"332f4a2d-a241-4c71-8ebb-a99618c0d3dc\") " pod="kube-system/global-pull-secret-syncer-vl6zb"
Nov 02 14:56:41.260145 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.260093    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-548bv\" (UniqueName: \"kubernetes.io/projected/b1715623-afd0-4d0e-9b63-b38499cec1e2-kube-api-access-548bv\") pod \"multus-additional-cni-plugins-fbnnl\" (UID: \"b1715623-afd0-4d0e-9b63-b38499cec1e2\") " pod="openshift-multus/multus-additional-cni-plugins-fbnnl"
Nov 02 14:56:41.260297 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.260106    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9vxjj\" (UniqueName: \"kubernetes.io/projected/03b3012d-0bdf-4a78-ad31-c03d1df51ec0-kube-api-access-9vxjj\") pod \"node-exporter-gpjvx\" (UID: \"03b3012d-0bdf-4a78-ad31-c03d1df51ec0\") " pod="openshift-monitoring/node-exporter-gpjvx"
Nov 02 14:56:41.260297 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.260174    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hz5t9\" (UniqueName: \"kubernetes.io/projected/caa8898b-6611-49b8-845d-3b9bdac6691f-kube-api-access-hz5t9\") pod \"network-metrics-daemon-484ld\" (UID: \"caa8898b-6611-49b8-845d-3b9bdac6691f\") " pod="openshift-multus/network-metrics-daemon-484ld"
Nov 02 14:56:41.260297 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.260157    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-prmck\" (UniqueName: \"kubernetes.io/projected/e1da99c3-8294-4e34-9d33-47e04a6182d3-kube-api-access-prmck\") pod \"tuned-sq4l4\" (UID: \"e1da99c3-8294-4e34-9d33-47e04a6182d3\") " pod="openshift-cluster-node-tuning-operator/tuned-sq4l4"
Nov 02 14:56:41.260297 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.260239    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f28t5\" (UniqueName: \"kubernetes.io/projected/720ef290-024b-4ed2-8417-92290f360dc9-kube-api-access-f28t5\") pod \"node-resolver-jpz27\" (UID: \"720ef290-024b-4ed2-8417-92290f360dc9\") " pod="openshift-dns/node-resolver-jpz27"
Nov 02 14:56:41.260552 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.260442    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ghkdw\" (UniqueName: \"kubernetes.io/projected/e4f9e55c-3232-430c-a123-30d17704cd2e-kube-api-access-ghkdw\") pod \"ingress-canary-fw8qc\" (UID: \"e4f9e55c-3232-430c-a123-30d17704cd2e\") " pod="openshift-ingress-canary/ingress-canary-fw8qc"
Nov 02 14:56:41.260805 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.260788    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7758v\" (UniqueName: \"kubernetes.io/projected/e4983b70-40f4-4e1e-a5da-80194d5d4786-kube-api-access-7758v\") pod \"node-ca-q9nx7\" (UID: \"e4983b70-40f4-4e1e-a5da-80194d5d4786\") " pod="openshift-image-registry/node-ca-q9nx7"
Nov 02 14:56:41.263237 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.263218    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8bt8x\" (UniqueName: \"kubernetes.io/projected/54dc6898-f041-4ed6-9fc9-1b5159d00d27-kube-api-access-8bt8x\") pod \"aws-ebs-csi-driver-node-6cj82\" (UID: \"54dc6898-f041-4ed6-9fc9-1b5159d00d27\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6cj82"
Nov 02 14:56:41.264933 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.264915    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r67xl\" (UniqueName: \"kubernetes.io/projected/080a338e-be02-49fb-bebe-f873e3f4017e-kube-api-access-r67xl\") pod \"dns-default-kfnkt\" (UID: \"080a338e-be02-49fb-bebe-f873e3f4017e\") " pod="openshift-dns/dns-default-kfnkt"
Nov 02 14:56:41.265215 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.265199    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ndjwt\" (UniqueName: \"kubernetes.io/projected/93d7d50e-b320-4217-b183-2bfc921f6c9d-kube-api-access-ndjwt\") pod \"multus-7slzh\" (UID: \"93d7d50e-b320-4217-b183-2bfc921f6c9d\") " pod="openshift-multus/multus-7slzh"
Nov 02 14:56:41.265719 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.265702    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ps49c\" (UniqueName: \"kubernetes.io/projected/65ce39fc-05c7-433b-81f4-c1b219f1bd71-kube-api-access-ps49c\") pod \"ovnkube-node-f8btv\" (UID: \"65ce39fc-05c7-433b-81f4-c1b219f1bd71\") " pod="openshift-ovn-kubernetes/ovnkube-node-f8btv"
Nov 02 14:56:41.266446 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.266429    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kw44n\" (UniqueName: \"kubernetes.io/projected/702f7864-7db1-40d5-ba62-33cb9477b7da-kube-api-access-kw44n\") pod \"insights-runtime-extractor-k6fjk\" (UID: \"702f7864-7db1-40d5-ba62-33cb9477b7da\") " pod="openshift-insights/insights-runtime-extractor-k6fjk"
Nov 02 14:56:41.269806 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.269788    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hpfk5\" (UniqueName: \"kubernetes.io/projected/0d6c5872-e278-4463-953d-e1388a001d0d-kube-api-access-hpfk5\") pod \"network-check-target-wns8r\" (UID: \"0d6c5872-e278-4463-953d-e1388a001d0d\") " pod="openshift-network-diagnostics/network-check-target-wns8r"
Nov 02 14:56:41.271303 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.271282    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8rqfc\" (UniqueName: \"kubernetes.io/projected/7fe9ba5e-a867-4ac3-8a6f-a316c6f64db4-kube-api-access-8rqfc\") pod \"iptables-alerter-pl8gn\" (UID: \"7fe9ba5e-a867-4ac3-8a6f-a316c6f64db4\") " pod="openshift-network-operator/iptables-alerter-pl8gn"
Nov 02 14:56:41.340877 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.340846    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/54dc6898-f041-4ed6-9fc9-1b5159d00d27-device-dir\") pod \"aws-ebs-csi-driver-node-6cj82\" (UID: \"54dc6898-f041-4ed6-9fc9-1b5159d00d27\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6cj82"
Nov 02 14:56:41.341033 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.340899    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/93d7d50e-b320-4217-b183-2bfc921f6c9d-host-var-lib-cni-bin\") pod \"multus-7slzh\" (UID: \"93d7d50e-b320-4217-b183-2bfc921f6c9d\") " pod="openshift-multus/multus-7slzh"
Nov 02 14:56:41.341033 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.340951    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/93d7d50e-b320-4217-b183-2bfc921f6c9d-host-var-lib-cni-bin\") pod \"multus-7slzh\" (UID: \"93d7d50e-b320-4217-b183-2bfc921f6c9d\") " pod="openshift-multus/multus-7slzh"
Nov 02 14:56:41.341033 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.340959    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/54dc6898-f041-4ed6-9fc9-1b5159d00d27-device-dir\") pod \"aws-ebs-csi-driver-node-6cj82\" (UID: \"54dc6898-f041-4ed6-9fc9-1b5159d00d27\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6cj82"
Nov 02 14:56:41.341033 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.340982    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/93d7d50e-b320-4217-b183-2bfc921f6c9d-multus-cni-dir\") pod \"multus-7slzh\" (UID: \"93d7d50e-b320-4217-b183-2bfc921f6c9d\") " pod="openshift-multus/multus-7slzh"
Nov 02 14:56:41.341033 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.341015    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/93d7d50e-b320-4217-b183-2bfc921f6c9d-host-var-lib-cni-multus\") pod \"multus-7slzh\" (UID: \"93d7d50e-b320-4217-b183-2bfc921f6c9d\") " pod="openshift-multus/multus-7slzh"
Nov 02 14:56:41.341291 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.341036    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e1da99c3-8294-4e34-9d33-47e04a6182d3-host\") pod \"tuned-sq4l4\" (UID: \"e1da99c3-8294-4e34-9d33-47e04a6182d3\") " pod="openshift-cluster-node-tuning-operator/tuned-sq4l4"
Nov 02 14:56:41.341291 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.341071    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b1715623-afd0-4d0e-9b63-b38499cec1e2-cnibin\") pod \"multus-additional-cni-plugins-fbnnl\" (UID: \"b1715623-afd0-4d0e-9b63-b38499cec1e2\") " pod="openshift-multus/multus-additional-cni-plugins-fbnnl"
Nov 02 14:56:41.341291 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.341090    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/93d7d50e-b320-4217-b183-2bfc921f6c9d-host-var-lib-cni-multus\") pod \"multus-7slzh\" (UID: \"93d7d50e-b320-4217-b183-2bfc921f6c9d\") " pod="openshift-multus/multus-7slzh"
Nov 02 14:56:41.341291 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.341097    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/93d7d50e-b320-4217-b183-2bfc921f6c9d-host-run-multus-certs\") pod \"multus-7slzh\" (UID: \"93d7d50e-b320-4217-b183-2bfc921f6c9d\") " pod="openshift-multus/multus-7slzh"
Nov 02 14:56:41.341291 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.341120    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e1da99c3-8294-4e34-9d33-47e04a6182d3-host\") pod \"tuned-sq4l4\" (UID: \"e1da99c3-8294-4e34-9d33-47e04a6182d3\") " pod="openshift-cluster-node-tuning-operator/tuned-sq4l4"
Nov 02 14:56:41.341291 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.341145    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b1715623-afd0-4d0e-9b63-b38499cec1e2-cnibin\") pod \"multus-additional-cni-plugins-fbnnl\" (UID: \"b1715623-afd0-4d0e-9b63-b38499cec1e2\") " pod="openshift-multus/multus-additional-cni-plugins-fbnnl"
Nov 02 14:56:41.341291 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.341143    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/65ce39fc-05c7-433b-81f4-c1b219f1bd71-systemd-units\") pod \"ovnkube-node-f8btv\" (UID: \"65ce39fc-05c7-433b-81f4-c1b219f1bd71\") " pod="openshift-ovn-kubernetes/ovnkube-node-f8btv"
Nov 02 14:56:41.341291 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.341089    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/93d7d50e-b320-4217-b183-2bfc921f6c9d-multus-cni-dir\") pod \"multus-7slzh\" (UID: \"93d7d50e-b320-4217-b183-2bfc921f6c9d\") " pod="openshift-multus/multus-7slzh"
Nov 02 14:56:41.341291 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.341185    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/93d7d50e-b320-4217-b183-2bfc921f6c9d-cnibin\") pod \"multus-7slzh\" (UID: \"93d7d50e-b320-4217-b183-2bfc921f6c9d\") " pod="openshift-multus/multus-7slzh"
Nov 02 14:56:41.341291 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.341197    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/93d7d50e-b320-4217-b183-2bfc921f6c9d-host-run-multus-certs\") pod \"multus-7slzh\" (UID: \"93d7d50e-b320-4217-b183-2bfc921f6c9d\") " pod="openshift-multus/multus-7slzh"
Nov 02 14:56:41.341291 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.341202    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/93d7d50e-b320-4217-b183-2bfc921f6c9d-host-run-k8s-cni-cncf-io\") pod \"multus-7slzh\" (UID: \"93d7d50e-b320-4217-b183-2bfc921f6c9d\") " pod="openshift-multus/multus-7slzh"
Nov 02 14:56:41.341291 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.341221    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/93d7d50e-b320-4217-b183-2bfc921f6c9d-host-run-k8s-cni-cncf-io\") pod \"multus-7slzh\" (UID: \"93d7d50e-b320-4217-b183-2bfc921f6c9d\") " pod="openshift-multus/multus-7slzh"
Nov 02 14:56:41.341291 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.341239    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/93d7d50e-b320-4217-b183-2bfc921f6c9d-host-var-lib-kubelet\") pod \"multus-7slzh\" (UID: \"93d7d50e-b320-4217-b183-2bfc921f6c9d\") " pod="openshift-multus/multus-7slzh"
Nov 02 14:56:41.341291 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.341257    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/65ce39fc-05c7-433b-81f4-c1b219f1bd71-systemd-units\") pod \"ovnkube-node-f8btv\" (UID: \"65ce39fc-05c7-433b-81f4-c1b219f1bd71\") " pod="openshift-ovn-kubernetes/ovnkube-node-f8btv"
Nov 02 14:56:41.341291 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.341262    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/03b3012d-0bdf-4a78-ad31-c03d1df51ec0-node-exporter-wtmp\") pod \"node-exporter-gpjvx\" (UID: \"03b3012d-0bdf-4a78-ad31-c03d1df51ec0\") " pod="openshift-monitoring/node-exporter-gpjvx"
Nov 02 14:56:41.341291 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.341279    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/93d7d50e-b320-4217-b183-2bfc921f6c9d-cnibin\") pod \"multus-7slzh\" (UID: \"93d7d50e-b320-4217-b183-2bfc921f6c9d\") " pod="openshift-multus/multus-7slzh"
Nov 02 14:56:41.341291 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.341297    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/03b3012d-0bdf-4a78-ad31-c03d1df51ec0-node-exporter-wtmp\") pod \"node-exporter-gpjvx\" (UID: \"03b3012d-0bdf-4a78-ad31-c03d1df51ec0\") " pod="openshift-monitoring/node-exporter-gpjvx"
Nov 02 14:56:41.341989 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.341302    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b1715623-afd0-4d0e-9b63-b38499cec1e2-system-cni-dir\") pod \"multus-additional-cni-plugins-fbnnl\" (UID: \"b1715623-afd0-4d0e-9b63-b38499cec1e2\") " pod="openshift-multus/multus-additional-cni-plugins-fbnnl"
Nov 02 14:56:41.341989 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.341316    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/93d7d50e-b320-4217-b183-2bfc921f6c9d-host-var-lib-kubelet\") pod \"multus-7slzh\" (UID: \"93d7d50e-b320-4217-b183-2bfc921f6c9d\") " pod="openshift-multus/multus-7slzh"
Nov 02 14:56:41.341989 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.341327    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b1715623-afd0-4d0e-9b63-b38499cec1e2-system-cni-dir\") pod \"multus-additional-cni-plugins-fbnnl\" (UID: \"b1715623-afd0-4d0e-9b63-b38499cec1e2\") " pod="openshift-multus/multus-additional-cni-plugins-fbnnl"
Nov 02 14:56:41.341989 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.341333    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/93d7d50e-b320-4217-b183-2bfc921f6c9d-host-run-netns\") pod \"multus-7slzh\" (UID: \"93d7d50e-b320-4217-b183-2bfc921f6c9d\") " pod="openshift-multus/multus-7slzh"
Nov 02 14:56:41.341989 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.341354    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7fe9ba5e-a867-4ac3-8a6f-a316c6f64db4-host-slash\") pod \"iptables-alerter-pl8gn\" (UID: \"7fe9ba5e-a867-4ac3-8a6f-a316c6f64db4\") " pod="openshift-network-operator/iptables-alerter-pl8gn"
Nov 02 14:56:41.341989 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.341369    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/65ce39fc-05c7-433b-81f4-c1b219f1bd71-run-openvswitch\") pod \"ovnkube-node-f8btv\" (UID: \"65ce39fc-05c7-433b-81f4-c1b219f1bd71\") " pod="openshift-ovn-kubernetes/ovnkube-node-f8btv"
Nov 02 14:56:41.341989 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.341384    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/65ce39fc-05c7-433b-81f4-c1b219f1bd71-run-ovn\") pod \"ovnkube-node-f8btv\" (UID: \"65ce39fc-05c7-433b-81f4-c1b219f1bd71\") " pod="openshift-ovn-kubernetes/ovnkube-node-f8btv"
Nov 02 14:56:41.341989 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.341418    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7fe9ba5e-a867-4ac3-8a6f-a316c6f64db4-host-slash\") pod \"iptables-alerter-pl8gn\" (UID: \"7fe9ba5e-a867-4ac3-8a6f-a316c6f64db4\") " pod="openshift-network-operator/iptables-alerter-pl8gn"
Nov 02 14:56:41.341989 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.341419    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/54dc6898-f041-4ed6-9fc9-1b5159d00d27-socket-dir\") pod \"aws-ebs-csi-driver-node-6cj82\" (UID: \"54dc6898-f041-4ed6-9fc9-1b5159d00d27\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6cj82"
Nov 02 14:56:41.341989 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.341436    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/65ce39fc-05c7-433b-81f4-c1b219f1bd71-run-openvswitch\") pod \"ovnkube-node-f8btv\" (UID: \"65ce39fc-05c7-433b-81f4-c1b219f1bd71\") " pod="openshift-ovn-kubernetes/ovnkube-node-f8btv"
Nov 02 14:56:41.341989 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.341434    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/93d7d50e-b320-4217-b183-2bfc921f6c9d-host-run-netns\") pod \"multus-7slzh\" (UID: \"93d7d50e-b320-4217-b183-2bfc921f6c9d\") " pod="openshift-multus/multus-7slzh"
Nov 02 14:56:41.341989 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.341461    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/65ce39fc-05c7-433b-81f4-c1b219f1bd71-run-ovn\") pod \"ovnkube-node-f8btv\" (UID: \"65ce39fc-05c7-433b-81f4-c1b219f1bd71\") " pod="openshift-ovn-kubernetes/ovnkube-node-f8btv"
Nov 02 14:56:41.341989 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.341467    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/54dc6898-f041-4ed6-9fc9-1b5159d00d27-socket-dir\") pod \"aws-ebs-csi-driver-node-6cj82\" (UID: \"54dc6898-f041-4ed6-9fc9-1b5159d00d27\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6cj82"
Nov 02 14:56:41.341989 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.341461    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/54dc6898-f041-4ed6-9fc9-1b5159d00d27-registration-dir\") pod \"aws-ebs-csi-driver-node-6cj82\" (UID: \"54dc6898-f041-4ed6-9fc9-1b5159d00d27\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6cj82"
Nov 02 14:56:41.341989 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.341497    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/54dc6898-f041-4ed6-9fc9-1b5159d00d27-registration-dir\") pod \"aws-ebs-csi-driver-node-6cj82\" (UID: \"54dc6898-f041-4ed6-9fc9-1b5159d00d27\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6cj82"
Nov 02 14:56:41.341989 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.341505    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/93d7d50e-b320-4217-b183-2bfc921f6c9d-multus-conf-dir\") pod \"multus-7slzh\" (UID: \"93d7d50e-b320-4217-b183-2bfc921f6c9d\") " pod="openshift-multus/multus-7slzh"
Nov 02 14:56:41.341989 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.341522    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e1da99c3-8294-4e34-9d33-47e04a6182d3-etc-sysconfig\") pod \"tuned-sq4l4\" (UID: \"e1da99c3-8294-4e34-9d33-47e04a6182d3\") " pod="openshift-cluster-node-tuning-operator/tuned-sq4l4"
Nov 02 14:56:41.341989 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.341556    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/93d7d50e-b320-4217-b183-2bfc921f6c9d-multus-conf-dir\") pod \"multus-7slzh\" (UID: \"93d7d50e-b320-4217-b183-2bfc921f6c9d\") " pod="openshift-multus/multus-7slzh"
Nov 02 14:56:41.341989 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.341560    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e1da99c3-8294-4e34-9d33-47e04a6182d3-etc-sysconfig\") pod \"tuned-sq4l4\" (UID: \"e1da99c3-8294-4e34-9d33-47e04a6182d3\") " pod="openshift-cluster-node-tuning-operator/tuned-sq4l4"
Nov 02 14:56:41.341989 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.341584    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e1da99c3-8294-4e34-9d33-47e04a6182d3-run\") pod \"tuned-sq4l4\" (UID: \"e1da99c3-8294-4e34-9d33-47e04a6182d3\") " pod="openshift-cluster-node-tuning-operator/tuned-sq4l4"
Nov 02 14:56:41.341989 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.341601    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/65ce39fc-05c7-433b-81f4-c1b219f1bd71-host-kubelet\") pod \"ovnkube-node-f8btv\" (UID: \"65ce39fc-05c7-433b-81f4-c1b219f1bd71\") " pod="openshift-ovn-kubernetes/ovnkube-node-f8btv"
Nov 02 14:56:41.341989 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.341620    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/65ce39fc-05c7-433b-81f4-c1b219f1bd71-host-run-netns\") pod \"ovnkube-node-f8btv\" (UID: \"65ce39fc-05c7-433b-81f4-c1b219f1bd71\") " pod="openshift-ovn-kubernetes/ovnkube-node-f8btv"
Nov 02 14:56:41.341989 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.341635    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/65ce39fc-05c7-433b-81f4-c1b219f1bd71-run-systemd\") pod \"ovnkube-node-f8btv\" (UID: \"65ce39fc-05c7-433b-81f4-c1b219f1bd71\") " pod="openshift-ovn-kubernetes/ovnkube-node-f8btv"
Nov 02 14:56:41.341989 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.341660    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/65ce39fc-05c7-433b-81f4-c1b219f1bd71-host-kubelet\") pod \"ovnkube-node-f8btv\" (UID: \"65ce39fc-05c7-433b-81f4-c1b219f1bd71\") " pod="openshift-ovn-kubernetes/ovnkube-node-f8btv"
Nov 02 14:56:41.341989 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.341674    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/65ce39fc-05c7-433b-81f4-c1b219f1bd71-run-systemd\") pod \"ovnkube-node-f8btv\" (UID: \"65ce39fc-05c7-433b-81f4-c1b219f1bd71\") " pod="openshift-ovn-kubernetes/ovnkube-node-f8btv"
Nov 02 14:56:41.341989 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.341717    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e1da99c3-8294-4e34-9d33-47e04a6182d3-run\") pod \"tuned-sq4l4\" (UID: \"e1da99c3-8294-4e34-9d33-47e04a6182d3\") " pod="openshift-cluster-node-tuning-operator/tuned-sq4l4"
Nov 02 14:56:41.341989 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.341733    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/65ce39fc-05c7-433b-81f4-c1b219f1bd71-node-log\") pod \"ovnkube-node-f8btv\" (UID: \"65ce39fc-05c7-433b-81f4-c1b219f1bd71\") " pod="openshift-ovn-kubernetes/ovnkube-node-f8btv"
Nov 02 14:56:41.341989 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.341751    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/332f4a2d-a241-4c71-8ebb-a99618c0d3dc-dbus\") pod \"global-pull-secret-syncer-vl6zb\" (UID: \"332f4a2d-a241-4c71-8ebb-a99618c0d3dc\") " pod="kube-system/global-pull-secret-syncer-vl6zb"
Nov 02 14:56:41.341989 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.341765    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/03b3012d-0bdf-4a78-ad31-c03d1df51ec0-sys\") pod \"node-exporter-gpjvx\" (UID: \"03b3012d-0bdf-4a78-ad31-c03d1df51ec0\") " pod="openshift-monitoring/node-exporter-gpjvx"
Nov 02 14:56:41.341989 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.341780    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/65ce39fc-05c7-433b-81f4-c1b219f1bd71-host-slash\") pod \"ovnkube-node-f8btv\" (UID: \"65ce39fc-05c7-433b-81f4-c1b219f1bd71\") " pod="openshift-ovn-kubernetes/ovnkube-node-f8btv"
Nov 02 14:56:41.341989 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.341795    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e1da99c3-8294-4e34-9d33-47e04a6182d3-etc-sysctl-conf\") pod \"tuned-sq4l4\" (UID: \"e1da99c3-8294-4e34-9d33-47e04a6182d3\") " pod="openshift-cluster-node-tuning-operator/tuned-sq4l4"
Nov 02 14:56:41.341989 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.341808    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/332f4a2d-a241-4c71-8ebb-a99618c0d3dc-dbus\") pod \"global-pull-secret-syncer-vl6zb\" (UID: \"332f4a2d-a241-4c71-8ebb-a99618c0d3dc\") " pod="kube-system/global-pull-secret-syncer-vl6zb"
Nov 02 14:56:41.341989 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.341819    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e1da99c3-8294-4e34-9d33-47e04a6182d3-lib-modules\") pod \"tuned-sq4l4\" (UID: \"e1da99c3-8294-4e34-9d33-47e04a6182d3\") " pod="openshift-cluster-node-tuning-operator/tuned-sq4l4"
Nov 02 14:56:41.341989 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.341842    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/93d7d50e-b320-4217-b183-2bfc921f6c9d-system-cni-dir\") pod \"multus-7slzh\" (UID: \"93d7d50e-b320-4217-b183-2bfc921f6c9d\") " pod="openshift-multus/multus-7slzh"
Nov 02 14:56:41.341989 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.341849    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/65ce39fc-05c7-433b-81f4-c1b219f1bd71-host-slash\") pod \"ovnkube-node-f8btv\" (UID: \"65ce39fc-05c7-433b-81f4-c1b219f1bd71\") " pod="openshift-ovn-kubernetes/ovnkube-node-f8btv"
Nov 02 14:56:41.341989 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.341856    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e1da99c3-8294-4e34-9d33-47e04a6182d3-sys\") pod \"tuned-sq4l4\" (UID: \"e1da99c3-8294-4e34-9d33-47e04a6182d3\") " pod="openshift-cluster-node-tuning-operator/tuned-sq4l4"
Nov 02 14:56:41.341989 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.341829    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/65ce39fc-05c7-433b-81f4-c1b219f1bd71-node-log\") pod \"ovnkube-node-f8btv\" (UID: \"65ce39fc-05c7-433b-81f4-c1b219f1bd71\") " pod="openshift-ovn-kubernetes/ovnkube-node-f8btv"
Nov 02 14:56:41.341989 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.341869    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/03b3012d-0bdf-4a78-ad31-c03d1df51ec0-root\") pod \"node-exporter-gpjvx\" (UID: \"03b3012d-0bdf-4a78-ad31-c03d1df51ec0\") " pod="openshift-monitoring/node-exporter-gpjvx"
Nov 02 14:56:41.341989 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.341817    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/03b3012d-0bdf-4a78-ad31-c03d1df51ec0-sys\") pod \"node-exporter-gpjvx\" (UID: \"03b3012d-0bdf-4a78-ad31-c03d1df51ec0\") " pod="openshift-monitoring/node-exporter-gpjvx"
Nov 02 14:56:41.341989 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.341892    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/65ce39fc-05c7-433b-81f4-c1b219f1bd71-host-run-netns\") pod \"ovnkube-node-f8btv\" (UID: \"65ce39fc-05c7-433b-81f4-c1b219f1bd71\") " pod="openshift-ovn-kubernetes/ovnkube-node-f8btv"
Nov 02 14:56:41.341989 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.341894    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/65ce39fc-05c7-433b-81f4-c1b219f1bd71-host-cni-netd\") pod \"ovnkube-node-f8btv\" (UID: \"65ce39fc-05c7-433b-81f4-c1b219f1bd71\") " pod="openshift-ovn-kubernetes/ovnkube-node-f8btv"
Nov 02 14:56:41.341989 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.341902    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/93d7d50e-b320-4217-b183-2bfc921f6c9d-system-cni-dir\") pod \"multus-7slzh\" (UID: \"93d7d50e-b320-4217-b183-2bfc921f6c9d\") " pod="openshift-multus/multus-7slzh"
Nov 02 14:56:41.341989 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.341929    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/65ce39fc-05c7-433b-81f4-c1b219f1bd71-host-cni-netd\") pod \"ovnkube-node-f8btv\" (UID: \"65ce39fc-05c7-433b-81f4-c1b219f1bd71\") " pod="openshift-ovn-kubernetes/ovnkube-node-f8btv"
Nov 02 14:56:41.341989 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.341936    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e1da99c3-8294-4e34-9d33-47e04a6182d3-etc-sysctl-conf\") pod \"tuned-sq4l4\" (UID: \"e1da99c3-8294-4e34-9d33-47e04a6182d3\") " pod="openshift-cluster-node-tuning-operator/tuned-sq4l4"
Nov 02 14:56:41.341989 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.341931    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e1da99c3-8294-4e34-9d33-47e04a6182d3-lib-modules\") pod \"tuned-sq4l4\" (UID: \"e1da99c3-8294-4e34-9d33-47e04a6182d3\") " pod="openshift-cluster-node-tuning-operator/tuned-sq4l4"
Nov 02 14:56:41.341989 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.341939    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/54dc6898-f041-4ed6-9fc9-1b5159d00d27-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6cj82\" (UID: \"54dc6898-f041-4ed6-9fc9-1b5159d00d27\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6cj82"
Nov 02 14:56:41.341989 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.341949    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e1da99c3-8294-4e34-9d33-47e04a6182d3-sys\") pod \"tuned-sq4l4\" (UID: \"e1da99c3-8294-4e34-9d33-47e04a6182d3\") " pod="openshift-cluster-node-tuning-operator/tuned-sq4l4"
Nov 02 14:56:41.341989 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.341943    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/03b3012d-0bdf-4a78-ad31-c03d1df51ec0-root\") pod \"node-exporter-gpjvx\" (UID: \"03b3012d-0bdf-4a78-ad31-c03d1df51ec0\") " pod="openshift-monitoring/node-exporter-gpjvx"
Nov 02 14:56:41.341989 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.341978    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/54dc6898-f041-4ed6-9fc9-1b5159d00d27-etc-selinux\") pod \"aws-ebs-csi-driver-node-6cj82\" (UID: \"54dc6898-f041-4ed6-9fc9-1b5159d00d27\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6cj82"
Nov 02 14:56:41.341989 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.342017    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/65ce39fc-05c7-433b-81f4-c1b219f1bd71-var-lib-openvswitch\") pod \"ovnkube-node-f8btv\" (UID: \"65ce39fc-05c7-433b-81f4-c1b219f1bd71\") " pod="openshift-ovn-kubernetes/ovnkube-node-f8btv"
Nov 02 14:56:41.341989 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.342023    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/54dc6898-f041-4ed6-9fc9-1b5159d00d27-etc-selinux\") pod \"aws-ebs-csi-driver-node-6cj82\" (UID: \"54dc6898-f041-4ed6-9fc9-1b5159d00d27\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6cj82"
Nov 02 14:56:41.343516 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.342024    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/54dc6898-f041-4ed6-9fc9-1b5159d00d27-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6cj82\" (UID: \"54dc6898-f041-4ed6-9fc9-1b5159d00d27\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6cj82"
Nov 02 14:56:41.343516 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.342043    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/65ce39fc-05c7-433b-81f4-c1b219f1bd71-etc-openvswitch\") pod \"ovnkube-node-f8btv\" (UID: \"65ce39fc-05c7-433b-81f4-c1b219f1bd71\") " pod="openshift-ovn-kubernetes/ovnkube-node-f8btv"
Nov 02 14:56:41.343516 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.342060    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e4983b70-40f4-4e1e-a5da-80194d5d4786-host\") pod \"node-ca-q9nx7\" (UID: \"e4983b70-40f4-4e1e-a5da-80194d5d4786\") " pod="openshift-image-registry/node-ca-q9nx7"
Nov 02 14:56:41.343516 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.342066    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/65ce39fc-05c7-433b-81f4-c1b219f1bd71-var-lib-openvswitch\") pod \"ovnkube-node-f8btv\" (UID: \"65ce39fc-05c7-433b-81f4-c1b219f1bd71\") " pod="openshift-ovn-kubernetes/ovnkube-node-f8btv"
Nov 02 14:56:41.343516 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.342076    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b1715623-afd0-4d0e-9b63-b38499cec1e2-tuning-conf-dir\") pod \"multus-additional-cni-plugins-fbnnl\" (UID: \"b1715623-afd0-4d0e-9b63-b38499cec1e2\") " pod="openshift-multus/multus-additional-cni-plugins-fbnnl"
Nov 02 14:56:41.343516 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.342093    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/93d7d50e-b320-4217-b183-2bfc921f6c9d-multus-socket-dir-parent\") pod \"multus-7slzh\" (UID: \"93d7d50e-b320-4217-b183-2bfc921f6c9d\") " pod="openshift-multus/multus-7slzh"
Nov 02 14:56:41.343516 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.342105    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/65ce39fc-05c7-433b-81f4-c1b219f1bd71-etc-openvswitch\") pod \"ovnkube-node-f8btv\" (UID: \"65ce39fc-05c7-433b-81f4-c1b219f1bd71\") " pod="openshift-ovn-kubernetes/ovnkube-node-f8btv"
Nov 02 14:56:41.343516 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.342106    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e4983b70-40f4-4e1e-a5da-80194d5d4786-host\") pod \"node-ca-q9nx7\" (UID: \"e4983b70-40f4-4e1e-a5da-80194d5d4786\") " pod="openshift-image-registry/node-ca-q9nx7"
Nov 02 14:56:41.343516 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.342108    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/93d7d50e-b320-4217-b183-2bfc921f6c9d-etc-kubernetes\") pod \"multus-7slzh\" (UID: \"93d7d50e-b320-4217-b183-2bfc921f6c9d\") " pod="openshift-multus/multus-7slzh"
Nov 02 14:56:41.343516 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.342152    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/93d7d50e-b320-4217-b183-2bfc921f6c9d-etc-kubernetes\") pod \"multus-7slzh\" (UID: \"93d7d50e-b320-4217-b183-2bfc921f6c9d\") " pod="openshift-multus/multus-7slzh"
Nov 02 14:56:41.343516 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.342155    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b1715623-afd0-4d0e-9b63-b38499cec1e2-tuning-conf-dir\") pod \"multus-additional-cni-plugins-fbnnl\" (UID: \"b1715623-afd0-4d0e-9b63-b38499cec1e2\") " pod="openshift-multus/multus-additional-cni-plugins-fbnnl"
Nov 02 14:56:41.343516 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.342152    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/93d7d50e-b320-4217-b183-2bfc921f6c9d-multus-socket-dir-parent\") pod \"multus-7slzh\" (UID: \"93d7d50e-b320-4217-b183-2bfc921f6c9d\") " pod="openshift-multus/multus-7slzh"
Nov 02 14:56:41.343516 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.342169    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e1da99c3-8294-4e34-9d33-47e04a6182d3-etc-kubernetes\") pod \"tuned-sq4l4\" (UID: \"e1da99c3-8294-4e34-9d33-47e04a6182d3\") " pod="openshift-cluster-node-tuning-operator/tuned-sq4l4"
Nov 02 14:56:41.343516 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.342193    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/65ce39fc-05c7-433b-81f4-c1b219f1bd71-host-run-ovn-kubernetes\") pod \"ovnkube-node-f8btv\" (UID: \"65ce39fc-05c7-433b-81f4-c1b219f1bd71\") " pod="openshift-ovn-kubernetes/ovnkube-node-f8btv"
Nov 02 14:56:41.343516 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.342229    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/332f4a2d-a241-4c71-8ebb-a99618c0d3dc-kubelet-config\") pod \"global-pull-secret-syncer-vl6zb\" (UID: \"332f4a2d-a241-4c71-8ebb-a99618c0d3dc\") " pod="kube-system/global-pull-secret-syncer-vl6zb"
Nov 02 14:56:41.343516 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.342245    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e1da99c3-8294-4e34-9d33-47e04a6182d3-etc-kubernetes\") pod \"tuned-sq4l4\" (UID: \"e1da99c3-8294-4e34-9d33-47e04a6182d3\") " pod="openshift-cluster-node-tuning-operator/tuned-sq4l4"
Nov 02 14:56:41.343516 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.342250    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/702f7864-7db1-40d5-ba62-33cb9477b7da-crio-socket\") pod \"insights-runtime-extractor-k6fjk\" (UID: \"702f7864-7db1-40d5-ba62-33cb9477b7da\") " pod="openshift-insights/insights-runtime-extractor-k6fjk"
Nov 02 14:56:41.343516 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.342273    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/93d7d50e-b320-4217-b183-2bfc921f6c9d-hostroot\") pod \"multus-7slzh\" (UID: \"93d7d50e-b320-4217-b183-2bfc921f6c9d\") " pod="openshift-multus/multus-7slzh"
Nov 02 14:56:41.343516 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.342275    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/65ce39fc-05c7-433b-81f4-c1b219f1bd71-host-run-ovn-kubernetes\") pod \"ovnkube-node-f8btv\" (UID: \"65ce39fc-05c7-433b-81f4-c1b219f1bd71\") " pod="openshift-ovn-kubernetes/ovnkube-node-f8btv"
Nov 02 14:56:41.343516 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.342290    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/702f7864-7db1-40d5-ba62-33cb9477b7da-crio-socket\") pod \"insights-runtime-extractor-k6fjk\" (UID: \"702f7864-7db1-40d5-ba62-33cb9477b7da\") " pod="openshift-insights/insights-runtime-extractor-k6fjk"
Nov 02 14:56:41.343516 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.342292    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/332f4a2d-a241-4c71-8ebb-a99618c0d3dc-kubelet-config\") pod \"global-pull-secret-syncer-vl6zb\" (UID: \"332f4a2d-a241-4c71-8ebb-a99618c0d3dc\") " pod="kube-system/global-pull-secret-syncer-vl6zb"
Nov 02 14:56:41.343516 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.342299    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/54dc6898-f041-4ed6-9fc9-1b5159d00d27-sys-fs\") pod \"aws-ebs-csi-driver-node-6cj82\" (UID: \"54dc6898-f041-4ed6-9fc9-1b5159d00d27\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6cj82"
Nov 02 14:56:41.343516 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.342318    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/93d7d50e-b320-4217-b183-2bfc921f6c9d-hostroot\") pod \"multus-7slzh\" (UID: \"93d7d50e-b320-4217-b183-2bfc921f6c9d\") " pod="openshift-multus/multus-7slzh"
Nov 02 14:56:41.343516 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.342327    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e1da99c3-8294-4e34-9d33-47e04a6182d3-etc-sysctl-d\") pod \"tuned-sq4l4\" (UID: \"e1da99c3-8294-4e34-9d33-47e04a6182d3\") " pod="openshift-cluster-node-tuning-operator/tuned-sq4l4"
Nov 02 14:56:41.343516 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.342342    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e1da99c3-8294-4e34-9d33-47e04a6182d3-etc-systemd\") pod \"tuned-sq4l4\" (UID: \"e1da99c3-8294-4e34-9d33-47e04a6182d3\") " pod="openshift-cluster-node-tuning-operator/tuned-sq4l4"
Nov 02 14:56:41.343516 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.342362    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/54dc6898-f041-4ed6-9fc9-1b5159d00d27-sys-fs\") pod \"aws-ebs-csi-driver-node-6cj82\" (UID: \"54dc6898-f041-4ed6-9fc9-1b5159d00d27\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6cj82"
Nov 02 14:56:41.343516 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.342364    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e1da99c3-8294-4e34-9d33-47e04a6182d3-var-lib-kubelet\") pod \"tuned-sq4l4\" (UID: \"e1da99c3-8294-4e34-9d33-47e04a6182d3\") " pod="openshift-cluster-node-tuning-operator/tuned-sq4l4"
Nov 02 14:56:41.343516 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.342403    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e1da99c3-8294-4e34-9d33-47e04a6182d3-var-lib-kubelet\") pod \"tuned-sq4l4\" (UID: \"e1da99c3-8294-4e34-9d33-47e04a6182d3\") " pod="openshift-cluster-node-tuning-operator/tuned-sq4l4"
Nov 02 14:56:41.343516 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.342410    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/65ce39fc-05c7-433b-81f4-c1b219f1bd71-log-socket\") pod \"ovnkube-node-f8btv\" (UID: \"65ce39fc-05c7-433b-81f4-c1b219f1bd71\") " pod="openshift-ovn-kubernetes/ovnkube-node-f8btv"
Nov 02 14:56:41.343516 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.342415    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e1da99c3-8294-4e34-9d33-47e04a6182d3-etc-systemd\") pod \"tuned-sq4l4\" (UID: \"e1da99c3-8294-4e34-9d33-47e04a6182d3\") " pod="openshift-cluster-node-tuning-operator/tuned-sq4l4"
Nov 02 14:56:41.343516 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.342434    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/65ce39fc-05c7-433b-81f4-c1b219f1bd71-host-cni-bin\") pod \"ovnkube-node-f8btv\" (UID: \"65ce39fc-05c7-433b-81f4-c1b219f1bd71\") " pod="openshift-ovn-kubernetes/ovnkube-node-f8btv"
Nov 02 14:56:41.343516 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.342462    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e1da99c3-8294-4e34-9d33-47e04a6182d3-etc-modprobe-d\") pod \"tuned-sq4l4\" (UID: \"e1da99c3-8294-4e34-9d33-47e04a6182d3\") " pod="openshift-cluster-node-tuning-operator/tuned-sq4l4"
Nov 02 14:56:41.343516 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.342477    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/65ce39fc-05c7-433b-81f4-c1b219f1bd71-log-socket\") pod \"ovnkube-node-f8btv\" (UID: \"65ce39fc-05c7-433b-81f4-c1b219f1bd71\") " pod="openshift-ovn-kubernetes/ovnkube-node-f8btv"
Nov 02 14:56:41.343516 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.342490    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/65ce39fc-05c7-433b-81f4-c1b219f1bd71-host-cni-bin\") pod \"ovnkube-node-f8btv\" (UID: \"65ce39fc-05c7-433b-81f4-c1b219f1bd71\") " pod="openshift-ovn-kubernetes/ovnkube-node-f8btv"
Nov 02 14:56:41.343516 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.342491    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/65ce39fc-05c7-433b-81f4-c1b219f1bd71-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-f8btv\" (UID: \"65ce39fc-05c7-433b-81f4-c1b219f1bd71\") " pod="openshift-ovn-kubernetes/ovnkube-node-f8btv"
Nov 02 14:56:41.343516 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.342522    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/720ef290-024b-4ed2-8417-92290f360dc9-hosts-file\") pod \"node-resolver-jpz27\" (UID: \"720ef290-024b-4ed2-8417-92290f360dc9\") " pod="openshift-dns/node-resolver-jpz27"
Nov 02 14:56:41.343516 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.342524    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/65ce39fc-05c7-433b-81f4-c1b219f1bd71-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-f8btv\" (UID: \"65ce39fc-05c7-433b-81f4-c1b219f1bd71\") " pod="openshift-ovn-kubernetes/ovnkube-node-f8btv"
Nov 02 14:56:41.343516 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.342491    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e1da99c3-8294-4e34-9d33-47e04a6182d3-etc-sysctl-d\") pod \"tuned-sq4l4\" (UID: \"e1da99c3-8294-4e34-9d33-47e04a6182d3\") " pod="openshift-cluster-node-tuning-operator/tuned-sq4l4"
Nov 02 14:56:41.343516 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.342558    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e1da99c3-8294-4e34-9d33-47e04a6182d3-etc-modprobe-d\") pod \"tuned-sq4l4\" (UID: \"e1da99c3-8294-4e34-9d33-47e04a6182d3\") " pod="openshift-cluster-node-tuning-operator/tuned-sq4l4"
Nov 02 14:56:41.343516 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.342574    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/720ef290-024b-4ed2-8417-92290f360dc9-hosts-file\") pod \"node-resolver-jpz27\" (UID: \"720ef290-024b-4ed2-8417-92290f360dc9\") " pod="openshift-dns/node-resolver-jpz27"
Nov 02 14:56:41.907823 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.907785    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-wns8r"
Nov 02 14:56:41.987521 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:41.987488    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-kfnkt"
Nov 02 14:56:46.853645 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:46.853618    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-f8btv"
Nov 02 14:56:47.763056 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:47.763028    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-f8btv"
Nov 02 14:56:47.763264 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:47.763242    5304 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 02 14:56:54.796012 ip-10-0-135-138 kubenswrapper[5304]: E1102 14:56:54.795928    5304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k6fjk" containerName="extractor"
Nov 02 14:56:56.504765 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:56:56.504737    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-f8btv"
Nov 02 14:57:02.213299 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:57:02.213269    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-d5t6w"
Nov 02 14:57:12.159948 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:57:12.159922    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-wns8r"
Nov 02 14:57:24.795487 ip-10-0-135-138 kubenswrapper[5304]: E1102 14:57:24.795444    5304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k6fjk" containerName="extractor"
Nov 02 14:57:54.796264 ip-10-0-135-138 kubenswrapper[5304]: E1102 14:57:54.796224    5304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k6fjk" containerName="extractor"
Nov 02 14:58:24.796566 ip-10-0-135-138 kubenswrapper[5304]: E1102 14:58:24.796478    5304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k6fjk" containerName="extractor"
Nov 02 14:58:54.795755 ip-10-0-135-138 kubenswrapper[5304]: E1102 14:58:54.795714    5304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k6fjk" containerName="extractor"
Nov 02 14:59:24.796246 ip-10-0-135-138 kubenswrapper[5304]: E1102 14:59:24.796208    5304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k6fjk" containerName="extractor"
Nov 02 14:59:44.312764 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:59:44.312726    5304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qhklf"]
Nov 02 14:59:44.314977 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:59:44.314951    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qhklf"
Nov 02 14:59:44.341018 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:59:44.341000    5304 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Nov 02 14:59:44.341122 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:59:44.341035    5304 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Nov 02 14:59:44.341122 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:59:44.341040    5304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e644421527c169d876fdb59235c927986d6bd2-9d2815fc-dockercfg-qjp47\""
Nov 02 14:59:44.357970 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:59:44.357949    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qhklf"]
Nov 02 14:59:44.492408 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:59:44.492364    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/de7c8172-694c-48a1-8120-24b799af286f-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qhklf\" (UID: \"de7c8172-694c-48a1-8120-24b799af286f\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qhklf"
Nov 02 14:59:44.492598 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:59:44.492504    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/de7c8172-694c-48a1-8120-24b799af286f-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qhklf\" (UID: \"de7c8172-694c-48a1-8120-24b799af286f\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qhklf"
Nov 02 14:59:44.492598 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:59:44.492540    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fhxrm\" (UniqueName: \"kubernetes.io/projected/de7c8172-694c-48a1-8120-24b799af286f-kube-api-access-fhxrm\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qhklf\" (UID: \"de7c8172-694c-48a1-8120-24b799af286f\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qhklf"
Nov 02 14:59:44.593556 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:59:44.593519    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/de7c8172-694c-48a1-8120-24b799af286f-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qhklf\" (UID: \"de7c8172-694c-48a1-8120-24b799af286f\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qhklf"
Nov 02 14:59:44.593699 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:59:44.593561    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fhxrm\" (UniqueName: \"kubernetes.io/projected/de7c8172-694c-48a1-8120-24b799af286f-kube-api-access-fhxrm\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qhklf\" (UID: \"de7c8172-694c-48a1-8120-24b799af286f\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qhklf"
Nov 02 14:59:44.593699 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:59:44.593583    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/de7c8172-694c-48a1-8120-24b799af286f-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qhklf\" (UID: \"de7c8172-694c-48a1-8120-24b799af286f\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qhklf"
Nov 02 14:59:44.593917 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:59:44.593903    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/de7c8172-694c-48a1-8120-24b799af286f-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qhklf\" (UID: \"de7c8172-694c-48a1-8120-24b799af286f\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qhklf"
Nov 02 14:59:44.594005 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:59:44.593984    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/de7c8172-694c-48a1-8120-24b799af286f-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qhklf\" (UID: \"de7c8172-694c-48a1-8120-24b799af286f\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qhklf"
Nov 02 14:59:44.614766 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:59:44.614740    5304 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Nov 02 14:59:44.616937 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:59:44.616917    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fhxrm\" (UniqueName: \"kubernetes.io/projected/de7c8172-694c-48a1-8120-24b799af286f-kube-api-access-fhxrm\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qhklf\" (UID: \"de7c8172-694c-48a1-8120-24b799af286f\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qhklf"
Nov 02 14:59:44.627762 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:59:44.627732    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qhklf"
Nov 02 14:59:44.847463 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:59:44.847388    5304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-r9b58"]
Nov 02 14:59:44.849873 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:59:44.849849    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-r9b58"
Nov 02 14:59:44.884074 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:59:44.884048    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-r9b58"]
Nov 02 14:59:44.892347 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:59:44.892324    5304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Nov 02 14:59:44.892347 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:59:44.892324    5304 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Nov 02 14:59:44.892544 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:59:44.892440    5304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Nov 02 14:59:44.892544 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:59:44.892492    5304 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Nov 02 14:59:44.892544 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:59:44.892507    5304 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Nov 02 14:59:44.892544 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:59:44.892493    5304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Nov 02 14:59:44.892712 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:59:44.892697    5304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-rcl8t\""
Nov 02 14:59:44.923878 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:59:44.923855    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qhklf"]
Nov 02 14:59:44.929940 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:59:44.929912    5304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde7c8172_694c_48a1_8120_24b799af286f.slice/crio-6f2e4bd4143058e6bfea783af5543750eefc5c706faf259f67b978bc740f51b7 WatchSource:0}: Error finding container 6f2e4bd4143058e6bfea783af5543750eefc5c706faf259f67b978bc740f51b7: Status 404 returned error can't find the container with id 6f2e4bd4143058e6bfea783af5543750eefc5c706faf259f67b978bc740f51b7
Nov 02 14:59:44.931934 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:59:44.931915    5304 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 02 14:59:44.995512 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:59:44.995466    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gsvsp\" (UniqueName: \"kubernetes.io/projected/9ded5d29-7101-4a51-a370-abdba1e29cf2-kube-api-access-gsvsp\") pod \"istiod-openshift-gateway-7876bfcb74-r9b58\" (UID: \"9ded5d29-7101-4a51-a370-abdba1e29cf2\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-r9b58"
Nov 02 14:59:44.995512 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:59:44.995516    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ded5d29-7101-4a51-a370-abdba1e29cf2-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-r9b58\" (UID: \"9ded5d29-7101-4a51-a370-abdba1e29cf2\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-r9b58"
Nov 02 14:59:44.995777 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:59:44.995534    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/9ded5d29-7101-4a51-a370-abdba1e29cf2-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-r9b58\" (UID: \"9ded5d29-7101-4a51-a370-abdba1e29cf2\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-r9b58"
Nov 02 14:59:44.995777 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:59:44.995622    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/9ded5d29-7101-4a51-a370-abdba1e29cf2-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-r9b58\" (UID: \"9ded5d29-7101-4a51-a370-abdba1e29cf2\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-r9b58"
Nov 02 14:59:44.995777 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:59:44.995750    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/9ded5d29-7101-4a51-a370-abdba1e29cf2-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-r9b58\" (UID: \"9ded5d29-7101-4a51-a370-abdba1e29cf2\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-r9b58"
Nov 02 14:59:44.995910 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:59:44.995792    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ded5d29-7101-4a51-a370-abdba1e29cf2-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-r9b58\" (UID: \"9ded5d29-7101-4a51-a370-abdba1e29cf2\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-r9b58"
Nov 02 14:59:44.995910 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:59:44.995817    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9ded5d29-7101-4a51-a370-abdba1e29cf2-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-r9b58\" (UID: \"9ded5d29-7101-4a51-a370-abdba1e29cf2\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-r9b58"
Nov 02 14:59:45.096304 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:59:45.096256    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/9ded5d29-7101-4a51-a370-abdba1e29cf2-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-r9b58\" (UID: \"9ded5d29-7101-4a51-a370-abdba1e29cf2\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-r9b58"
Nov 02 14:59:45.096304 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:59:45.096306    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/9ded5d29-7101-4a51-a370-abdba1e29cf2-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-r9b58\" (UID: \"9ded5d29-7101-4a51-a370-abdba1e29cf2\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-r9b58"
Nov 02 14:59:45.096565 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:59:45.096333    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ded5d29-7101-4a51-a370-abdba1e29cf2-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-r9b58\" (UID: \"9ded5d29-7101-4a51-a370-abdba1e29cf2\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-r9b58"
Nov 02 14:59:45.096565 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:59:45.096348    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9ded5d29-7101-4a51-a370-abdba1e29cf2-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-r9b58\" (UID: \"9ded5d29-7101-4a51-a370-abdba1e29cf2\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-r9b58"
Nov 02 14:59:45.096565 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:59:45.096389    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gsvsp\" (UniqueName: \"kubernetes.io/projected/9ded5d29-7101-4a51-a370-abdba1e29cf2-kube-api-access-gsvsp\") pod \"istiod-openshift-gateway-7876bfcb74-r9b58\" (UID: \"9ded5d29-7101-4a51-a370-abdba1e29cf2\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-r9b58"
Nov 02 14:59:45.096565 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:59:45.096411    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ded5d29-7101-4a51-a370-abdba1e29cf2-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-r9b58\" (UID: \"9ded5d29-7101-4a51-a370-abdba1e29cf2\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-r9b58"
Nov 02 14:59:45.096565 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:59:45.096425    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/9ded5d29-7101-4a51-a370-abdba1e29cf2-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-r9b58\" (UID: \"9ded5d29-7101-4a51-a370-abdba1e29cf2\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-r9b58"
Nov 02 14:59:45.096992 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:59:45.096960    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/9ded5d29-7101-4a51-a370-abdba1e29cf2-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-r9b58\" (UID: \"9ded5d29-7101-4a51-a370-abdba1e29cf2\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-r9b58"
Nov 02 14:59:45.098832 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:59:45.098764    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ded5d29-7101-4a51-a370-abdba1e29cf2-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-r9b58\" (UID: \"9ded5d29-7101-4a51-a370-abdba1e29cf2\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-r9b58"
Nov 02 14:59:45.098832 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:59:45.098791    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9ded5d29-7101-4a51-a370-abdba1e29cf2-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-r9b58\" (UID: \"9ded5d29-7101-4a51-a370-abdba1e29cf2\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-r9b58"
Nov 02 14:59:45.098953 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:59:45.098895    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/9ded5d29-7101-4a51-a370-abdba1e29cf2-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-r9b58\" (UID: \"9ded5d29-7101-4a51-a370-abdba1e29cf2\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-r9b58"
Nov 02 14:59:45.098953 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:59:45.098928    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/9ded5d29-7101-4a51-a370-abdba1e29cf2-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-r9b58\" (UID: \"9ded5d29-7101-4a51-a370-abdba1e29cf2\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-r9b58"
Nov 02 14:59:45.123626 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:59:45.123603    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ded5d29-7101-4a51-a370-abdba1e29cf2-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-r9b58\" (UID: \"9ded5d29-7101-4a51-a370-abdba1e29cf2\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-r9b58"
Nov 02 14:59:45.123838 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:59:45.123819    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gsvsp\" (UniqueName: \"kubernetes.io/projected/9ded5d29-7101-4a51-a370-abdba1e29cf2-kube-api-access-gsvsp\") pod \"istiod-openshift-gateway-7876bfcb74-r9b58\" (UID: \"9ded5d29-7101-4a51-a370-abdba1e29cf2\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-r9b58"
Nov 02 14:59:45.160433 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:59:45.160403    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-r9b58"
Nov 02 14:59:45.469033 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:59:45.468944    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-r9b58"]
Nov 02 14:59:45.475081 ip-10-0-135-138 kubenswrapper[5304]: W1102 14:59:45.475039    5304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ded5d29_7101_4a51_a370_abdba1e29cf2.slice/crio-e1d2e800d0c989d93ae5d62141d2aae77880bd63f914d2eba246e88b72045ccc WatchSource:0}: Error finding container e1d2e800d0c989d93ae5d62141d2aae77880bd63f914d2eba246e88b72045ccc: Status 404 returned error can't find the container with id e1d2e800d0c989d93ae5d62141d2aae77880bd63f914d2eba246e88b72045ccc
Nov 02 14:59:45.692638 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:59:45.692596    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-r9b58" event={"ID":"9ded5d29-7101-4a51-a370-abdba1e29cf2","Type":"ContainerStarted","Data":"e1d2e800d0c989d93ae5d62141d2aae77880bd63f914d2eba246e88b72045ccc"}
Nov 02 14:59:45.693975 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:59:45.693947    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qhklf" event={"ID":"de7c8172-694c-48a1-8120-24b799af286f","Type":"ContainerStarted","Data":"6f2e4bd4143058e6bfea783af5543750eefc5c706faf259f67b978bc740f51b7"}
Nov 02 14:59:48.316581 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:59:48.316538    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 14:59:48.316908 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:59:48.316626    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 14:59:48.707231 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:59:48.707195    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-r9b58" event={"ID":"9ded5d29-7101-4a51-a370-abdba1e29cf2","Type":"ContainerStarted","Data":"2f31a103549587f3eb98a3feb09b1cbe5a55ecf013004b563d0e2aec35e15f03"}
Nov 02 14:59:48.707419 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:59:48.707306    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-r9b58"
Nov 02 14:59:48.775690 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:59:48.775625    5304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-r9b58" podStartSLOduration=1.937490095 podStartE2EDuration="4.775607365s" totalImagesPullingTime="2.83811727s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:59:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:59:48.763618321 +0000 UTC m=+188.757442730" watchObservedRunningTime="2026-11-02 14:59:48.775607365 +0000 UTC m=+188.769431775"
Nov 02 14:59:49.712868 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:59:49.712821    5304 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7876bfcb74-r9b58 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 14:59:49.713377 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:59:49.712888    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-r9b58" podUID="9ded5d29-7101-4a51-a370-abdba1e29cf2" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 14:59:51.718233 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:59:51.718193    5304 generic.go:356] "Generic (PLEG): container finished" podID="de7c8172-694c-48a1-8120-24b799af286f" containerID="fb2fb237d5f330f0fdef3e35b141ecdc7851cc1ab5d4096d16c3ce334058e08f" exitCode=0
Nov 02 14:59:51.718641 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:59:51.718240    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qhklf" event={"ID":"de7c8172-694c-48a1-8120-24b799af286f","Type":"ContainerDied","Data":"fb2fb237d5f330f0fdef3e35b141ecdc7851cc1ab5d4096d16c3ce334058e08f"}
Nov 02 14:59:52.712458 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:59:52.712422    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-r9b58"
Nov 02 14:59:52.722986 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:59:52.722958    5304 generic.go:356] "Generic (PLEG): container finished" podID="de7c8172-694c-48a1-8120-24b799af286f" containerID="859b15bdce00c797b30f1d9abdec11a9f74347d4a937f76b696d727cc945e9ed" exitCode=0
Nov 02 14:59:52.723353 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:59:52.723003    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qhklf" event={"ID":"de7c8172-694c-48a1-8120-24b799af286f","Type":"ContainerDied","Data":"859b15bdce00c797b30f1d9abdec11a9f74347d4a937f76b696d727cc945e9ed"}
Nov 02 14:59:54.801040 ip-10-0-135-138 kubenswrapper[5304]: E1102 14:59:54.801003    5304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k6fjk" containerName="extractor"
Nov 02 14:59:58.746484 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:59:58.746453    5304 generic.go:356] "Generic (PLEG): container finished" podID="de7c8172-694c-48a1-8120-24b799af286f" containerID="15fe60c72b30b377f55f8cf2e2aa99416c9ce484ac206ea89a93b69528d90808" exitCode=0
Nov 02 14:59:58.746822 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:59:58.746502    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qhklf" event={"ID":"de7c8172-694c-48a1-8120-24b799af286f","Type":"ContainerDied","Data":"15fe60c72b30b377f55f8cf2e2aa99416c9ce484ac206ea89a93b69528d90808"}
Nov 02 14:59:59.867470 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:59:59.867445    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qhklf"
Nov 02 14:59:59.914661 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:59:59.914627    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de7c8172-694c-48a1-8120-24b799af286f-util\" (UniqueName: \"kubernetes.io/empty-dir/de7c8172-694c-48a1-8120-24b799af286f-util\") pod \"de7c8172-694c-48a1-8120-24b799af286f\" (UID: \"de7c8172-694c-48a1-8120-24b799af286f\") "
Nov 02 14:59:59.914806 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:59:59.914721    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de7c8172-694c-48a1-8120-24b799af286f-kube-api-access-fhxrm\" (UniqueName: \"kubernetes.io/projected/de7c8172-694c-48a1-8120-24b799af286f-kube-api-access-fhxrm\") pod \"de7c8172-694c-48a1-8120-24b799af286f\" (UID: \"de7c8172-694c-48a1-8120-24b799af286f\") "
Nov 02 14:59:59.917151 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:59:59.917100    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de7c8172-694c-48a1-8120-24b799af286f-kube-api-access-fhxrm" pod "de7c8172-694c-48a1-8120-24b799af286f" (UID: "de7c8172-694c-48a1-8120-24b799af286f"). InnerVolumeSpecName "kube-api-access-fhxrm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:59:59.920614 ip-10-0-135-138 kubenswrapper[5304]: I1102 14:59:59.920582    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de7c8172-694c-48a1-8120-24b799af286f-util" pod "de7c8172-694c-48a1-8120-24b799af286f" (UID: "de7c8172-694c-48a1-8120-24b799af286f"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:00:00.015645 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:00:00.015554    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de7c8172-694c-48a1-8120-24b799af286f-bundle\" (UniqueName: \"kubernetes.io/empty-dir/de7c8172-694c-48a1-8120-24b799af286f-bundle\") pod \"de7c8172-694c-48a1-8120-24b799af286f\" (UID: \"de7c8172-694c-48a1-8120-24b799af286f\") "
Nov 02 15:00:00.015772 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:00:00.015692    5304 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/de7c8172-694c-48a1-8120-24b799af286f-util\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:00:00.015772 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:00:00.015704    5304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fhxrm\" (UniqueName: \"kubernetes.io/projected/de7c8172-694c-48a1-8120-24b799af286f-kube-api-access-fhxrm\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:00:00.016066 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:00:00.016042    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de7c8172-694c-48a1-8120-24b799af286f-bundle" pod "de7c8172-694c-48a1-8120-24b799af286f" (UID: "de7c8172-694c-48a1-8120-24b799af286f"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:00:00.116011 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:00:00.115979    5304 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/de7c8172-694c-48a1-8120-24b799af286f-bundle\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:00:00.754907 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:00:00.754875    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qhklf" event={"ID":"de7c8172-694c-48a1-8120-24b799af286f","Type":"ContainerDied","Data":"6f2e4bd4143058e6bfea783af5543750eefc5c706faf259f67b978bc740f51b7"}
Nov 02 15:00:00.755070 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:00:00.754923    5304 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6f2e4bd4143058e6bfea783af5543750eefc5c706faf259f67b978bc740f51b7"
Nov 02 15:00:00.755070 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:00:00.754926    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qhklf"
Nov 02 15:00:24.796045 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:00:24.796007    5304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k6fjk" containerName="extractor"
Nov 02 15:00:33.880220 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:00:33.880187    5304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-f6qf4"]
Nov 02 15:00:33.883381 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:00:33.880411    5304 cpu_manager.go:414] "containerMap: removing container" podUID="de7c8172-694c-48a1-8120-24b799af286f" containerName="util"
Nov 02 15:00:33.883381 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:00:33.880422    5304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de7c8172-694c-48a1-8120-24b799af286f" containerName="util"
Nov 02 15:00:33.883381 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:00:33.880437    5304 cpu_manager.go:414] "containerMap: removing container" podUID="de7c8172-694c-48a1-8120-24b799af286f" containerName="pull"
Nov 02 15:00:33.883381 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:00:33.880443    5304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de7c8172-694c-48a1-8120-24b799af286f" containerName="pull"
Nov 02 15:00:33.883381 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:00:33.880454    5304 cpu_manager.go:414] "containerMap: removing container" podUID="de7c8172-694c-48a1-8120-24b799af286f" containerName="extract"
Nov 02 15:00:33.883381 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:00:33.880460    5304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de7c8172-694c-48a1-8120-24b799af286f" containerName="extract"
Nov 02 15:00:33.883381 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:00:33.880800    5304 memory_manager.go:365] "RemoveStaleState removing state" podUID="de7c8172-694c-48a1-8120-24b799af286f" containerName="extract"
Nov 02 15:00:33.884182 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:00:33.884153    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-f6qf4"
Nov 02 15:00:33.901313 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:00:33.901290    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-f6qf4"]
Nov 02 15:00:33.912505 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:00:33.912483    5304 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Nov 02 15:00:33.912631 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:00:33.912515    5304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-bn4xg\""
Nov 02 15:00:33.912631 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:00:33.912609    5304 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Nov 02 15:00:33.938170 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:00:33.938143    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/27a9941a-7c9e-4844-b617-91f9d9890d78-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-f6qf4\" (UID: \"27a9941a-7c9e-4844-b617-91f9d9890d78\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-f6qf4"
Nov 02 15:00:33.938283 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:00:33.938187    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-894xc\" (UniqueName: \"kubernetes.io/projected/27a9941a-7c9e-4844-b617-91f9d9890d78-kube-api-access-894xc\") pod \"cert-manager-cainjector-5c57bbd4fc-f6qf4\" (UID: \"27a9941a-7c9e-4844-b617-91f9d9890d78\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-f6qf4"
Nov 02 15:00:34.039196 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:00:34.039154    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-894xc\" (UniqueName: \"kubernetes.io/projected/27a9941a-7c9e-4844-b617-91f9d9890d78-kube-api-access-894xc\") pod \"cert-manager-cainjector-5c57bbd4fc-f6qf4\" (UID: \"27a9941a-7c9e-4844-b617-91f9d9890d78\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-f6qf4"
Nov 02 15:00:34.039367 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:00:34.039219    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/27a9941a-7c9e-4844-b617-91f9d9890d78-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-f6qf4\" (UID: \"27a9941a-7c9e-4844-b617-91f9d9890d78\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-f6qf4"
Nov 02 15:00:34.063173 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:00:34.063146    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/27a9941a-7c9e-4844-b617-91f9d9890d78-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-f6qf4\" (UID: \"27a9941a-7c9e-4844-b617-91f9d9890d78\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-f6qf4"
Nov 02 15:00:34.063696 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:00:34.063675    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-894xc\" (UniqueName: \"kubernetes.io/projected/27a9941a-7c9e-4844-b617-91f9d9890d78-kube-api-access-894xc\") pod \"cert-manager-cainjector-5c57bbd4fc-f6qf4\" (UID: \"27a9941a-7c9e-4844-b617-91f9d9890d78\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-f6qf4"
Nov 02 15:00:34.210420 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:00:34.210331    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-f6qf4"
Nov 02 15:00:34.511337 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:00:34.511268    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-f6qf4"]
Nov 02 15:00:34.516450 ip-10-0-135-138 kubenswrapper[5304]: W1102 15:00:34.516421    5304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod27a9941a_7c9e_4844_b617_91f9d9890d78.slice/crio-91bcc28e86ecc2c279785288bdbe7447e5eedf9d6528cb703bf8308a7227c997 WatchSource:0}: Error finding container 91bcc28e86ecc2c279785288bdbe7447e5eedf9d6528cb703bf8308a7227c997: Status 404 returned error can't find the container with id 91bcc28e86ecc2c279785288bdbe7447e5eedf9d6528cb703bf8308a7227c997
Nov 02 15:00:34.863440 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:00:34.863402    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-f6qf4" event={"ID":"27a9941a-7c9e-4844-b617-91f9d9890d78","Type":"ContainerStarted","Data":"91bcc28e86ecc2c279785288bdbe7447e5eedf9d6528cb703bf8308a7227c997"}
Nov 02 15:00:38.879839 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:00:38.879804    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-f6qf4" event={"ID":"27a9941a-7c9e-4844-b617-91f9d9890d78","Type":"ContainerStarted","Data":"46f9c65d9e0cffda2f3b8519e2272c7114620a60870078c00cf0166c96ad1f70"}
Nov 02 15:00:39.893713 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:00:39.893663    5304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-f6qf4" podStartSLOduration=3.503170398 podStartE2EDuration="6.893648895s" totalImagesPullingTime="3.390478497s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:00:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:00:38.930911958 +0000 UTC m=+238.924736367" watchObservedRunningTime="2026-11-02 15:00:39.893648895 +0000 UTC m=+239.887473307"
Nov 02 15:00:39.894110 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:00:39.894033    5304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-66c8cfc56f-kvp4b"]
Nov 02 15:00:39.897192 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:00:39.897158    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-66c8cfc56f-kvp4b"
Nov 02 15:00:39.940999 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:00:39.940969    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-66c8cfc56f-kvp4b"]
Nov 02 15:00:39.982407 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:00:39.982366    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/2e65dce3-87ad-498d-b369-5015a1833753-manifests\") pod \"opendatahub-operator-controller-manager-66c8cfc56f-kvp4b\" (UID: \"2e65dce3-87ad-498d-b369-5015a1833753\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-66c8cfc56f-kvp4b"
Nov 02 15:00:39.982407 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:00:39.982406    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/2e65dce3-87ad-498d-b369-5015a1833753-metrics-cert\") pod \"opendatahub-operator-controller-manager-66c8cfc56f-kvp4b\" (UID: \"2e65dce3-87ad-498d-b369-5015a1833753\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-66c8cfc56f-kvp4b"
Nov 02 15:00:39.982654 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:00:39.982424    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6nxxf\" (UniqueName: \"kubernetes.io/projected/2e65dce3-87ad-498d-b369-5015a1833753-kube-api-access-6nxxf\") pod \"opendatahub-operator-controller-manager-66c8cfc56f-kvp4b\" (UID: \"2e65dce3-87ad-498d-b369-5015a1833753\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-66c8cfc56f-kvp4b"
Nov 02 15:00:39.982654 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:00:39.982528    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2e65dce3-87ad-498d-b369-5015a1833753-webhook-cert\") pod \"opendatahub-operator-controller-manager-66c8cfc56f-kvp4b\" (UID: \"2e65dce3-87ad-498d-b369-5015a1833753\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-66c8cfc56f-kvp4b"
Nov 02 15:00:39.982654 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:00:39.982564    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2e65dce3-87ad-498d-b369-5015a1833753-tmp\") pod \"opendatahub-operator-controller-manager-66c8cfc56f-kvp4b\" (UID: \"2e65dce3-87ad-498d-b369-5015a1833753\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-66c8cfc56f-kvp4b"
Nov 02 15:00:39.982654 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:00:39.982589    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/2e65dce3-87ad-498d-b369-5015a1833753-apiservice-cert\") pod \"opendatahub-operator-controller-manager-66c8cfc56f-kvp4b\" (UID: \"2e65dce3-87ad-498d-b369-5015a1833753\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-66c8cfc56f-kvp4b"
Nov 02 15:00:40.003910 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:00:40.003881    5304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\""
Nov 02 15:00:40.004161 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:00:40.004115    5304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\""
Nov 02 15:00:40.004414 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:00:40.004399    5304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-46gbv\""
Nov 02 15:00:40.004629 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:00:40.004617    5304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\""
Nov 02 15:00:40.006716 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:00:40.006702    5304 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\""
Nov 02 15:00:40.006764 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:00:40.006721    5304 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\""
Nov 02 15:00:40.083077 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:00:40.083043    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/2e65dce3-87ad-498d-b369-5015a1833753-manifests\") pod \"opendatahub-operator-controller-manager-66c8cfc56f-kvp4b\" (UID: \"2e65dce3-87ad-498d-b369-5015a1833753\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-66c8cfc56f-kvp4b"
Nov 02 15:00:40.083077 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:00:40.083076    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/2e65dce3-87ad-498d-b369-5015a1833753-metrics-cert\") pod \"opendatahub-operator-controller-manager-66c8cfc56f-kvp4b\" (UID: \"2e65dce3-87ad-498d-b369-5015a1833753\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-66c8cfc56f-kvp4b"
Nov 02 15:00:40.083321 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:00:40.083094    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6nxxf\" (UniqueName: \"kubernetes.io/projected/2e65dce3-87ad-498d-b369-5015a1833753-kube-api-access-6nxxf\") pod \"opendatahub-operator-controller-manager-66c8cfc56f-kvp4b\" (UID: \"2e65dce3-87ad-498d-b369-5015a1833753\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-66c8cfc56f-kvp4b"
Nov 02 15:00:40.083321 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:00:40.083119    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2e65dce3-87ad-498d-b369-5015a1833753-webhook-cert\") pod \"opendatahub-operator-controller-manager-66c8cfc56f-kvp4b\" (UID: \"2e65dce3-87ad-498d-b369-5015a1833753\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-66c8cfc56f-kvp4b"
Nov 02 15:00:40.083321 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:00:40.083170    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2e65dce3-87ad-498d-b369-5015a1833753-tmp\") pod \"opendatahub-operator-controller-manager-66c8cfc56f-kvp4b\" (UID: \"2e65dce3-87ad-498d-b369-5015a1833753\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-66c8cfc56f-kvp4b"
Nov 02 15:00:40.083321 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:00:40.083207    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/2e65dce3-87ad-498d-b369-5015a1833753-apiservice-cert\") pod \"opendatahub-operator-controller-manager-66c8cfc56f-kvp4b\" (UID: \"2e65dce3-87ad-498d-b369-5015a1833753\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-66c8cfc56f-kvp4b"
Nov 02 15:00:40.083478 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:00:40.083455    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/2e65dce3-87ad-498d-b369-5015a1833753-manifests\") pod \"opendatahub-operator-controller-manager-66c8cfc56f-kvp4b\" (UID: \"2e65dce3-87ad-498d-b369-5015a1833753\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-66c8cfc56f-kvp4b"
Nov 02 15:00:40.085324 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:00:40.085303    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2e65dce3-87ad-498d-b369-5015a1833753-tmp\") pod \"opendatahub-operator-controller-manager-66c8cfc56f-kvp4b\" (UID: \"2e65dce3-87ad-498d-b369-5015a1833753\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-66c8cfc56f-kvp4b"
Nov 02 15:00:40.097460 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:00:40.097435    5304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\""
Nov 02 15:00:40.097634 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:00:40.097617    5304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\""
Nov 02 15:00:40.105993 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:00:40.105971    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2e65dce3-87ad-498d-b369-5015a1833753-webhook-cert\") pod \"opendatahub-operator-controller-manager-66c8cfc56f-kvp4b\" (UID: \"2e65dce3-87ad-498d-b369-5015a1833753\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-66c8cfc56f-kvp4b"
Nov 02 15:00:40.106154 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:00:40.106083    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/2e65dce3-87ad-498d-b369-5015a1833753-metrics-cert\") pod \"opendatahub-operator-controller-manager-66c8cfc56f-kvp4b\" (UID: \"2e65dce3-87ad-498d-b369-5015a1833753\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-66c8cfc56f-kvp4b"
Nov 02 15:00:40.106221 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:00:40.106194    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/2e65dce3-87ad-498d-b369-5015a1833753-apiservice-cert\") pod \"opendatahub-operator-controller-manager-66c8cfc56f-kvp4b\" (UID: \"2e65dce3-87ad-498d-b369-5015a1833753\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-66c8cfc56f-kvp4b"
Nov 02 15:00:40.136598 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:00:40.136570    5304 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\""
Nov 02 15:00:40.156594 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:00:40.156542    5304 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\""
Nov 02 15:00:40.166101 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:00:40.166072    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6nxxf\" (UniqueName: \"kubernetes.io/projected/2e65dce3-87ad-498d-b369-5015a1833753-kube-api-access-6nxxf\") pod \"opendatahub-operator-controller-manager-66c8cfc56f-kvp4b\" (UID: \"2e65dce3-87ad-498d-b369-5015a1833753\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-66c8cfc56f-kvp4b"
Nov 02 15:00:40.224562 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:00:40.224532    5304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-46gbv\""
Nov 02 15:00:40.231792 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:00:40.231764    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-66c8cfc56f-kvp4b"
Nov 02 15:00:40.604799 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:00:40.604769    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-66c8cfc56f-kvp4b"]
Nov 02 15:00:40.610303 ip-10-0-135-138 kubenswrapper[5304]: W1102 15:00:40.610277    5304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e65dce3_87ad_498d_b369_5015a1833753.slice/crio-7a0492c74ae2e51afacb09b0ec27810e4dd966d6545c697b79a904688a381404 WatchSource:0}: Error finding container 7a0492c74ae2e51afacb09b0ec27810e4dd966d6545c697b79a904688a381404: Status 404 returned error can't find the container with id 7a0492c74ae2e51afacb09b0ec27810e4dd966d6545c697b79a904688a381404
Nov 02 15:00:40.888098 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:00:40.888009    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-66c8cfc56f-kvp4b" event={"ID":"2e65dce3-87ad-498d-b369-5015a1833753","Type":"ContainerStarted","Data":"7a0492c74ae2e51afacb09b0ec27810e4dd966d6545c697b79a904688a381404"}
Nov 02 15:00:41.722176 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:00:41.722140    5304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-n9gf2"]
Nov 02 15:00:41.724528 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:00:41.724469    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-n9gf2"
Nov 02 15:00:41.746709 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:00:41.746681    5304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-g2v6q\""
Nov 02 15:00:41.752447 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:00:41.752424    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-n9gf2"]
Nov 02 15:00:41.758481 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:00:41.758457    5304 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Nov 02 15:00:41.758625 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:00:41.758537    5304 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Nov 02 15:00:41.758625 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:00:41.758589    5304 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Nov 02 15:00:41.795844 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:00:41.795810    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8q5pf\" (UniqueName: \"kubernetes.io/projected/6f7db1ab-d7ad-4aa8-b04e-147b63a46ad6-kube-api-access-8q5pf\") pod \"kuadrant-console-plugin-66c4fb575d-n9gf2\" (UID: \"6f7db1ab-d7ad-4aa8-b04e-147b63a46ad6\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-n9gf2"
Nov 02 15:00:41.796058 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:00:41.795860    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/6f7db1ab-d7ad-4aa8-b04e-147b63a46ad6-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-n9gf2\" (UID: \"6f7db1ab-d7ad-4aa8-b04e-147b63a46ad6\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-n9gf2"
Nov 02 15:00:41.796058 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:00:41.795972    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/6f7db1ab-d7ad-4aa8-b04e-147b63a46ad6-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-n9gf2\" (UID: \"6f7db1ab-d7ad-4aa8-b04e-147b63a46ad6\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-n9gf2"
Nov 02 15:00:41.846491 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:00:41.846460    5304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Nov 02 15:00:41.896429 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:00:41.896391    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8q5pf\" (UniqueName: \"kubernetes.io/projected/6f7db1ab-d7ad-4aa8-b04e-147b63a46ad6-kube-api-access-8q5pf\") pod \"kuadrant-console-plugin-66c4fb575d-n9gf2\" (UID: \"6f7db1ab-d7ad-4aa8-b04e-147b63a46ad6\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-n9gf2"
Nov 02 15:00:41.896429 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:00:41.896438    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/6f7db1ab-d7ad-4aa8-b04e-147b63a46ad6-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-n9gf2\" (UID: \"6f7db1ab-d7ad-4aa8-b04e-147b63a46ad6\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-n9gf2"
Nov 02 15:00:41.896669 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:00:41.896466    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/6f7db1ab-d7ad-4aa8-b04e-147b63a46ad6-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-n9gf2\" (UID: \"6f7db1ab-d7ad-4aa8-b04e-147b63a46ad6\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-n9gf2"
Nov 02 15:00:41.897236 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:00:41.897198    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/6f7db1ab-d7ad-4aa8-b04e-147b63a46ad6-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-n9gf2\" (UID: \"6f7db1ab-d7ad-4aa8-b04e-147b63a46ad6\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-n9gf2"
Nov 02 15:00:41.900243 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:00:41.900215    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/6f7db1ab-d7ad-4aa8-b04e-147b63a46ad6-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-n9gf2\" (UID: \"6f7db1ab-d7ad-4aa8-b04e-147b63a46ad6\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-n9gf2"
Nov 02 15:00:41.924857 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:00:41.924830    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8q5pf\" (UniqueName: \"kubernetes.io/projected/6f7db1ab-d7ad-4aa8-b04e-147b63a46ad6-kube-api-access-8q5pf\") pod \"kuadrant-console-plugin-66c4fb575d-n9gf2\" (UID: \"6f7db1ab-d7ad-4aa8-b04e-147b63a46ad6\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-n9gf2"
Nov 02 15:00:42.037700 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:00:42.037613    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-n9gf2"
Nov 02 15:00:42.346311 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:00:42.346275    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-n9gf2"]
Nov 02 15:00:42.351925 ip-10-0-135-138 kubenswrapper[5304]: W1102 15:00:42.351892    5304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6f7db1ab_d7ad_4aa8_b04e_147b63a46ad6.slice/crio-0e175a5a704e351ba5d45005ffa05c56d99a63b6f9d8e84cd4a4255690a19472 WatchSource:0}: Error finding container 0e175a5a704e351ba5d45005ffa05c56d99a63b6f9d8e84cd4a4255690a19472: Status 404 returned error can't find the container with id 0e175a5a704e351ba5d45005ffa05c56d99a63b6f9d8e84cd4a4255690a19472
Nov 02 15:00:42.896481 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:00:42.896435    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-n9gf2" event={"ID":"6f7db1ab-d7ad-4aa8-b04e-147b63a46ad6","Type":"ContainerStarted","Data":"0e175a5a704e351ba5d45005ffa05c56d99a63b6f9d8e84cd4a4255690a19472"}
Nov 02 15:00:43.901831 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:00:43.901789    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-66c8cfc56f-kvp4b" event={"ID":"2e65dce3-87ad-498d-b369-5015a1833753","Type":"ContainerStarted","Data":"e7ce5f1e3927b4e4d01c644912a19dbb06e6b437237d31c8c81181640f3aaa49"}
Nov 02 15:00:44.907969 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:00:44.907921    5304 generic.go:356] "Generic (PLEG): container finished" podID="2e65dce3-87ad-498d-b369-5015a1833753" containerID="e7ce5f1e3927b4e4d01c644912a19dbb06e6b437237d31c8c81181640f3aaa49" exitCode=0
Nov 02 15:00:44.907969 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:00:44.907967    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-66c8cfc56f-kvp4b" event={"ID":"2e65dce3-87ad-498d-b369-5015a1833753","Type":"ContainerDied","Data":"e7ce5f1e3927b4e4d01c644912a19dbb06e6b437237d31c8c81181640f3aaa49"}
Nov 02 15:00:45.914617 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:00:45.914576    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-66c8cfc56f-kvp4b" event={"ID":"2e65dce3-87ad-498d-b369-5015a1833753","Type":"ContainerStarted","Data":"178d40f2739fb030559c608c5b215eee6c9019be22e10f9738ea8686cf7fc3da"}
Nov 02 15:00:45.915040 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:00:45.914721    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-66c8cfc56f-kvp4b"
Nov 02 15:00:46.006735 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:00:46.006677    5304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-66c8cfc56f-kvp4b" podStartSLOduration=2.862694282 podStartE2EDuration="7.006662114s" totalImagesPullingTime="3.552331145s" totalInitContainerRuntime="591.636687ms" isStatefulPod=false podCreationTimestamp="2026-11-02 15:00:39 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:00:45.994683385 +0000 UTC m=+245.988507794" watchObservedRunningTime="2026-11-02 15:00:46.006662114 +0000 UTC m=+246.000486522"
Nov 02 15:00:47.923715 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:00:47.923678    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-n9gf2" event={"ID":"6f7db1ab-d7ad-4aa8-b04e-147b63a46ad6","Type":"ContainerStarted","Data":"251add0aa4de536bcc6fa1ac6ee084c4b2656bc15153c5fb7169ecf5aa87b8c9"}
Nov 02 15:00:47.989719 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:00:47.989666    5304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-n9gf2" podStartSLOduration=2.443851306 podStartE2EDuration="6.989652059s" totalImagesPullingTime="4.545800753s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:00:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:00:47.988616996 +0000 UTC m=+247.982441405" watchObservedRunningTime="2026-11-02 15:00:47.989652059 +0000 UTC m=+247.983476462"
Nov 02 15:00:54.796803 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:00:54.796758    5304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k6fjk" containerName="extractor"
Nov 02 15:00:56.921644 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:00:56.921613    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-66c8cfc56f-kvp4b"
Nov 02 15:01:19.198443 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:01:19.198410    5304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-77f469495f-6cphj"]
Nov 02 15:01:19.218026 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:01:19.218000    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-77f469495f-6cphj"
Nov 02 15:01:19.262441 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:01:19.262401    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-77f469495f-6cphj"]
Nov 02 15:01:19.274242 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:01:19.274214    5304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Nov 02 15:01:19.274242 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:01:19.274234    5304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Nov 02 15:01:19.274468 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:01:19.274324    5304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-m9qjk\""
Nov 02 15:01:19.366589 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:01:19.366556    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a427d6de-97f8-4bbd-99d8-6327efcf1794-tls-certs\") pod \"kube-auth-proxy-77f469495f-6cphj\" (UID: \"a427d6de-97f8-4bbd-99d8-6327efcf1794\") " pod="openshift-ingress/kube-auth-proxy-77f469495f-6cphj"
Nov 02 15:01:19.366589 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:01:19.366594    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a427d6de-97f8-4bbd-99d8-6327efcf1794-tmp\") pod \"kube-auth-proxy-77f469495f-6cphj\" (UID: \"a427d6de-97f8-4bbd-99d8-6327efcf1794\") " pod="openshift-ingress/kube-auth-proxy-77f469495f-6cphj"
Nov 02 15:01:19.366833 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:01:19.366616    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-25jtv\" (UniqueName: \"kubernetes.io/projected/a427d6de-97f8-4bbd-99d8-6327efcf1794-kube-api-access-25jtv\") pod \"kube-auth-proxy-77f469495f-6cphj\" (UID: \"a427d6de-97f8-4bbd-99d8-6327efcf1794\") " pod="openshift-ingress/kube-auth-proxy-77f469495f-6cphj"
Nov 02 15:01:19.467969 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:01:19.467897    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a427d6de-97f8-4bbd-99d8-6327efcf1794-tls-certs\") pod \"kube-auth-proxy-77f469495f-6cphj\" (UID: \"a427d6de-97f8-4bbd-99d8-6327efcf1794\") " pod="openshift-ingress/kube-auth-proxy-77f469495f-6cphj"
Nov 02 15:01:19.467969 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:01:19.467934    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a427d6de-97f8-4bbd-99d8-6327efcf1794-tmp\") pod \"kube-auth-proxy-77f469495f-6cphj\" (UID: \"a427d6de-97f8-4bbd-99d8-6327efcf1794\") " pod="openshift-ingress/kube-auth-proxy-77f469495f-6cphj"
Nov 02 15:01:19.467969 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:01:19.467956    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-25jtv\" (UniqueName: \"kubernetes.io/projected/a427d6de-97f8-4bbd-99d8-6327efcf1794-kube-api-access-25jtv\") pod \"kube-auth-proxy-77f469495f-6cphj\" (UID: \"a427d6de-97f8-4bbd-99d8-6327efcf1794\") " pod="openshift-ingress/kube-auth-proxy-77f469495f-6cphj"
Nov 02 15:01:19.470331 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:01:19.470300    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a427d6de-97f8-4bbd-99d8-6327efcf1794-tmp\") pod \"kube-auth-proxy-77f469495f-6cphj\" (UID: \"a427d6de-97f8-4bbd-99d8-6327efcf1794\") " pod="openshift-ingress/kube-auth-proxy-77f469495f-6cphj"
Nov 02 15:01:19.470493 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:01:19.470478    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a427d6de-97f8-4bbd-99d8-6327efcf1794-tls-certs\") pod \"kube-auth-proxy-77f469495f-6cphj\" (UID: \"a427d6de-97f8-4bbd-99d8-6327efcf1794\") " pod="openshift-ingress/kube-auth-proxy-77f469495f-6cphj"
Nov 02 15:01:19.493083 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:01:19.493055    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-25jtv\" (UniqueName: \"kubernetes.io/projected/a427d6de-97f8-4bbd-99d8-6327efcf1794-kube-api-access-25jtv\") pod \"kube-auth-proxy-77f469495f-6cphj\" (UID: \"a427d6de-97f8-4bbd-99d8-6327efcf1794\") " pod="openshift-ingress/kube-auth-proxy-77f469495f-6cphj"
Nov 02 15:01:19.528687 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:01:19.528655    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-77f469495f-6cphj"
Nov 02 15:01:19.827372 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:01:19.827287    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-77f469495f-6cphj"]
Nov 02 15:01:19.832297 ip-10-0-135-138 kubenswrapper[5304]: W1102 15:01:19.832259    5304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda427d6de_97f8_4bbd_99d8_6327efcf1794.slice/crio-6ae69c9b3fdf4aa2fa6518ab9ae002fe9d60e4382a101dd5c134b4655ebd3dc3 WatchSource:0}: Error finding container 6ae69c9b3fdf4aa2fa6518ab9ae002fe9d60e4382a101dd5c134b4655ebd3dc3: Status 404 returned error can't find the container with id 6ae69c9b3fdf4aa2fa6518ab9ae002fe9d60e4382a101dd5c134b4655ebd3dc3
Nov 02 15:01:20.034198 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:01:20.034161    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-77f469495f-6cphj" event={"ID":"a427d6de-97f8-4bbd-99d8-6327efcf1794","Type":"ContainerStarted","Data":"6ae69c9b3fdf4aa2fa6518ab9ae002fe9d60e4382a101dd5c134b4655ebd3dc3"}
Nov 02 15:01:21.775906 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:01:21.775822    5304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-86cbcdbc8-6s5c5"]
Nov 02 15:01:21.778079 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:01:21.778058    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-86cbcdbc8-6s5c5"
Nov 02 15:01:21.794041 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:01:21.793990    5304 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"openshift-service-ca.crt\""
Nov 02 15:01:21.794201 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:01:21.794181    5304 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"jobset-manager-config\""
Nov 02 15:01:21.806087 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:01:21.806062    5304 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"kube-root-ca.crt\""
Nov 02 15:01:21.806236 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:01:21.806103    5304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"webhook-server-cert\""
Nov 02 15:01:21.806236 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:01:21.806114    5304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"metrics-server-cert\""
Nov 02 15:01:21.806346 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:01:21.806296    5304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-controller-manager-dockercfg-9krn5\""
Nov 02 15:01:21.818389 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:01:21.818364    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-86cbcdbc8-6s5c5"]
Nov 02 15:01:21.888772 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:01:21.888728    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0646bc9a-e23d-48e0-96ca-16468f810e7b-metrics-certs\") pod \"jobset-controller-manager-86cbcdbc8-6s5c5\" (UID: \"0646bc9a-e23d-48e0-96ca-16468f810e7b\") " pod="openshift-jobset-operator/jobset-controller-manager-86cbcdbc8-6s5c5"
Nov 02 15:01:21.888964 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:01:21.888853    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/0646bc9a-e23d-48e0-96ca-16468f810e7b-manager-config\") pod \"jobset-controller-manager-86cbcdbc8-6s5c5\" (UID: \"0646bc9a-e23d-48e0-96ca-16468f810e7b\") " pod="openshift-jobset-operator/jobset-controller-manager-86cbcdbc8-6s5c5"
Nov 02 15:01:21.888964 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:01:21.888890    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0646bc9a-e23d-48e0-96ca-16468f810e7b-cert\") pod \"jobset-controller-manager-86cbcdbc8-6s5c5\" (UID: \"0646bc9a-e23d-48e0-96ca-16468f810e7b\") " pod="openshift-jobset-operator/jobset-controller-manager-86cbcdbc8-6s5c5"
Nov 02 15:01:21.889167 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:01:21.889014    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kltw2\" (UniqueName: \"kubernetes.io/projected/0646bc9a-e23d-48e0-96ca-16468f810e7b-kube-api-access-kltw2\") pod \"jobset-controller-manager-86cbcdbc8-6s5c5\" (UID: \"0646bc9a-e23d-48e0-96ca-16468f810e7b\") " pod="openshift-jobset-operator/jobset-controller-manager-86cbcdbc8-6s5c5"
Nov 02 15:01:21.989993 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:01:21.989952    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0646bc9a-e23d-48e0-96ca-16468f810e7b-metrics-certs\") pod \"jobset-controller-manager-86cbcdbc8-6s5c5\" (UID: \"0646bc9a-e23d-48e0-96ca-16468f810e7b\") " pod="openshift-jobset-operator/jobset-controller-manager-86cbcdbc8-6s5c5"
Nov 02 15:01:21.990203 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:01:21.990002    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/0646bc9a-e23d-48e0-96ca-16468f810e7b-manager-config\") pod \"jobset-controller-manager-86cbcdbc8-6s5c5\" (UID: \"0646bc9a-e23d-48e0-96ca-16468f810e7b\") " pod="openshift-jobset-operator/jobset-controller-manager-86cbcdbc8-6s5c5"
Nov 02 15:01:21.990203 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:01:21.990030    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0646bc9a-e23d-48e0-96ca-16468f810e7b-cert\") pod \"jobset-controller-manager-86cbcdbc8-6s5c5\" (UID: \"0646bc9a-e23d-48e0-96ca-16468f810e7b\") " pod="openshift-jobset-operator/jobset-controller-manager-86cbcdbc8-6s5c5"
Nov 02 15:01:21.990203 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:01:21.990068    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kltw2\" (UniqueName: \"kubernetes.io/projected/0646bc9a-e23d-48e0-96ca-16468f810e7b-kube-api-access-kltw2\") pod \"jobset-controller-manager-86cbcdbc8-6s5c5\" (UID: \"0646bc9a-e23d-48e0-96ca-16468f810e7b\") " pod="openshift-jobset-operator/jobset-controller-manager-86cbcdbc8-6s5c5"
Nov 02 15:01:21.990804 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:01:21.990781    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/0646bc9a-e23d-48e0-96ca-16468f810e7b-manager-config\") pod \"jobset-controller-manager-86cbcdbc8-6s5c5\" (UID: \"0646bc9a-e23d-48e0-96ca-16468f810e7b\") " pod="openshift-jobset-operator/jobset-controller-manager-86cbcdbc8-6s5c5"
Nov 02 15:01:21.992919 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:01:21.992895    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0646bc9a-e23d-48e0-96ca-16468f810e7b-cert\") pod \"jobset-controller-manager-86cbcdbc8-6s5c5\" (UID: \"0646bc9a-e23d-48e0-96ca-16468f810e7b\") " pod="openshift-jobset-operator/jobset-controller-manager-86cbcdbc8-6s5c5"
Nov 02 15:01:21.993033 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:01:21.992922    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0646bc9a-e23d-48e0-96ca-16468f810e7b-metrics-certs\") pod \"jobset-controller-manager-86cbcdbc8-6s5c5\" (UID: \"0646bc9a-e23d-48e0-96ca-16468f810e7b\") " pod="openshift-jobset-operator/jobset-controller-manager-86cbcdbc8-6s5c5"
Nov 02 15:01:22.013534 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:01:22.013508    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kltw2\" (UniqueName: \"kubernetes.io/projected/0646bc9a-e23d-48e0-96ca-16468f810e7b-kube-api-access-kltw2\") pod \"jobset-controller-manager-86cbcdbc8-6s5c5\" (UID: \"0646bc9a-e23d-48e0-96ca-16468f810e7b\") " pod="openshift-jobset-operator/jobset-controller-manager-86cbcdbc8-6s5c5"
Nov 02 15:01:22.089558 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:01:22.089518    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-86cbcdbc8-6s5c5"
Nov 02 15:01:22.410830 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:01:22.410752    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-86cbcdbc8-6s5c5"]
Nov 02 15:01:22.415842 ip-10-0-135-138 kubenswrapper[5304]: W1102 15:01:22.415815    5304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0646bc9a_e23d_48e0_96ca_16468f810e7b.slice/crio-f26202fab00c9f4dbfbba2c54bacd7710a2ebd11e165b36a6ea8fc4fd9d5295e WatchSource:0}: Error finding container f26202fab00c9f4dbfbba2c54bacd7710a2ebd11e165b36a6ea8fc4fd9d5295e: Status 404 returned error can't find the container with id f26202fab00c9f4dbfbba2c54bacd7710a2ebd11e165b36a6ea8fc4fd9d5295e
Nov 02 15:01:23.046731 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:01:23.046695    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-77f469495f-6cphj" event={"ID":"a427d6de-97f8-4bbd-99d8-6327efcf1794","Type":"ContainerStarted","Data":"21dec41816669e7341dad9b230d95b4cf85113981ec874ce25c25ce7b4322af8"}
Nov 02 15:01:23.047974 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:01:23.047951    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-86cbcdbc8-6s5c5" event={"ID":"0646bc9a-e23d-48e0-96ca-16468f810e7b","Type":"ContainerStarted","Data":"f26202fab00c9f4dbfbba2c54bacd7710a2ebd11e165b36a6ea8fc4fd9d5295e"}
Nov 02 15:01:23.087949 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:01:23.087890    5304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-77f469495f-6cphj" podStartSLOduration=1.788276965 podStartE2EDuration="4.087872638s" totalImagesPullingTime="2.299595673s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:01:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:01:23.087752334 +0000 UTC m=+283.081576743" watchObservedRunningTime="2026-11-02 15:01:23.087872638 +0000 UTC m=+283.081697049"
Nov 02 15:01:24.800084 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:01:24.800043    5304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k6fjk" containerName="extractor"
Nov 02 15:01:26.060353 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:01:26.060317    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-86cbcdbc8-6s5c5" event={"ID":"0646bc9a-e23d-48e0-96ca-16468f810e7b","Type":"ContainerStarted","Data":"cf0ffc0ee9c8da7250f0676d40440dd5f0437ab0924bf96883b6d7e6018e10d3"}
Nov 02 15:01:26.060741 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:01:26.060398    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-jobset-operator/jobset-controller-manager-86cbcdbc8-6s5c5"
Nov 02 15:01:26.105561 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:01:26.105510    5304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-controller-manager-86cbcdbc8-6s5c5" podStartSLOduration=2.2799407289999998 podStartE2EDuration="5.105495089s" totalImagesPullingTime="2.82555436s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:01:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:01:26.103656659 +0000 UTC m=+286.097481068" watchObservedRunningTime="2026-11-02 15:01:26.105495089 +0000 UTC m=+286.099319561"
Nov 02 15:01:37.068881 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:01:37.068844    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-jobset-operator/jobset-controller-manager-86cbcdbc8-6s5c5"
Nov 02 15:01:54.797473 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:01:54.797432    5304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k6fjk" containerName="extractor"
Nov 02 15:02:24.739436 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:02:24.739402    5304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-769875db56-7lk7m"]
Nov 02 15:02:24.741913 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:02:24.741885    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-7lk7m"
Nov 02 15:02:24.756537 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:02:24.756513    5304 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Nov 02 15:02:24.768672 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:02:24.768647    5304 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Nov 02 15:02:24.769473 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:02:24.769449    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-7lk7m"]
Nov 02 15:02:24.774138 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:02:24.774106    5304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-metrics-tls\""
Nov 02 15:02:24.780369 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:02:24.780342    5304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-sl5x5\""
Nov 02 15:02:24.780465 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:02:24.780373    5304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\""
Nov 02 15:02:24.796651 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:02:24.796614    5304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k6fjk" containerName="extractor"
Nov 02 15:02:24.831416 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:02:24.831387    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3fc11777-186c-445a-8b71-2aefdf642e75-cert\") pod \"odh-model-controller-769875db56-7lk7m\" (UID: \"3fc11777-186c-445a-8b71-2aefdf642e75\") " pod="opendatahub/odh-model-controller-769875db56-7lk7m"
Nov 02 15:02:24.831603 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:02:24.831443    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7m6h2\" (UniqueName: \"kubernetes.io/projected/3fc11777-186c-445a-8b71-2aefdf642e75-kube-api-access-7m6h2\") pod \"odh-model-controller-769875db56-7lk7m\" (UID: \"3fc11777-186c-445a-8b71-2aefdf642e75\") " pod="opendatahub/odh-model-controller-769875db56-7lk7m"
Nov 02 15:02:24.831603 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:02:24.831507    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3fc11777-186c-445a-8b71-2aefdf642e75-metrics-certs\") pod \"odh-model-controller-769875db56-7lk7m\" (UID: \"3fc11777-186c-445a-8b71-2aefdf642e75\") " pod="opendatahub/odh-model-controller-769875db56-7lk7m"
Nov 02 15:02:24.932360 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:02:24.932324    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7m6h2\" (UniqueName: \"kubernetes.io/projected/3fc11777-186c-445a-8b71-2aefdf642e75-kube-api-access-7m6h2\") pod \"odh-model-controller-769875db56-7lk7m\" (UID: \"3fc11777-186c-445a-8b71-2aefdf642e75\") " pod="opendatahub/odh-model-controller-769875db56-7lk7m"
Nov 02 15:02:24.932360 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:02:24.932361    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3fc11777-186c-445a-8b71-2aefdf642e75-metrics-certs\") pod \"odh-model-controller-769875db56-7lk7m\" (UID: \"3fc11777-186c-445a-8b71-2aefdf642e75\") " pod="opendatahub/odh-model-controller-769875db56-7lk7m"
Nov 02 15:02:24.932600 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:02:24.932406    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3fc11777-186c-445a-8b71-2aefdf642e75-cert\") pod \"odh-model-controller-769875db56-7lk7m\" (UID: \"3fc11777-186c-445a-8b71-2aefdf642e75\") " pod="opendatahub/odh-model-controller-769875db56-7lk7m"
Nov 02 15:02:24.934833 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:02:24.934807    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3fc11777-186c-445a-8b71-2aefdf642e75-cert\") pod \"odh-model-controller-769875db56-7lk7m\" (UID: \"3fc11777-186c-445a-8b71-2aefdf642e75\") " pod="opendatahub/odh-model-controller-769875db56-7lk7m"
Nov 02 15:02:24.934949 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:02:24.934834    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3fc11777-186c-445a-8b71-2aefdf642e75-metrics-certs\") pod \"odh-model-controller-769875db56-7lk7m\" (UID: \"3fc11777-186c-445a-8b71-2aefdf642e75\") " pod="opendatahub/odh-model-controller-769875db56-7lk7m"
Nov 02 15:02:24.955015 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:02:24.954991    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7m6h2\" (UniqueName: \"kubernetes.io/projected/3fc11777-186c-445a-8b71-2aefdf642e75-kube-api-access-7m6h2\") pod \"odh-model-controller-769875db56-7lk7m\" (UID: \"3fc11777-186c-445a-8b71-2aefdf642e75\") " pod="opendatahub/odh-model-controller-769875db56-7lk7m"
Nov 02 15:02:25.052545 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:02:25.052462    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-7lk7m"
Nov 02 15:02:25.347447 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:02:25.347412    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-7lk7m"]
Nov 02 15:02:25.352547 ip-10-0-135-138 kubenswrapper[5304]: W1102 15:02:25.352517    5304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3fc11777_186c_445a_8b71_2aefdf642e75.slice/crio-8917b476d0690e30d7dda074252ceffa2ff04db35ebe51401a97dc66f276536d WatchSource:0}: Error finding container 8917b476d0690e30d7dda074252ceffa2ff04db35ebe51401a97dc66f276536d: Status 404 returned error can't find the container with id 8917b476d0690e30d7dda074252ceffa2ff04db35ebe51401a97dc66f276536d
Nov 02 15:02:26.258956 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:02:26.258922    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-7lk7m" event={"ID":"3fc11777-186c-445a-8b71-2aefdf642e75","Type":"ContainerStarted","Data":"8917b476d0690e30d7dda074252ceffa2ff04db35ebe51401a97dc66f276536d"}
Nov 02 15:02:28.213989 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:02:28.213951    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-r9b58"]
Nov 02 15:02:28.214540 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:02:28.214345    5304 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-r9b58" podUID="9ded5d29-7101-4a51-a370-abdba1e29cf2" containerName="discovery" containerID="cri-o://2f31a103549587f3eb98a3feb09b1cbe5a55ecf013004b563d0e2aec35e15f03" gracePeriod=30
Nov 02 15:02:28.267441 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:02:28.267411    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-7lk7m" event={"ID":"3fc11777-186c-445a-8b71-2aefdf642e75","Type":"ContainerStarted","Data":"79e0d02299ef0c620441c6764245f83e1ffb1e04b2c2d964c684461076d3b60f"}
Nov 02 15:02:28.267538 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:02:28.267483    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-769875db56-7lk7m"
Nov 02 15:02:28.307850 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:02:28.307732    5304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/odh-model-controller-769875db56-7lk7m" podStartSLOduration=1.970864164 podStartE2EDuration="4.307714961s" totalImagesPullingTime="2.336850797s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:02:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:02:28.307661912 +0000 UTC m=+348.301486328" watchObservedRunningTime="2026-11-02 15:02:28.307714961 +0000 UTC m=+348.301539369"
Nov 02 15:02:28.446818 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:02:28.446795    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-r9b58"
Nov 02 15:02:28.562958 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:02:28.562859    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9ded5d29-7101-4a51-a370-abdba1e29cf2-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/9ded5d29-7101-4a51-a370-abdba1e29cf2-istio-csr-ca-configmap\") pod \"9ded5d29-7101-4a51-a370-abdba1e29cf2\" (UID: \"9ded5d29-7101-4a51-a370-abdba1e29cf2\") "
Nov 02 15:02:28.562958 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:02:28.562918    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ded5d29-7101-4a51-a370-abdba1e29cf2-istio-token\" (UniqueName: \"kubernetes.io/projected/9ded5d29-7101-4a51-a370-abdba1e29cf2-istio-token\") pod \"9ded5d29-7101-4a51-a370-abdba1e29cf2\" (UID: \"9ded5d29-7101-4a51-a370-abdba1e29cf2\") "
Nov 02 15:02:28.562958 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:02:28.562947    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ded5d29-7101-4a51-a370-abdba1e29cf2-kube-api-access-gsvsp\" (UniqueName: \"kubernetes.io/projected/9ded5d29-7101-4a51-a370-abdba1e29cf2-kube-api-access-gsvsp\") pod \"9ded5d29-7101-4a51-a370-abdba1e29cf2\" (UID: \"9ded5d29-7101-4a51-a370-abdba1e29cf2\") "
Nov 02 15:02:28.563277 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:02:28.562975    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9ded5d29-7101-4a51-a370-abdba1e29cf2-cacerts\" (UniqueName: \"kubernetes.io/secret/9ded5d29-7101-4a51-a370-abdba1e29cf2-cacerts\") pod \"9ded5d29-7101-4a51-a370-abdba1e29cf2\" (UID: \"9ded5d29-7101-4a51-a370-abdba1e29cf2\") "
Nov 02 15:02:28.563277 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:02:28.563018    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9ded5d29-7101-4a51-a370-abdba1e29cf2-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/9ded5d29-7101-4a51-a370-abdba1e29cf2-istio-csr-dns-cert\") pod \"9ded5d29-7101-4a51-a370-abdba1e29cf2\" (UID: \"9ded5d29-7101-4a51-a370-abdba1e29cf2\") "
Nov 02 15:02:28.563277 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:02:28.563065    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ded5d29-7101-4a51-a370-abdba1e29cf2-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ded5d29-7101-4a51-a370-abdba1e29cf2-local-certs\") pod \"9ded5d29-7101-4a51-a370-abdba1e29cf2\" (UID: \"9ded5d29-7101-4a51-a370-abdba1e29cf2\") "
Nov 02 15:02:28.563277 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:02:28.563088    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9ded5d29-7101-4a51-a370-abdba1e29cf2-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9ded5d29-7101-4a51-a370-abdba1e29cf2-istio-kubeconfig\") pod \"9ded5d29-7101-4a51-a370-abdba1e29cf2\" (UID: \"9ded5d29-7101-4a51-a370-abdba1e29cf2\") "
Nov 02 15:02:28.563486 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:02:28.563346    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ded5d29-7101-4a51-a370-abdba1e29cf2-istio-csr-ca-configmap" pod "9ded5d29-7101-4a51-a370-abdba1e29cf2" (UID: "9ded5d29-7101-4a51-a370-abdba1e29cf2"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:02:28.565378 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:02:28.565343    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9ded5d29-7101-4a51-a370-abdba1e29cf2-istio-kubeconfig" pod "9ded5d29-7101-4a51-a370-abdba1e29cf2" (UID: "9ded5d29-7101-4a51-a370-abdba1e29cf2"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:02:28.565509 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:02:28.565463    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ded5d29-7101-4a51-a370-abdba1e29cf2-istio-token" pod "9ded5d29-7101-4a51-a370-abdba1e29cf2" (UID: "9ded5d29-7101-4a51-a370-abdba1e29cf2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:02:28.565754 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:02:28.565736    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ded5d29-7101-4a51-a370-abdba1e29cf2-local-certs" pod "9ded5d29-7101-4a51-a370-abdba1e29cf2" (UID: "9ded5d29-7101-4a51-a370-abdba1e29cf2"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:02:28.565821 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:02:28.565751    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ded5d29-7101-4a51-a370-abdba1e29cf2-kube-api-access-gsvsp" pod "9ded5d29-7101-4a51-a370-abdba1e29cf2" (UID: "9ded5d29-7101-4a51-a370-abdba1e29cf2"). InnerVolumeSpecName "kube-api-access-gsvsp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:02:28.565821 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:02:28.565759    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9ded5d29-7101-4a51-a370-abdba1e29cf2-cacerts" pod "9ded5d29-7101-4a51-a370-abdba1e29cf2" (UID: "9ded5d29-7101-4a51-a370-abdba1e29cf2"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:02:28.565821 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:02:28.565769    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9ded5d29-7101-4a51-a370-abdba1e29cf2-istio-csr-dns-cert" pod "9ded5d29-7101-4a51-a370-abdba1e29cf2" (UID: "9ded5d29-7101-4a51-a370-abdba1e29cf2"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:02:28.664566 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:02:28.664522    5304 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ded5d29-7101-4a51-a370-abdba1e29cf2-local-certs\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:02:28.664566 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:02:28.664557    5304 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9ded5d29-7101-4a51-a370-abdba1e29cf2-istio-kubeconfig\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:02:28.664566 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:02:28.664567    5304 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/9ded5d29-7101-4a51-a370-abdba1e29cf2-istio-csr-ca-configmap\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:02:28.664566 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:02:28.664575    5304 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ded5d29-7101-4a51-a370-abdba1e29cf2-istio-token\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:02:28.664831 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:02:28.664585    5304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gsvsp\" (UniqueName: \"kubernetes.io/projected/9ded5d29-7101-4a51-a370-abdba1e29cf2-kube-api-access-gsvsp\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:02:28.664831 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:02:28.664594    5304 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/9ded5d29-7101-4a51-a370-abdba1e29cf2-cacerts\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:02:28.664831 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:02:28.664604    5304 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/9ded5d29-7101-4a51-a370-abdba1e29cf2-istio-csr-dns-cert\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:02:29.272194 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:02:29.272154    5304 generic.go:356] "Generic (PLEG): container finished" podID="9ded5d29-7101-4a51-a370-abdba1e29cf2" containerID="2f31a103549587f3eb98a3feb09b1cbe5a55ecf013004b563d0e2aec35e15f03" exitCode=0
Nov 02 15:02:29.272663 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:02:29.272211    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-r9b58"
Nov 02 15:02:29.272663 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:02:29.272244    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-r9b58" event={"ID":"9ded5d29-7101-4a51-a370-abdba1e29cf2","Type":"ContainerDied","Data":"2f31a103549587f3eb98a3feb09b1cbe5a55ecf013004b563d0e2aec35e15f03"}
Nov 02 15:02:29.272663 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:02:29.272315    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-r9b58" event={"ID":"9ded5d29-7101-4a51-a370-abdba1e29cf2","Type":"ContainerDied","Data":"e1d2e800d0c989d93ae5d62141d2aae77880bd63f914d2eba246e88b72045ccc"}
Nov 02 15:02:29.272663 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:02:29.272347    5304 scope.go:122] "RemoveContainer" containerID="2f31a103549587f3eb98a3feb09b1cbe5a55ecf013004b563d0e2aec35e15f03"
Nov 02 15:02:29.285206 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:02:29.285186    5304 scope.go:122] "RemoveContainer" containerID="2f31a103549587f3eb98a3feb09b1cbe5a55ecf013004b563d0e2aec35e15f03"
Nov 02 15:02:29.285498 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:02:29.285476    5304 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f31a103549587f3eb98a3feb09b1cbe5a55ecf013004b563d0e2aec35e15f03\": container with ID starting with 2f31a103549587f3eb98a3feb09b1cbe5a55ecf013004b563d0e2aec35e15f03 not found: ID does not exist" containerID="2f31a103549587f3eb98a3feb09b1cbe5a55ecf013004b563d0e2aec35e15f03"
Nov 02 15:02:29.285542 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:02:29.285507    5304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f31a103549587f3eb98a3feb09b1cbe5a55ecf013004b563d0e2aec35e15f03"} err="failed to get container status \"2f31a103549587f3eb98a3feb09b1cbe5a55ecf013004b563d0e2aec35e15f03\": rpc error: code = NotFound desc = could not find container \"2f31a103549587f3eb98a3feb09b1cbe5a55ecf013004b563d0e2aec35e15f03\": container with ID starting with 2f31a103549587f3eb98a3feb09b1cbe5a55ecf013004b563d0e2aec35e15f03 not found: ID does not exist"
Nov 02 15:02:29.333343 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:02:29.333307    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-r9b58"]
Nov 02 15:02:29.344899 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:02:29.344863    5304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-r9b58"]
Nov 02 15:02:30.115485 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:02:30.115449    5304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9ded5d29-7101-4a51-a370-abdba1e29cf2" path="/var/lib/kubelet/pods/9ded5d29-7101-4a51-a370-abdba1e29cf2/volumes"
Nov 02 15:02:39.274801 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:02:39.274772    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-769875db56-7lk7m"
Nov 02 15:02:49.055252 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:02:49.055219    5304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-fnz8x"]
Nov 02 15:02:49.055619 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:02:49.055495    5304 cpu_manager.go:414] "containerMap: removing container" podUID="9ded5d29-7101-4a51-a370-abdba1e29cf2" containerName="discovery"
Nov 02 15:02:49.055619 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:02:49.055506    5304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ded5d29-7101-4a51-a370-abdba1e29cf2" containerName="discovery"
Nov 02 15:02:49.055619 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:02:49.055551    5304 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ded5d29-7101-4a51-a370-abdba1e29cf2" containerName="discovery"
Nov 02 15:02:49.060443 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:02:49.060419    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-fnz8x"
Nov 02 15:02:49.078475 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:02:49.078448    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-fnz8x"]
Nov 02 15:02:49.079309 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:02:49.079284    5304 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 15:02:49.079584 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:02:49.079565    5304 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 15:02:49.079692 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:02:49.079680    5304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Nov 02 15:02:49.079732 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:02:49.079710    5304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Nov 02 15:02:49.119072 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:02:49.119037    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9558ffc6-a62e-4516-9449-2ca11e0199cd-metrics-certs\") pod \"maas-api-6d669c7fcd-fnz8x\" (UID: \"9558ffc6-a62e-4516-9449-2ca11e0199cd\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-fnz8x"
Nov 02 15:02:49.119230 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:02:49.119100    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9558ffc6-a62e-4516-9449-2ca11e0199cd-openshift-service-ca\") pod \"maas-api-6d669c7fcd-fnz8x\" (UID: \"9558ffc6-a62e-4516-9449-2ca11e0199cd\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-fnz8x"
Nov 02 15:02:49.119289 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:02:49.119231    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-568xd\" (UniqueName: \"kubernetes.io/projected/9558ffc6-a62e-4516-9449-2ca11e0199cd-kube-api-access-568xd\") pod \"maas-api-6d669c7fcd-fnz8x\" (UID: \"9558ffc6-a62e-4516-9449-2ca11e0199cd\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-fnz8x"
Nov 02 15:02:49.119289 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:02:49.119266    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9558ffc6-a62e-4516-9449-2ca11e0199cd-maas-api-tls\") pod \"maas-api-6d669c7fcd-fnz8x\" (UID: \"9558ffc6-a62e-4516-9449-2ca11e0199cd\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-fnz8x"
Nov 02 15:02:49.220399 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:02:49.220358    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9558ffc6-a62e-4516-9449-2ca11e0199cd-openshift-service-ca\") pod \"maas-api-6d669c7fcd-fnz8x\" (UID: \"9558ffc6-a62e-4516-9449-2ca11e0199cd\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-fnz8x"
Nov 02 15:02:49.220566 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:02:49.220404    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-568xd\" (UniqueName: \"kubernetes.io/projected/9558ffc6-a62e-4516-9449-2ca11e0199cd-kube-api-access-568xd\") pod \"maas-api-6d669c7fcd-fnz8x\" (UID: \"9558ffc6-a62e-4516-9449-2ca11e0199cd\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-fnz8x"
Nov 02 15:02:49.220566 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:02:49.220430    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9558ffc6-a62e-4516-9449-2ca11e0199cd-maas-api-tls\") pod \"maas-api-6d669c7fcd-fnz8x\" (UID: \"9558ffc6-a62e-4516-9449-2ca11e0199cd\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-fnz8x"
Nov 02 15:02:49.220566 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:02:49.220450    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9558ffc6-a62e-4516-9449-2ca11e0199cd-metrics-certs\") pod \"maas-api-6d669c7fcd-fnz8x\" (UID: \"9558ffc6-a62e-4516-9449-2ca11e0199cd\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-fnz8x"
Nov 02 15:02:49.221093 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:02:49.221073    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9558ffc6-a62e-4516-9449-2ca11e0199cd-openshift-service-ca\") pod \"maas-api-6d669c7fcd-fnz8x\" (UID: \"9558ffc6-a62e-4516-9449-2ca11e0199cd\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-fnz8x"
Nov 02 15:02:49.222844 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:02:49.222823    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9558ffc6-a62e-4516-9449-2ca11e0199cd-metrics-certs\") pod \"maas-api-6d669c7fcd-fnz8x\" (UID: \"9558ffc6-a62e-4516-9449-2ca11e0199cd\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-fnz8x"
Nov 02 15:02:49.222933 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:02:49.222857    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9558ffc6-a62e-4516-9449-2ca11e0199cd-maas-api-tls\") pod \"maas-api-6d669c7fcd-fnz8x\" (UID: \"9558ffc6-a62e-4516-9449-2ca11e0199cd\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-fnz8x"
Nov 02 15:02:49.243918 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:02:49.243888    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-568xd\" (UniqueName: \"kubernetes.io/projected/9558ffc6-a62e-4516-9449-2ca11e0199cd-kube-api-access-568xd\") pod \"maas-api-6d669c7fcd-fnz8x\" (UID: \"9558ffc6-a62e-4516-9449-2ca11e0199cd\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-fnz8x"
Nov 02 15:02:49.372107 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:02:49.372071    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-fnz8x"
Nov 02 15:02:49.666075 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:02:49.665990    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-fnz8x"]
Nov 02 15:02:49.671015 ip-10-0-135-138 kubenswrapper[5304]: W1102 15:02:49.670986    5304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9558ffc6_a62e_4516_9449_2ca11e0199cd.slice/crio-7bf5c03aac522e3bfcba9114596c90c52a73d2f1a79c7f6a9fc9bb11d1bef4d6 WatchSource:0}: Error finding container 7bf5c03aac522e3bfcba9114596c90c52a73d2f1a79c7f6a9fc9bb11d1bef4d6: Status 404 returned error can't find the container with id 7bf5c03aac522e3bfcba9114596c90c52a73d2f1a79c7f6a9fc9bb11d1bef4d6
Nov 02 15:02:50.344369 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:02:50.344327    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-fnz8x" event={"ID":"9558ffc6-a62e-4516-9449-2ca11e0199cd","Type":"ContainerStarted","Data":"7bf5c03aac522e3bfcba9114596c90c52a73d2f1a79c7f6a9fc9bb11d1bef4d6"}
Nov 02 15:02:53.357035 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:02:53.356996    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-fnz8x" event={"ID":"9558ffc6-a62e-4516-9449-2ca11e0199cd","Type":"ContainerStarted","Data":"39dfbf1e12b29864423648ef22442c74f78a47de741199c4db9b12e553908414"}
Nov 02 15:02:53.357476 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:02:53.357153    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-fnz8x"
Nov 02 15:02:53.400389 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:02:53.400327    5304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-fnz8x" podStartSLOduration=0.87572724 podStartE2EDuration="4.400314213s" totalImagesPullingTime="3.524586973s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:02:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:02:53.400008937 +0000 UTC m=+373.393833346" watchObservedRunningTime="2026-11-02 15:02:53.400314213 +0000 UTC m=+373.394138622"
Nov 02 15:02:54.796121 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:02:54.796084    5304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k6fjk" containerName="extractor"
Nov 02 15:02:59.366276 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:02:59.366236    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-fnz8x"
Nov 02 15:03:00.185859 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:03:00.185821    5304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-7594s"]
Nov 02 15:03:00.187921 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:03:00.187901    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-7594s"
Nov 02 15:03:00.202858 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:03:00.202837    5304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-rsl2r\""
Nov 02 15:03:00.215418 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:03:00.215394    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-7594s"]
Nov 02 15:03:00.305454 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:03:00.305425    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wsqv4\" (UniqueName: \"kubernetes.io/projected/b871e6ba-1a58-4fca-8a2c-6feff17c71d2-kube-api-access-wsqv4\") pod \"authorino-6f6944bcdd-7594s\" (UID: \"b871e6ba-1a58-4fca-8a2c-6feff17c71d2\") " pod="kuadrant-system/authorino-6f6944bcdd-7594s"
Nov 02 15:03:00.406086 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:03:00.406048    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wsqv4\" (UniqueName: \"kubernetes.io/projected/b871e6ba-1a58-4fca-8a2c-6feff17c71d2-kube-api-access-wsqv4\") pod \"authorino-6f6944bcdd-7594s\" (UID: \"b871e6ba-1a58-4fca-8a2c-6feff17c71d2\") " pod="kuadrant-system/authorino-6f6944bcdd-7594s"
Nov 02 15:03:00.441028 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:03:00.440963    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wsqv4\" (UniqueName: \"kubernetes.io/projected/b871e6ba-1a58-4fca-8a2c-6feff17c71d2-kube-api-access-wsqv4\") pod \"authorino-6f6944bcdd-7594s\" (UID: \"b871e6ba-1a58-4fca-8a2c-6feff17c71d2\") " pod="kuadrant-system/authorino-6f6944bcdd-7594s"
Nov 02 15:03:00.466726 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:03:00.466697    5304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-59cd689d68-cznfp"]
Nov 02 15:03:00.469010 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:03:00.468991    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-59cd689d68-cznfp"
Nov 02 15:03:00.472319 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:03:00.472292    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-7594s"]
Nov 02 15:03:00.472476 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:03:00.472465    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-7594s"
Nov 02 15:03:00.489363 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:03:00.489335    5304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-cz4f2\""
Nov 02 15:03:00.498138 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:03:00.498098    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-59cd689d68-cznfp"]
Nov 02 15:03:00.509917 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:03:00.509888    5304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-n76cp"]
Nov 02 15:03:00.518377 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:03:00.518350    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-n76cp"
Nov 02 15:03:00.533383 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:03:00.533311    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-n76cp"]
Nov 02 15:03:00.592145 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:03:00.592090    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-n76cp"]
Nov 02 15:03:00.592313 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:03:00.592294    5304 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-6kdl6 tls-cert], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-6889cbd8c4-n76cp" podUID="68f963d5-d946-46aa-83d5-d1b1c0108217"
Nov 02 15:03:00.595159 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:03:00.595136    5304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Nov 02 15:03:00.607491 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:03:00.607467    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ff2f1cbd-70c5-48d0-b133-0e4ea14e8dec-maas-api-tls\") pod \"maas-api-59cd689d68-cznfp\" (UID: \"ff2f1cbd-70c5-48d0-b133-0e4ea14e8dec\") " pod="odh-ai-gateway-infra/maas-api-59cd689d68-cznfp"
Nov 02 15:03:00.607547 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:03:00.607498    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9rwsk\" (UniqueName: \"kubernetes.io/projected/ff2f1cbd-70c5-48d0-b133-0e4ea14e8dec-kube-api-access-9rwsk\") pod \"maas-api-59cd689d68-cznfp\" (UID: \"ff2f1cbd-70c5-48d0-b133-0e4ea14e8dec\") " pod="odh-ai-gateway-infra/maas-api-59cd689d68-cznfp"
Nov 02 15:03:00.607547 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:03:00.607521    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/68f963d5-d946-46aa-83d5-d1b1c0108217-tls-cert\") pod \"authorino-6889cbd8c4-n76cp\" (UID: \"68f963d5-d946-46aa-83d5-d1b1c0108217\") " pod="kuadrant-system/authorino-6889cbd8c4-n76cp"
Nov 02 15:03:00.607653 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:03:00.607641    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff2f1cbd-70c5-48d0-b133-0e4ea14e8dec-openshift-service-ca\") pod \"maas-api-59cd689d68-cznfp\" (UID: \"ff2f1cbd-70c5-48d0-b133-0e4ea14e8dec\") " pod="odh-ai-gateway-infra/maas-api-59cd689d68-cznfp"
Nov 02 15:03:00.607696 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:03:00.607678    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6kdl6\" (UniqueName: \"kubernetes.io/projected/68f963d5-d946-46aa-83d5-d1b1c0108217-kube-api-access-6kdl6\") pod \"authorino-6889cbd8c4-n76cp\" (UID: \"68f963d5-d946-46aa-83d5-d1b1c0108217\") " pod="kuadrant-system/authorino-6889cbd8c4-n76cp"
Nov 02 15:03:00.607733 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:03:00.607708    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ff2f1cbd-70c5-48d0-b133-0e4ea14e8dec-metrics-certs\") pod \"maas-api-59cd689d68-cznfp\" (UID: \"ff2f1cbd-70c5-48d0-b133-0e4ea14e8dec\") " pod="odh-ai-gateway-infra/maas-api-59cd689d68-cznfp"
Nov 02 15:03:00.624358 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:03:00.624332    5304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5c48987dbd-bk7v4"]
Nov 02 15:03:00.626947 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:03:00.626926    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5c48987dbd-bk7v4"
Nov 02 15:03:00.646229 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:03:00.646205    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5c48987dbd-bk7v4"]
Nov 02 15:03:00.708170 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:03:00.708083    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff2f1cbd-70c5-48d0-b133-0e4ea14e8dec-openshift-service-ca\") pod \"maas-api-59cd689d68-cznfp\" (UID: \"ff2f1cbd-70c5-48d0-b133-0e4ea14e8dec\") " pod="odh-ai-gateway-infra/maas-api-59cd689d68-cznfp"
Nov 02 15:03:00.708170 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:03:00.708140    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6kdl6\" (UniqueName: \"kubernetes.io/projected/68f963d5-d946-46aa-83d5-d1b1c0108217-kube-api-access-6kdl6\") pod \"authorino-6889cbd8c4-n76cp\" (UID: \"68f963d5-d946-46aa-83d5-d1b1c0108217\") " pod="kuadrant-system/authorino-6889cbd8c4-n76cp"
Nov 02 15:03:00.708391 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:03:00.708178    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ff2f1cbd-70c5-48d0-b133-0e4ea14e8dec-metrics-certs\") pod \"maas-api-59cd689d68-cznfp\" (UID: \"ff2f1cbd-70c5-48d0-b133-0e4ea14e8dec\") " pod="odh-ai-gateway-infra/maas-api-59cd689d68-cznfp"
Nov 02 15:03:00.708391 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:03:00.708197    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5hfhp\" (UniqueName: \"kubernetes.io/projected/aae93f38-7cf6-465a-8645-e5672b198d53-kube-api-access-5hfhp\") pod \"authorino-5c48987dbd-bk7v4\" (UID: \"aae93f38-7cf6-465a-8645-e5672b198d53\") " pod="kuadrant-system/authorino-5c48987dbd-bk7v4"
Nov 02 15:03:00.708391 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:03:00.708222    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ff2f1cbd-70c5-48d0-b133-0e4ea14e8dec-maas-api-tls\") pod \"maas-api-59cd689d68-cznfp\" (UID: \"ff2f1cbd-70c5-48d0-b133-0e4ea14e8dec\") " pod="odh-ai-gateway-infra/maas-api-59cd689d68-cznfp"
Nov 02 15:03:00.708391 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:03:00.708246    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9rwsk\" (UniqueName: \"kubernetes.io/projected/ff2f1cbd-70c5-48d0-b133-0e4ea14e8dec-kube-api-access-9rwsk\") pod \"maas-api-59cd689d68-cznfp\" (UID: \"ff2f1cbd-70c5-48d0-b133-0e4ea14e8dec\") " pod="odh-ai-gateway-infra/maas-api-59cd689d68-cznfp"
Nov 02 15:03:00.708391 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:03:00.708319    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/aae93f38-7cf6-465a-8645-e5672b198d53-tls-cert\") pod \"authorino-5c48987dbd-bk7v4\" (UID: \"aae93f38-7cf6-465a-8645-e5672b198d53\") " pod="kuadrant-system/authorino-5c48987dbd-bk7v4"
Nov 02 15:03:00.708675 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:03:00.708404    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/68f963d5-d946-46aa-83d5-d1b1c0108217-tls-cert\") pod \"authorino-6889cbd8c4-n76cp\" (UID: \"68f963d5-d946-46aa-83d5-d1b1c0108217\") " pod="kuadrant-system/authorino-6889cbd8c4-n76cp"
Nov 02 15:03:00.708945 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:03:00.708920    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff2f1cbd-70c5-48d0-b133-0e4ea14e8dec-openshift-service-ca\") pod \"maas-api-59cd689d68-cznfp\" (UID: \"ff2f1cbd-70c5-48d0-b133-0e4ea14e8dec\") " pod="odh-ai-gateway-infra/maas-api-59cd689d68-cznfp"
Nov 02 15:03:00.710718 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:03:00.710698    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ff2f1cbd-70c5-48d0-b133-0e4ea14e8dec-maas-api-tls\") pod \"maas-api-59cd689d68-cznfp\" (UID: \"ff2f1cbd-70c5-48d0-b133-0e4ea14e8dec\") " pod="odh-ai-gateway-infra/maas-api-59cd689d68-cznfp"
Nov 02 15:03:00.710832 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:03:00.710757    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ff2f1cbd-70c5-48d0-b133-0e4ea14e8dec-metrics-certs\") pod \"maas-api-59cd689d68-cznfp\" (UID: \"ff2f1cbd-70c5-48d0-b133-0e4ea14e8dec\") " pod="odh-ai-gateway-infra/maas-api-59cd689d68-cznfp"
Nov 02 15:03:00.710832 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:03:00.710794    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/68f963d5-d946-46aa-83d5-d1b1c0108217-tls-cert\") pod \"authorino-6889cbd8c4-n76cp\" (UID: \"68f963d5-d946-46aa-83d5-d1b1c0108217\") " pod="kuadrant-system/authorino-6889cbd8c4-n76cp"
Nov 02 15:03:00.732264 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:03:00.732237    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6kdl6\" (UniqueName: \"kubernetes.io/projected/68f963d5-d946-46aa-83d5-d1b1c0108217-kube-api-access-6kdl6\") pod \"authorino-6889cbd8c4-n76cp\" (UID: \"68f963d5-d946-46aa-83d5-d1b1c0108217\") " pod="kuadrant-system/authorino-6889cbd8c4-n76cp"
Nov 02 15:03:00.732399 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:03:00.732237    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9rwsk\" (UniqueName: \"kubernetes.io/projected/ff2f1cbd-70c5-48d0-b133-0e4ea14e8dec-kube-api-access-9rwsk\") pod \"maas-api-59cd689d68-cznfp\" (UID: \"ff2f1cbd-70c5-48d0-b133-0e4ea14e8dec\") " pod="odh-ai-gateway-infra/maas-api-59cd689d68-cznfp"
Nov 02 15:03:00.769820 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:03:00.769763    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-7594s"]
Nov 02 15:03:00.773960 ip-10-0-135-138 kubenswrapper[5304]: W1102 15:03:00.773935    5304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb871e6ba_1a58_4fca_8a2c_6feff17c71d2.slice/crio-7de4592c97b1540fbf666be0f5c0b1293bcc4a454f56847e9b0dad62cb46b1cc WatchSource:0}: Error finding container 7de4592c97b1540fbf666be0f5c0b1293bcc4a454f56847e9b0dad62cb46b1cc: Status 404 returned error can't find the container with id 7de4592c97b1540fbf666be0f5c0b1293bcc4a454f56847e9b0dad62cb46b1cc
Nov 02 15:03:00.781076 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:03:00.781055    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-59cd689d68-cznfp"
Nov 02 15:03:00.808876 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:03:00.808846    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5hfhp\" (UniqueName: \"kubernetes.io/projected/aae93f38-7cf6-465a-8645-e5672b198d53-kube-api-access-5hfhp\") pod \"authorino-5c48987dbd-bk7v4\" (UID: \"aae93f38-7cf6-465a-8645-e5672b198d53\") " pod="kuadrant-system/authorino-5c48987dbd-bk7v4"
Nov 02 15:03:00.808984 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:03:00.808882    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/aae93f38-7cf6-465a-8645-e5672b198d53-tls-cert\") pod \"authorino-5c48987dbd-bk7v4\" (UID: \"aae93f38-7cf6-465a-8645-e5672b198d53\") " pod="kuadrant-system/authorino-5c48987dbd-bk7v4"
Nov 02 15:03:00.811173 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:03:00.811155    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/aae93f38-7cf6-465a-8645-e5672b198d53-tls-cert\") pod \"authorino-5c48987dbd-bk7v4\" (UID: \"aae93f38-7cf6-465a-8645-e5672b198d53\") " pod="kuadrant-system/authorino-5c48987dbd-bk7v4"
Nov 02 15:03:00.832789 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:03:00.832767    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5hfhp\" (UniqueName: \"kubernetes.io/projected/aae93f38-7cf6-465a-8645-e5672b198d53-kube-api-access-5hfhp\") pod \"authorino-5c48987dbd-bk7v4\" (UID: \"aae93f38-7cf6-465a-8645-e5672b198d53\") " pod="kuadrant-system/authorino-5c48987dbd-bk7v4"
Nov 02 15:03:00.937047 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:03:00.937015    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5c48987dbd-bk7v4"
Nov 02 15:03:01.082028 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:03:01.081994    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-59cd689d68-cznfp"]
Nov 02 15:03:01.087176 ip-10-0-135-138 kubenswrapper[5304]: W1102 15:03:01.087149    5304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff2f1cbd_70c5_48d0_b133_0e4ea14e8dec.slice/crio-df7745087b8641938f66c42b877358300c3070a2943d7eb1ec2126a67e8395d6 WatchSource:0}: Error finding container df7745087b8641938f66c42b877358300c3070a2943d7eb1ec2126a67e8395d6: Status 404 returned error can't find the container with id df7745087b8641938f66c42b877358300c3070a2943d7eb1ec2126a67e8395d6
Nov 02 15:03:01.231206 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:03:01.231168    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5c48987dbd-bk7v4"]
Nov 02 15:03:01.235079 ip-10-0-135-138 kubenswrapper[5304]: W1102 15:03:01.235051    5304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podaae93f38_7cf6_465a_8645_e5672b198d53.slice/crio-d71f01efa850270e3e8a490350a62bc34a680664db02e1edb1c2217b28506d0e WatchSource:0}: Error finding container d71f01efa850270e3e8a490350a62bc34a680664db02e1edb1c2217b28506d0e: Status 404 returned error can't find the container with id d71f01efa850270e3e8a490350a62bc34a680664db02e1edb1c2217b28506d0e
Nov 02 15:03:01.386563 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:03:01.386527    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-7594s" event={"ID":"b871e6ba-1a58-4fca-8a2c-6feff17c71d2","Type":"ContainerStarted","Data":"7de4592c97b1540fbf666be0f5c0b1293bcc4a454f56847e9b0dad62cb46b1cc"}
Nov 02 15:03:01.387776 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:03:01.387746    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5c48987dbd-bk7v4" event={"ID":"aae93f38-7cf6-465a-8645-e5672b198d53","Type":"ContainerStarted","Data":"d71f01efa850270e3e8a490350a62bc34a680664db02e1edb1c2217b28506d0e"}
Nov 02 15:03:01.388888 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:03:01.388866    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-59cd689d68-cznfp" event={"ID":"ff2f1cbd-70c5-48d0-b133-0e4ea14e8dec","Type":"ContainerStarted","Data":"df7745087b8641938f66c42b877358300c3070a2943d7eb1ec2126a67e8395d6"}
Nov 02 15:03:01.388994 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:03:01.388890    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-n76cp"
Nov 02 15:03:01.396345 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:03:01.396323    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-n76cp"
Nov 02 15:03:01.514174 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:03:01.514078    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/68f963d5-d946-46aa-83d5-d1b1c0108217-tls-cert\" (UniqueName: \"kubernetes.io/secret/68f963d5-d946-46aa-83d5-d1b1c0108217-tls-cert\") pod \"68f963d5-d946-46aa-83d5-d1b1c0108217\" (UID: \"68f963d5-d946-46aa-83d5-d1b1c0108217\") "
Nov 02 15:03:01.514644 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:03:01.514185    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/68f963d5-d946-46aa-83d5-d1b1c0108217-kube-api-access-6kdl6\" (UniqueName: \"kubernetes.io/projected/68f963d5-d946-46aa-83d5-d1b1c0108217-kube-api-access-6kdl6\") pod \"68f963d5-d946-46aa-83d5-d1b1c0108217\" (UID: \"68f963d5-d946-46aa-83d5-d1b1c0108217\") "
Nov 02 15:03:01.522289 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:03:01.522257    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/68f963d5-d946-46aa-83d5-d1b1c0108217-tls-cert" pod "68f963d5-d946-46aa-83d5-d1b1c0108217" (UID: "68f963d5-d946-46aa-83d5-d1b1c0108217"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:03:01.533742 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:03:01.531812    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68f963d5-d946-46aa-83d5-d1b1c0108217-kube-api-access-6kdl6" pod "68f963d5-d946-46aa-83d5-d1b1c0108217" (UID: "68f963d5-d946-46aa-83d5-d1b1c0108217"). InnerVolumeSpecName "kube-api-access-6kdl6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:03:01.615856 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:03:01.615816    5304 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/68f963d5-d946-46aa-83d5-d1b1c0108217-tls-cert\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:03:01.615856 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:03:01.615855    5304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6kdl6\" (UniqueName: \"kubernetes.io/projected/68f963d5-d946-46aa-83d5-d1b1c0108217-kube-api-access-6kdl6\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:03:02.395231 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:03:02.395164    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-n76cp"
Nov 02 15:03:02.396012 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:03:02.395984    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-59cd689d68-cznfp" event={"ID":"ff2f1cbd-70c5-48d0-b133-0e4ea14e8dec","Type":"ContainerStarted","Data":"9e8be530eaadf1113ff4ac06481154ce80ce59a6fd181dc92c5a6ec9c3b5a103"}
Nov 02 15:03:02.396111 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:03:02.396042    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-59cd689d68-cznfp"
Nov 02 15:03:02.498137 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:03:02.498058    5304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-59cd689d68-cznfp" podStartSLOduration=2.185478662 podStartE2EDuration="2.498034646s" totalImagesPullingTime="312.555984ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:03:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:03:02.443246165 +0000 UTC m=+382.437070566" watchObservedRunningTime="2026-11-02 15:03:02.498034646 +0000 UTC m=+382.491859056"
Nov 02 15:03:02.520725 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:03:02.520689    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-n76cp"]
Nov 02 15:03:02.524961 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:03:02.524908    5304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-n76cp"]
Nov 02 15:03:04.116658 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:03:04.116624    5304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="68f963d5-d946-46aa-83d5-d1b1c0108217" path="/var/lib/kubelet/pods/68f963d5-d946-46aa-83d5-d1b1c0108217/volumes"
Nov 02 15:03:04.415484 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:03:04.415428    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5c48987dbd-bk7v4" event={"ID":"aae93f38-7cf6-465a-8645-e5672b198d53","Type":"ContainerStarted","Data":"ea81ca00dec69b86596fe0adc276ddee9b9918274f7b398377cd2b7ed588b522"}
Nov 02 15:03:04.417447 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:03:04.417409    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-7594s" event={"ID":"b871e6ba-1a58-4fca-8a2c-6feff17c71d2","Type":"ContainerStarted","Data":"dc83062c1adfa20b776785c5d1c536d9a15157b921f9236710d12663d9dd8646"}
Nov 02 15:03:04.417576 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:03:04.417512    5304 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-6f6944bcdd-7594s" podUID="b871e6ba-1a58-4fca-8a2c-6feff17c71d2" containerName="authorino" containerID="cri-o://dc83062c1adfa20b776785c5d1c536d9a15157b921f9236710d12663d9dd8646" gracePeriod=30
Nov 02 15:03:04.477376 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:03:04.477321    5304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-5c48987dbd-bk7v4" podStartSLOduration=1.857558496 podStartE2EDuration="4.477308244s" totalImagesPullingTime="2.619749748s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:03:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:03:04.477226305 +0000 UTC m=+384.471050716" watchObservedRunningTime="2026-11-02 15:03:04.477308244 +0000 UTC m=+384.471132652"
Nov 02 15:03:04.561748 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:03:04.561681    5304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-6f6944bcdd-7594s" podStartSLOduration=1.6967493139999998 podStartE2EDuration="4.561661974s" totalImagesPullingTime="2.86491266s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:03:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:03:04.559928979 +0000 UTC m=+384.553753388" watchObservedRunningTime="2026-11-02 15:03:04.561661974 +0000 UTC m=+384.555486384"
Nov 02 15:03:04.676951 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:03:04.676881    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-7594s"
Nov 02 15:03:04.847065 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:03:04.847020    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b871e6ba-1a58-4fca-8a2c-6feff17c71d2-kube-api-access-wsqv4\" (UniqueName: \"kubernetes.io/projected/b871e6ba-1a58-4fca-8a2c-6feff17c71d2-kube-api-access-wsqv4\") pod \"b871e6ba-1a58-4fca-8a2c-6feff17c71d2\" (UID: \"b871e6ba-1a58-4fca-8a2c-6feff17c71d2\") "
Nov 02 15:03:04.849294 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:03:04.849269    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b871e6ba-1a58-4fca-8a2c-6feff17c71d2-kube-api-access-wsqv4" pod "b871e6ba-1a58-4fca-8a2c-6feff17c71d2" (UID: "b871e6ba-1a58-4fca-8a2c-6feff17c71d2"). InnerVolumeSpecName "kube-api-access-wsqv4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:03:04.947697 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:03:04.947614    5304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wsqv4\" (UniqueName: \"kubernetes.io/projected/b871e6ba-1a58-4fca-8a2c-6feff17c71d2-kube-api-access-wsqv4\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:03:05.422300 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:03:05.422265    5304 generic.go:356] "Generic (PLEG): container finished" podID="b871e6ba-1a58-4fca-8a2c-6feff17c71d2" containerID="dc83062c1adfa20b776785c5d1c536d9a15157b921f9236710d12663d9dd8646" exitCode=0
Nov 02 15:03:05.422781 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:03:05.422317    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-7594s"
Nov 02 15:03:05.422781 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:03:05.422346    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-7594s" event={"ID":"b871e6ba-1a58-4fca-8a2c-6feff17c71d2","Type":"ContainerDied","Data":"dc83062c1adfa20b776785c5d1c536d9a15157b921f9236710d12663d9dd8646"}
Nov 02 15:03:05.422781 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:03:05.422385    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-7594s" event={"ID":"b871e6ba-1a58-4fca-8a2c-6feff17c71d2","Type":"ContainerDied","Data":"7de4592c97b1540fbf666be0f5c0b1293bcc4a454f56847e9b0dad62cb46b1cc"}
Nov 02 15:03:05.422781 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:03:05.422404    5304 scope.go:122] "RemoveContainer" containerID="dc83062c1adfa20b776785c5d1c536d9a15157b921f9236710d12663d9dd8646"
Nov 02 15:03:05.431408 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:03:05.431384    5304 scope.go:122] "RemoveContainer" containerID="dc83062c1adfa20b776785c5d1c536d9a15157b921f9236710d12663d9dd8646"
Nov 02 15:03:05.431659 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:03:05.431641    5304 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc83062c1adfa20b776785c5d1c536d9a15157b921f9236710d12663d9dd8646\": container with ID starting with dc83062c1adfa20b776785c5d1c536d9a15157b921f9236710d12663d9dd8646 not found: ID does not exist" containerID="dc83062c1adfa20b776785c5d1c536d9a15157b921f9236710d12663d9dd8646"
Nov 02 15:03:05.431715 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:03:05.431669    5304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc83062c1adfa20b776785c5d1c536d9a15157b921f9236710d12663d9dd8646"} err="failed to get container status \"dc83062c1adfa20b776785c5d1c536d9a15157b921f9236710d12663d9dd8646\": rpc error: code = NotFound desc = could not find container \"dc83062c1adfa20b776785c5d1c536d9a15157b921f9236710d12663d9dd8646\": container with ID starting with dc83062c1adfa20b776785c5d1c536d9a15157b921f9236710d12663d9dd8646 not found: ID does not exist"
Nov 02 15:03:05.475211 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:03:05.475179    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-7594s"]
Nov 02 15:03:05.479140 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:03:05.479104    5304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-7594s"]
Nov 02 15:03:06.115226 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:03:06.115196    5304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b871e6ba-1a58-4fca-8a2c-6feff17c71d2" path="/var/lib/kubelet/pods/b871e6ba-1a58-4fca-8a2c-6feff17c71d2/volumes"
Nov 02 15:03:08.409167 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:03:08.409142    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-59cd689d68-cznfp"
Nov 02 15:03:08.562412 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:03:08.562378    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-fnz8x"]
Nov 02 15:03:08.562603 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:03:08.562580    5304 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-fnz8x" podUID="9558ffc6-a62e-4516-9449-2ca11e0199cd" containerName="maas-api" containerID="cri-o://39dfbf1e12b29864423648ef22442c74f78a47de741199c4db9b12e553908414" gracePeriod=30
Nov 02 15:03:08.801064 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:03:08.801037    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-fnz8x"
Nov 02 15:03:08.980593 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:03:08.980499    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9558ffc6-a62e-4516-9449-2ca11e0199cd-metrics-certs\" (UniqueName: \"kubernetes.io/secret/9558ffc6-a62e-4516-9449-2ca11e0199cd-metrics-certs\") pod \"9558ffc6-a62e-4516-9449-2ca11e0199cd\" (UID: \"9558ffc6-a62e-4516-9449-2ca11e0199cd\") "
Nov 02 15:03:08.980593 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:03:08.980563    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9558ffc6-a62e-4516-9449-2ca11e0199cd-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9558ffc6-a62e-4516-9449-2ca11e0199cd-openshift-service-ca\") pod \"9558ffc6-a62e-4516-9449-2ca11e0199cd\" (UID: \"9558ffc6-a62e-4516-9449-2ca11e0199cd\") "
Nov 02 15:03:08.980820 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:03:08.980608    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9558ffc6-a62e-4516-9449-2ca11e0199cd-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9558ffc6-a62e-4516-9449-2ca11e0199cd-maas-api-tls\") pod \"9558ffc6-a62e-4516-9449-2ca11e0199cd\" (UID: \"9558ffc6-a62e-4516-9449-2ca11e0199cd\") "
Nov 02 15:03:08.980820 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:03:08.980655    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9558ffc6-a62e-4516-9449-2ca11e0199cd-kube-api-access-568xd\" (UniqueName: \"kubernetes.io/projected/9558ffc6-a62e-4516-9449-2ca11e0199cd-kube-api-access-568xd\") pod \"9558ffc6-a62e-4516-9449-2ca11e0199cd\" (UID: \"9558ffc6-a62e-4516-9449-2ca11e0199cd\") "
Nov 02 15:03:08.981083 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:03:08.981050    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9558ffc6-a62e-4516-9449-2ca11e0199cd-openshift-service-ca" pod "9558ffc6-a62e-4516-9449-2ca11e0199cd" (UID: "9558ffc6-a62e-4516-9449-2ca11e0199cd"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:03:08.982857 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:03:08.982828    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9558ffc6-a62e-4516-9449-2ca11e0199cd-kube-api-access-568xd" pod "9558ffc6-a62e-4516-9449-2ca11e0199cd" (UID: "9558ffc6-a62e-4516-9449-2ca11e0199cd"). InnerVolumeSpecName "kube-api-access-568xd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:03:08.982955 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:03:08.982865    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9558ffc6-a62e-4516-9449-2ca11e0199cd-metrics-certs" pod "9558ffc6-a62e-4516-9449-2ca11e0199cd" (UID: "9558ffc6-a62e-4516-9449-2ca11e0199cd"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:03:08.982955 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:03:08.982921    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9558ffc6-a62e-4516-9449-2ca11e0199cd-maas-api-tls" pod "9558ffc6-a62e-4516-9449-2ca11e0199cd" (UID: "9558ffc6-a62e-4516-9449-2ca11e0199cd"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:03:09.082189 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:03:09.082151    5304 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9558ffc6-a62e-4516-9449-2ca11e0199cd-openshift-service-ca\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:03:09.082189 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:03:09.082184    5304 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9558ffc6-a62e-4516-9449-2ca11e0199cd-maas-api-tls\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:03:09.082189 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:03:09.082194    5304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-568xd\" (UniqueName: \"kubernetes.io/projected/9558ffc6-a62e-4516-9449-2ca11e0199cd-kube-api-access-568xd\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:03:09.082419 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:03:09.082203    5304 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9558ffc6-a62e-4516-9449-2ca11e0199cd-metrics-certs\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:03:09.439332 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:03:09.439293    5304 generic.go:356] "Generic (PLEG): container finished" podID="9558ffc6-a62e-4516-9449-2ca11e0199cd" containerID="39dfbf1e12b29864423648ef22442c74f78a47de741199c4db9b12e553908414" exitCode=0
Nov 02 15:03:09.439821 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:03:09.439343    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-fnz8x" event={"ID":"9558ffc6-a62e-4516-9449-2ca11e0199cd","Type":"ContainerDied","Data":"39dfbf1e12b29864423648ef22442c74f78a47de741199c4db9b12e553908414"}
Nov 02 15:03:09.439821 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:03:09.439375    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-fnz8x"
Nov 02 15:03:09.439821 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:03:09.439387    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-fnz8x" event={"ID":"9558ffc6-a62e-4516-9449-2ca11e0199cd","Type":"ContainerDied","Data":"7bf5c03aac522e3bfcba9114596c90c52a73d2f1a79c7f6a9fc9bb11d1bef4d6"}
Nov 02 15:03:09.439821 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:03:09.439409    5304 scope.go:122] "RemoveContainer" containerID="39dfbf1e12b29864423648ef22442c74f78a47de741199c4db9b12e553908414"
Nov 02 15:03:09.447865 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:03:09.447842    5304 scope.go:122] "RemoveContainer" containerID="39dfbf1e12b29864423648ef22442c74f78a47de741199c4db9b12e553908414"
Nov 02 15:03:09.448118 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:03:09.448099    5304 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"39dfbf1e12b29864423648ef22442c74f78a47de741199c4db9b12e553908414\": container with ID starting with 39dfbf1e12b29864423648ef22442c74f78a47de741199c4db9b12e553908414 not found: ID does not exist" containerID="39dfbf1e12b29864423648ef22442c74f78a47de741199c4db9b12e553908414"
Nov 02 15:03:09.448206 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:03:09.448146    5304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"39dfbf1e12b29864423648ef22442c74f78a47de741199c4db9b12e553908414"} err="failed to get container status \"39dfbf1e12b29864423648ef22442c74f78a47de741199c4db9b12e553908414\": rpc error: code = NotFound desc = could not find container \"39dfbf1e12b29864423648ef22442c74f78a47de741199c4db9b12e553908414\": container with ID starting with 39dfbf1e12b29864423648ef22442c74f78a47de741199c4db9b12e553908414 not found: ID does not exist"
Nov 02 15:03:09.489919 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:03:09.489879    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-fnz8x"]
Nov 02 15:03:09.502094 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:03:09.502054    5304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-fnz8x"]
Nov 02 15:03:10.115806 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:03:10.115769    5304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9558ffc6-a62e-4516-9449-2ca11e0199cd" path="/var/lib/kubelet/pods/9558ffc6-a62e-4516-9449-2ca11e0199cd/volumes"
Nov 02 15:03:24.798113 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:03:24.798070    5304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k6fjk" containerName="extractor"
Nov 02 15:03:54.796211 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:03:54.796170    5304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k6fjk" containerName="extractor"
Nov 02 15:04:17.118187 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:04:17.118120    5304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-t8nst"]
Nov 02 15:04:17.118767 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:04:17.118432    5304 cpu_manager.go:414] "containerMap: removing container" podUID="9558ffc6-a62e-4516-9449-2ca11e0199cd" containerName="maas-api"
Nov 02 15:04:17.118767 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:04:17.118450    5304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9558ffc6-a62e-4516-9449-2ca11e0199cd" containerName="maas-api"
Nov 02 15:04:17.118767 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:04:17.118468    5304 cpu_manager.go:414] "containerMap: removing container" podUID="b871e6ba-1a58-4fca-8a2c-6feff17c71d2" containerName="authorino"
Nov 02 15:04:17.118767 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:04:17.118475    5304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b871e6ba-1a58-4fca-8a2c-6feff17c71d2" containerName="authorino"
Nov 02 15:04:17.118767 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:04:17.118553    5304 memory_manager.go:365] "RemoveStaleState removing state" podUID="b871e6ba-1a58-4fca-8a2c-6feff17c71d2" containerName="authorino"
Nov 02 15:04:17.118767 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:04:17.118569    5304 memory_manager.go:365] "RemoveStaleState removing state" podUID="9558ffc6-a62e-4516-9449-2ca11e0199cd" containerName="maas-api"
Nov 02 15:04:17.121213 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:04:17.121174    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-t8nst"
Nov 02 15:04:17.143316 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:04:17.143278    5304 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Nov 02 15:04:17.149100 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:04:17.149071    5304 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Nov 02 15:04:17.149238 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:04:17.149177    5304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"echo-simulator-kserve-self-signed-certs\""
Nov 02 15:04:17.149238 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:04:17.149226    5304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-w4xr6\""
Nov 02 15:04:17.156012 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:04:17.155990    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-t8nst"]
Nov 02 15:04:17.165588 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:04:17.165566    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6eaab677-88a0-49d8-9c89-5deec4c81a4d-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-t8nst\" (UID: \"6eaab677-88a0-49d8-9c89-5deec4c81a4d\") " pod="llm/echo-simulator-kserve-64b6f8f96c-t8nst"
Nov 02 15:04:17.165710 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:04:17.165597    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6eaab677-88a0-49d8-9c89-5deec4c81a4d-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-t8nst\" (UID: \"6eaab677-88a0-49d8-9c89-5deec4c81a4d\") " pod="llm/echo-simulator-kserve-64b6f8f96c-t8nst"
Nov 02 15:04:17.165710 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:04:17.165622    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6eaab677-88a0-49d8-9c89-5deec4c81a4d-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-t8nst\" (UID: \"6eaab677-88a0-49d8-9c89-5deec4c81a4d\") " pod="llm/echo-simulator-kserve-64b6f8f96c-t8nst"
Nov 02 15:04:17.165710 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:04:17.165646    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6eaab677-88a0-49d8-9c89-5deec4c81a4d-home\") pod \"echo-simulator-kserve-64b6f8f96c-t8nst\" (UID: \"6eaab677-88a0-49d8-9c89-5deec4c81a4d\") " pod="llm/echo-simulator-kserve-64b6f8f96c-t8nst"
Nov 02 15:04:17.165814 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:04:17.165759    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6eaab677-88a0-49d8-9c89-5deec4c81a4d-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-t8nst\" (UID: \"6eaab677-88a0-49d8-9c89-5deec4c81a4d\") " pod="llm/echo-simulator-kserve-64b6f8f96c-t8nst"
Nov 02 15:04:17.165814 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:04:17.165794    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wdzfm\" (UniqueName: \"kubernetes.io/projected/6eaab677-88a0-49d8-9c89-5deec4c81a4d-kube-api-access-wdzfm\") pod \"echo-simulator-kserve-64b6f8f96c-t8nst\" (UID: \"6eaab677-88a0-49d8-9c89-5deec4c81a4d\") " pod="llm/echo-simulator-kserve-64b6f8f96c-t8nst"
Nov 02 15:04:17.266587 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:04:17.266549    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6eaab677-88a0-49d8-9c89-5deec4c81a4d-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-t8nst\" (UID: \"6eaab677-88a0-49d8-9c89-5deec4c81a4d\") " pod="llm/echo-simulator-kserve-64b6f8f96c-t8nst"
Nov 02 15:04:17.266587 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:04:17.266589    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6eaab677-88a0-49d8-9c89-5deec4c81a4d-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-t8nst\" (UID: \"6eaab677-88a0-49d8-9c89-5deec4c81a4d\") " pod="llm/echo-simulator-kserve-64b6f8f96c-t8nst"
Nov 02 15:04:17.266826 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:04:17.266617    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6eaab677-88a0-49d8-9c89-5deec4c81a4d-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-t8nst\" (UID: \"6eaab677-88a0-49d8-9c89-5deec4c81a4d\") " pod="llm/echo-simulator-kserve-64b6f8f96c-t8nst"
Nov 02 15:04:17.266826 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:04:17.266644    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6eaab677-88a0-49d8-9c89-5deec4c81a4d-home\") pod \"echo-simulator-kserve-64b6f8f96c-t8nst\" (UID: \"6eaab677-88a0-49d8-9c89-5deec4c81a4d\") " pod="llm/echo-simulator-kserve-64b6f8f96c-t8nst"
Nov 02 15:04:17.266826 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:04:17.266684    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6eaab677-88a0-49d8-9c89-5deec4c81a4d-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-t8nst\" (UID: \"6eaab677-88a0-49d8-9c89-5deec4c81a4d\") " pod="llm/echo-simulator-kserve-64b6f8f96c-t8nst"
Nov 02 15:04:17.266826 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:04:17.266719    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wdzfm\" (UniqueName: \"kubernetes.io/projected/6eaab677-88a0-49d8-9c89-5deec4c81a4d-kube-api-access-wdzfm\") pod \"echo-simulator-kserve-64b6f8f96c-t8nst\" (UID: \"6eaab677-88a0-49d8-9c89-5deec4c81a4d\") " pod="llm/echo-simulator-kserve-64b6f8f96c-t8nst"
Nov 02 15:04:17.267107 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:04:17.267074    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6eaab677-88a0-49d8-9c89-5deec4c81a4d-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-t8nst\" (UID: \"6eaab677-88a0-49d8-9c89-5deec4c81a4d\") " pod="llm/echo-simulator-kserve-64b6f8f96c-t8nst"
Nov 02 15:04:17.267255 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:04:17.267085    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6eaab677-88a0-49d8-9c89-5deec4c81a4d-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-t8nst\" (UID: \"6eaab677-88a0-49d8-9c89-5deec4c81a4d\") " pod="llm/echo-simulator-kserve-64b6f8f96c-t8nst"
Nov 02 15:04:17.267255 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:04:17.267150    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6eaab677-88a0-49d8-9c89-5deec4c81a4d-home\") pod \"echo-simulator-kserve-64b6f8f96c-t8nst\" (UID: \"6eaab677-88a0-49d8-9c89-5deec4c81a4d\") " pod="llm/echo-simulator-kserve-64b6f8f96c-t8nst"
Nov 02 15:04:17.269058 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:04:17.269036    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6eaab677-88a0-49d8-9c89-5deec4c81a4d-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-t8nst\" (UID: \"6eaab677-88a0-49d8-9c89-5deec4c81a4d\") " pod="llm/echo-simulator-kserve-64b6f8f96c-t8nst"
Nov 02 15:04:17.269255 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:04:17.269236    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6eaab677-88a0-49d8-9c89-5deec4c81a4d-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-t8nst\" (UID: \"6eaab677-88a0-49d8-9c89-5deec4c81a4d\") " pod="llm/echo-simulator-kserve-64b6f8f96c-t8nst"
Nov 02 15:04:17.290501 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:04:17.290473    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wdzfm\" (UniqueName: \"kubernetes.io/projected/6eaab677-88a0-49d8-9c89-5deec4c81a4d-kube-api-access-wdzfm\") pod \"echo-simulator-kserve-64b6f8f96c-t8nst\" (UID: \"6eaab677-88a0-49d8-9c89-5deec4c81a4d\") " pod="llm/echo-simulator-kserve-64b6f8f96c-t8nst"
Nov 02 15:04:17.434618 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:04:17.434521    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-t8nst"
Nov 02 15:04:17.728379 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:04:17.728276    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-t8nst"]
Nov 02 15:04:17.733145 ip-10-0-135-138 kubenswrapper[5304]: W1102 15:04:17.733105    5304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6eaab677_88a0_49d8_9c89_5deec4c81a4d.slice/crio-a46ffedeeedea1d27eae1c5cf1ccbfa074796b3d95397bf8f17cca47850935b9 WatchSource:0}: Error finding container a46ffedeeedea1d27eae1c5cf1ccbfa074796b3d95397bf8f17cca47850935b9: Status 404 returned error can't find the container with id a46ffedeeedea1d27eae1c5cf1ccbfa074796b3d95397bf8f17cca47850935b9
Nov 02 15:04:18.673885 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:04:18.673846    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-t8nst" event={"ID":"6eaab677-88a0-49d8-9c89-5deec4c81a4d","Type":"ContainerStarted","Data":"a46ffedeeedea1d27eae1c5cf1ccbfa074796b3d95397bf8f17cca47850935b9"}
Nov 02 15:04:19.370780 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:04:19.370745    5304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-dzs48"]
Nov 02 15:04:19.372981 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:04:19.372954    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-dzs48"
Nov 02 15:04:19.387542 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:04:19.387515    5304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\""
Nov 02 15:04:19.400195 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:04:19.400169    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-dzs48"]
Nov 02 15:04:19.487566 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:04:19.487478    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ed68913c-1ebd-4bb6-b3d0-58a9a19e2192-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-dzs48\" (UID: \"ed68913c-1ebd-4bb6-b3d0-58a9a19e2192\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-dzs48"
Nov 02 15:04:19.487566 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:04:19.487517    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rcvkg\" (UniqueName: \"kubernetes.io/projected/ed68913c-1ebd-4bb6-b3d0-58a9a19e2192-kube-api-access-rcvkg\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-dzs48\" (UID: \"ed68913c-1ebd-4bb6-b3d0-58a9a19e2192\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-dzs48"
Nov 02 15:04:19.487779 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:04:19.487659    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ed68913c-1ebd-4bb6-b3d0-58a9a19e2192-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-dzs48\" (UID: \"ed68913c-1ebd-4bb6-b3d0-58a9a19e2192\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-dzs48"
Nov 02 15:04:19.487779 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:04:19.487677    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ed68913c-1ebd-4bb6-b3d0-58a9a19e2192-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-dzs48\" (UID: \"ed68913c-1ebd-4bb6-b3d0-58a9a19e2192\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-dzs48"
Nov 02 15:04:19.487779 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:04:19.487694    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ed68913c-1ebd-4bb6-b3d0-58a9a19e2192-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-dzs48\" (UID: \"ed68913c-1ebd-4bb6-b3d0-58a9a19e2192\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-dzs48"
Nov 02 15:04:19.487779 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:04:19.487708    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ed68913c-1ebd-4bb6-b3d0-58a9a19e2192-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-dzs48\" (UID: \"ed68913c-1ebd-4bb6-b3d0-58a9a19e2192\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-dzs48"
Nov 02 15:04:19.588351 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:04:19.588306    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ed68913c-1ebd-4bb6-b3d0-58a9a19e2192-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-dzs48\" (UID: \"ed68913c-1ebd-4bb6-b3d0-58a9a19e2192\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-dzs48"
Nov 02 15:04:19.588351 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:04:19.588358    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ed68913c-1ebd-4bb6-b3d0-58a9a19e2192-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-dzs48\" (UID: \"ed68913c-1ebd-4bb6-b3d0-58a9a19e2192\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-dzs48"
Nov 02 15:04:19.588611 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:04:19.588485    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ed68913c-1ebd-4bb6-b3d0-58a9a19e2192-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-dzs48\" (UID: \"ed68913c-1ebd-4bb6-b3d0-58a9a19e2192\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-dzs48"
Nov 02 15:04:19.588611 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:04:19.588529    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ed68913c-1ebd-4bb6-b3d0-58a9a19e2192-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-dzs48\" (UID: \"ed68913c-1ebd-4bb6-b3d0-58a9a19e2192\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-dzs48"
Nov 02 15:04:19.588611 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:04:19.588566    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ed68913c-1ebd-4bb6-b3d0-58a9a19e2192-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-dzs48\" (UID: \"ed68913c-1ebd-4bb6-b3d0-58a9a19e2192\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-dzs48"
Nov 02 15:04:19.588611 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:04:19.588596    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rcvkg\" (UniqueName: \"kubernetes.io/projected/ed68913c-1ebd-4bb6-b3d0-58a9a19e2192-kube-api-access-rcvkg\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-dzs48\" (UID: \"ed68913c-1ebd-4bb6-b3d0-58a9a19e2192\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-dzs48"
Nov 02 15:04:19.588814 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:04:19.588775    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ed68913c-1ebd-4bb6-b3d0-58a9a19e2192-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-dzs48\" (UID: \"ed68913c-1ebd-4bb6-b3d0-58a9a19e2192\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-dzs48"
Nov 02 15:04:19.588860 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:04:19.588836    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ed68913c-1ebd-4bb6-b3d0-58a9a19e2192-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-dzs48\" (UID: \"ed68913c-1ebd-4bb6-b3d0-58a9a19e2192\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-dzs48"
Nov 02 15:04:19.588912 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:04:19.588891    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ed68913c-1ebd-4bb6-b3d0-58a9a19e2192-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-dzs48\" (UID: \"ed68913c-1ebd-4bb6-b3d0-58a9a19e2192\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-dzs48"
Nov 02 15:04:19.590872 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:04:19.590846    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ed68913c-1ebd-4bb6-b3d0-58a9a19e2192-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-dzs48\" (UID: \"ed68913c-1ebd-4bb6-b3d0-58a9a19e2192\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-dzs48"
Nov 02 15:04:19.591110 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:04:19.591093    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ed68913c-1ebd-4bb6-b3d0-58a9a19e2192-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-dzs48\" (UID: \"ed68913c-1ebd-4bb6-b3d0-58a9a19e2192\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-dzs48"
Nov 02 15:04:19.612283 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:04:19.612247    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rcvkg\" (UniqueName: \"kubernetes.io/projected/ed68913c-1ebd-4bb6-b3d0-58a9a19e2192-kube-api-access-rcvkg\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-dzs48\" (UID: \"ed68913c-1ebd-4bb6-b3d0-58a9a19e2192\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-dzs48"
Nov 02 15:04:19.678684 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:04:19.678642    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-t8nst" event={"ID":"6eaab677-88a0-49d8-9c89-5deec4c81a4d","Type":"ContainerStarted","Data":"05516acd1eadcfd21f72cf9c2fd35b01ec5757cf68d44b4641065960694b6bc1"}
Nov 02 15:04:19.686014 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:04:19.685985    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-dzs48"
Nov 02 15:04:19.723094 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:04:19.721831    5304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/echo-simulator-kserve-64b6f8f96c-t8nst" podStartSLOduration=1.288999162 podStartE2EDuration="2.721809985s" totalImagesPullingTime="1.432810823s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:04:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:04:19.721271915 +0000 UTC m=+459.715096324" watchObservedRunningTime="2026-11-02 15:04:19.721809985 +0000 UTC m=+459.715634395"
Nov 02 15:04:19.979253 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:04:19.979209    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-dzs48"]
Nov 02 15:04:19.983880 ip-10-0-135-138 kubenswrapper[5304]: W1102 15:04:19.983854    5304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded68913c_1ebd_4bb6_b3d0_58a9a19e2192.slice/crio-be2bccca088150339ffea90f825ec6e917fe48f16aa45ff753e145583d525c61 WatchSource:0}: Error finding container be2bccca088150339ffea90f825ec6e917fe48f16aa45ff753e145583d525c61: Status 404 returned error can't find the container with id be2bccca088150339ffea90f825ec6e917fe48f16aa45ff753e145583d525c61
Nov 02 15:04:20.435085 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:04:20.435054    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/echo-simulator-kserve-64b6f8f96c-t8nst"
Nov 02 15:04:20.684193 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:04:20.684150    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-dzs48" event={"ID":"ed68913c-1ebd-4bb6-b3d0-58a9a19e2192","Type":"ContainerStarted","Data":"be2bccca088150339ffea90f825ec6e917fe48f16aa45ff753e145583d525c61"}
Nov 02 15:04:21.689225 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:04:21.689114    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-dzs48" event={"ID":"ed68913c-1ebd-4bb6-b3d0-58a9a19e2192","Type":"ContainerStarted","Data":"3bfe1f0f1fc12c9714831960972f9150654c3654f78db014807e2cf89d23ccec"}
Nov 02 15:04:21.731719 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:04:21.731663    5304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-dzs48" podStartSLOduration=1.31341715 podStartE2EDuration="2.731650356s" totalImagesPullingTime="1.418233206s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:04:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:04:21.730849782 +0000 UTC m=+461.724674195" watchObservedRunningTime="2026-11-02 15:04:21.731650356 +0000 UTC m=+461.725474764"
Nov 02 15:04:21.747355 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:04:21.747324    5304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nkbrk"]
Nov 02 15:04:21.750707 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:04:21.750684    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nkbrk"
Nov 02 15:04:21.769546 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:04:21.769518    5304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Nov 02 15:04:21.770548 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:04:21.770526    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nkbrk"]
Nov 02 15:04:21.815096 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:04:21.815062    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1f68d837-fff2-418b-84d7-c2d7eb21a57e-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-nkbrk\" (UID: \"1f68d837-fff2-418b-84d7-c2d7eb21a57e\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nkbrk"
Nov 02 15:04:21.815319 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:04:21.815143    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1f68d837-fff2-418b-84d7-c2d7eb21a57e-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-nkbrk\" (UID: \"1f68d837-fff2-418b-84d7-c2d7eb21a57e\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nkbrk"
Nov 02 15:04:21.815319 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:04:21.815271    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f7544\" (UniqueName: \"kubernetes.io/projected/1f68d837-fff2-418b-84d7-c2d7eb21a57e-kube-api-access-f7544\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-nkbrk\" (UID: \"1f68d837-fff2-418b-84d7-c2d7eb21a57e\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nkbrk"
Nov 02 15:04:21.815430 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:04:21.815339    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1f68d837-fff2-418b-84d7-c2d7eb21a57e-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-nkbrk\" (UID: \"1f68d837-fff2-418b-84d7-c2d7eb21a57e\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nkbrk"
Nov 02 15:04:21.815430 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:04:21.815407    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1f68d837-fff2-418b-84d7-c2d7eb21a57e-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-nkbrk\" (UID: \"1f68d837-fff2-418b-84d7-c2d7eb21a57e\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nkbrk"
Nov 02 15:04:21.815500 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:04:21.815442    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1f68d837-fff2-418b-84d7-c2d7eb21a57e-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-nkbrk\" (UID: \"1f68d837-fff2-418b-84d7-c2d7eb21a57e\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nkbrk"
Nov 02 15:04:21.916358 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:04:21.916321    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1f68d837-fff2-418b-84d7-c2d7eb21a57e-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-nkbrk\" (UID: \"1f68d837-fff2-418b-84d7-c2d7eb21a57e\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nkbrk"
Nov 02 15:04:21.916534 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:04:21.916381    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1f68d837-fff2-418b-84d7-c2d7eb21a57e-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-nkbrk\" (UID: \"1f68d837-fff2-418b-84d7-c2d7eb21a57e\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nkbrk"
Nov 02 15:04:21.916534 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:04:21.916406    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1f68d837-fff2-418b-84d7-c2d7eb21a57e-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-nkbrk\" (UID: \"1f68d837-fff2-418b-84d7-c2d7eb21a57e\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nkbrk"
Nov 02 15:04:21.916534 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:04:21.916426    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f7544\" (UniqueName: \"kubernetes.io/projected/1f68d837-fff2-418b-84d7-c2d7eb21a57e-kube-api-access-f7544\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-nkbrk\" (UID: \"1f68d837-fff2-418b-84d7-c2d7eb21a57e\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nkbrk"
Nov 02 15:04:21.916534 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:04:21.916442    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1f68d837-fff2-418b-84d7-c2d7eb21a57e-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-nkbrk\" (UID: \"1f68d837-fff2-418b-84d7-c2d7eb21a57e\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nkbrk"
Nov 02 15:04:21.916534 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:04:21.916483    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1f68d837-fff2-418b-84d7-c2d7eb21a57e-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-nkbrk\" (UID: \"1f68d837-fff2-418b-84d7-c2d7eb21a57e\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nkbrk"
Nov 02 15:04:21.916843 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:04:21.916819    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1f68d837-fff2-418b-84d7-c2d7eb21a57e-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-nkbrk\" (UID: \"1f68d837-fff2-418b-84d7-c2d7eb21a57e\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nkbrk"
Nov 02 15:04:21.916896 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:04:21.916852    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1f68d837-fff2-418b-84d7-c2d7eb21a57e-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-nkbrk\" (UID: \"1f68d837-fff2-418b-84d7-c2d7eb21a57e\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nkbrk"
Nov 02 15:04:21.916931 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:04:21.916909    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1f68d837-fff2-418b-84d7-c2d7eb21a57e-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-nkbrk\" (UID: \"1f68d837-fff2-418b-84d7-c2d7eb21a57e\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nkbrk"
Nov 02 15:04:21.918571 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:04:21.918555    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1f68d837-fff2-418b-84d7-c2d7eb21a57e-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-nkbrk\" (UID: \"1f68d837-fff2-418b-84d7-c2d7eb21a57e\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nkbrk"
Nov 02 15:04:21.918895 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:04:21.918879    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1f68d837-fff2-418b-84d7-c2d7eb21a57e-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-nkbrk\" (UID: \"1f68d837-fff2-418b-84d7-c2d7eb21a57e\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nkbrk"
Nov 02 15:04:21.952652 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:04:21.952583    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f7544\" (UniqueName: \"kubernetes.io/projected/1f68d837-fff2-418b-84d7-c2d7eb21a57e-kube-api-access-f7544\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-nkbrk\" (UID: \"1f68d837-fff2-418b-84d7-c2d7eb21a57e\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nkbrk"
Nov 02 15:04:22.061824 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:04:22.061792    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nkbrk"
Nov 02 15:04:22.358456 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:04:22.358418    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nkbrk"]
Nov 02 15:04:22.362991 ip-10-0-135-138 kubenswrapper[5304]: W1102 15:04:22.362964    5304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1f68d837_fff2_418b_84d7_c2d7eb21a57e.slice/crio-ef10866e4f7b8c6d1d5b64879f342908845116ed8ccaf8028e9cac07db1f9c3f WatchSource:0}: Error finding container ef10866e4f7b8c6d1d5b64879f342908845116ed8ccaf8028e9cac07db1f9c3f: Status 404 returned error can't find the container with id ef10866e4f7b8c6d1d5b64879f342908845116ed8ccaf8028e9cac07db1f9c3f
Nov 02 15:04:22.686794 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:04:22.686754    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-dzs48"
Nov 02 15:04:22.695724 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:04:22.695690    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nkbrk" event={"ID":"1f68d837-fff2-418b-84d7-c2d7eb21a57e","Type":"ContainerStarted","Data":"743e860ad61016ae4f7a8c23fccc789967d258b35109a0f59145be0761c12509"}
Nov 02 15:04:22.696164 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:04:22.695733    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nkbrk" event={"ID":"1f68d837-fff2-418b-84d7-c2d7eb21a57e","Type":"ContainerStarted","Data":"ef10866e4f7b8c6d1d5b64879f342908845116ed8ccaf8028e9cac07db1f9c3f"}
Nov 02 15:04:22.738832 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:04:22.738768    5304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nkbrk" podStartSLOduration=1.487083053 podStartE2EDuration="1.738754802s" totalImagesPullingTime="251.671749ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:04:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:04:22.737726242 +0000 UTC m=+462.731550670" watchObservedRunningTime="2026-11-02 15:04:22.738754802 +0000 UTC m=+462.732579211"
Nov 02 15:04:23.062307 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:04:23.062228    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nkbrk"
Nov 02 15:04:23.765829 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:04:23.765791    5304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-7t7f8"]
Nov 02 15:04:23.769457 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:04:23.769434    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-7t7f8"
Nov 02 15:04:23.784282 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:04:23.784255    5304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\""
Nov 02 15:04:23.797632 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:04:23.797598    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-7t7f8"]
Nov 02 15:04:23.834041 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:04:23.834006    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dr578\" (UniqueName: \"kubernetes.io/projected/b76c8a20-92cc-41ac-b1be-2a779ec1a5ed-kube-api-access-dr578\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-7t7f8\" (UID: \"b76c8a20-92cc-41ac-b1be-2a779ec1a5ed\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-7t7f8"
Nov 02 15:04:23.834238 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:04:23.834050    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b76c8a20-92cc-41ac-b1be-2a779ec1a5ed-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-7t7f8\" (UID: \"b76c8a20-92cc-41ac-b1be-2a779ec1a5ed\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-7t7f8"
Nov 02 15:04:23.834238 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:04:23.834141    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b76c8a20-92cc-41ac-b1be-2a779ec1a5ed-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-7t7f8\" (UID: \"b76c8a20-92cc-41ac-b1be-2a779ec1a5ed\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-7t7f8"
Nov 02 15:04:23.834238 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:04:23.834209    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b76c8a20-92cc-41ac-b1be-2a779ec1a5ed-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-7t7f8\" (UID: \"b76c8a20-92cc-41ac-b1be-2a779ec1a5ed\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-7t7f8"
Nov 02 15:04:23.834361 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:04:23.834304    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b76c8a20-92cc-41ac-b1be-2a779ec1a5ed-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-7t7f8\" (UID: \"b76c8a20-92cc-41ac-b1be-2a779ec1a5ed\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-7t7f8"
Nov 02 15:04:23.834407 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:04:23.834367    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b76c8a20-92cc-41ac-b1be-2a779ec1a5ed-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-7t7f8\" (UID: \"b76c8a20-92cc-41ac-b1be-2a779ec1a5ed\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-7t7f8"
Nov 02 15:04:23.935153 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:04:23.935092    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b76c8a20-92cc-41ac-b1be-2a779ec1a5ed-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-7t7f8\" (UID: \"b76c8a20-92cc-41ac-b1be-2a779ec1a5ed\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-7t7f8"
Nov 02 15:04:23.935153 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:04:23.935141    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b76c8a20-92cc-41ac-b1be-2a779ec1a5ed-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-7t7f8\" (UID: \"b76c8a20-92cc-41ac-b1be-2a779ec1a5ed\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-7t7f8"
Nov 02 15:04:23.935408 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:04:23.935284    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b76c8a20-92cc-41ac-b1be-2a779ec1a5ed-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-7t7f8\" (UID: \"b76c8a20-92cc-41ac-b1be-2a779ec1a5ed\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-7t7f8"
Nov 02 15:04:23.935408 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:04:23.935329    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b76c8a20-92cc-41ac-b1be-2a779ec1a5ed-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-7t7f8\" (UID: \"b76c8a20-92cc-41ac-b1be-2a779ec1a5ed\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-7t7f8"
Nov 02 15:04:23.935408 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:04:23.935381    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dr578\" (UniqueName: \"kubernetes.io/projected/b76c8a20-92cc-41ac-b1be-2a779ec1a5ed-kube-api-access-dr578\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-7t7f8\" (UID: \"b76c8a20-92cc-41ac-b1be-2a779ec1a5ed\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-7t7f8"
Nov 02 15:04:23.935582 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:04:23.935433    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b76c8a20-92cc-41ac-b1be-2a779ec1a5ed-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-7t7f8\" (UID: \"b76c8a20-92cc-41ac-b1be-2a779ec1a5ed\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-7t7f8"
Nov 02 15:04:23.935582 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:04:23.935535    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b76c8a20-92cc-41ac-b1be-2a779ec1a5ed-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-7t7f8\" (UID: \"b76c8a20-92cc-41ac-b1be-2a779ec1a5ed\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-7t7f8"
Nov 02 15:04:23.935681 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:04:23.935630    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b76c8a20-92cc-41ac-b1be-2a779ec1a5ed-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-7t7f8\" (UID: \"b76c8a20-92cc-41ac-b1be-2a779ec1a5ed\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-7t7f8"
Nov 02 15:04:23.935736 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:04:23.935672    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b76c8a20-92cc-41ac-b1be-2a779ec1a5ed-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-7t7f8\" (UID: \"b76c8a20-92cc-41ac-b1be-2a779ec1a5ed\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-7t7f8"
Nov 02 15:04:23.937524 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:04:23.937496    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b76c8a20-92cc-41ac-b1be-2a779ec1a5ed-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-7t7f8\" (UID: \"b76c8a20-92cc-41ac-b1be-2a779ec1a5ed\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-7t7f8"
Nov 02 15:04:23.937686 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:04:23.937668    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b76c8a20-92cc-41ac-b1be-2a779ec1a5ed-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-7t7f8\" (UID: \"b76c8a20-92cc-41ac-b1be-2a779ec1a5ed\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-7t7f8"
Nov 02 15:04:23.959951 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:04:23.959912    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dr578\" (UniqueName: \"kubernetes.io/projected/b76c8a20-92cc-41ac-b1be-2a779ec1a5ed-kube-api-access-dr578\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-7t7f8\" (UID: \"b76c8a20-92cc-41ac-b1be-2a779ec1a5ed\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-7t7f8"
Nov 02 15:04:24.103145 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:04:24.097321    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-7t7f8"
Nov 02 15:04:24.412452 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:04:24.412361    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-7t7f8"]
Nov 02 15:04:24.416694 ip-10-0-135-138 kubenswrapper[5304]: W1102 15:04:24.416666    5304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb76c8a20_92cc_41ac_b1be_2a779ec1a5ed.slice/crio-2ea84bf2c39e51ec5fcd2f00c9106cd18323bcb4ddc8ddf36bd85cec2047d9c0 WatchSource:0}: Error finding container 2ea84bf2c39e51ec5fcd2f00c9106cd18323bcb4ddc8ddf36bd85cec2047d9c0: Status 404 returned error can't find the container with id 2ea84bf2c39e51ec5fcd2f00c9106cd18323bcb4ddc8ddf36bd85cec2047d9c0
Nov 02 15:04:24.707723 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:04:24.707689    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-7t7f8" event={"ID":"b76c8a20-92cc-41ac-b1be-2a779ec1a5ed","Type":"ContainerStarted","Data":"66047b1f861929c85848396a72c516ef7eb44912d60f118afcc1355d9948ab39"}
Nov 02 15:04:24.707899 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:04:24.707745    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-7t7f8" event={"ID":"b76c8a20-92cc-41ac-b1be-2a779ec1a5ed","Type":"ContainerStarted","Data":"2ea84bf2c39e51ec5fcd2f00c9106cd18323bcb4ddc8ddf36bd85cec2047d9c0"}
Nov 02 15:04:24.750570 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:04:24.750516    5304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-7t7f8" podStartSLOduration=1.539370641 podStartE2EDuration="1.750501195s" totalImagesPullingTime="211.130554ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:04:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:04:24.750011917 +0000 UTC m=+464.743836327" watchObservedRunningTime="2026-11-02 15:04:24.750501195 +0000 UTC m=+464.744325595"
Nov 02 15:04:24.796192 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:04:24.796158    5304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k6fjk" containerName="extractor"
Nov 02 15:04:25.097835 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:04:25.097801    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-7t7f8"
Nov 02 15:04:25.932343 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:04:25.929816    5304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-rmdsn"]
Nov 02 15:04:25.933705 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:04:25.933680    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-rmdsn"
Nov 02 15:04:25.965506 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:04:25.965477    5304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\""
Nov 02 15:04:25.967060 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:04:25.967018    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-rmdsn"]
Nov 02 15:04:26.052706 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:04:26.052676    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8cca3fa3-d2b1-4abb-bbb5-a38dfaa3c6b3-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-rmdsn\" (UID: \"8cca3fa3-d2b1-4abb-bbb5-a38dfaa3c6b3\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-rmdsn"
Nov 02 15:04:26.052869 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:04:26.052717    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8cca3fa3-d2b1-4abb-bbb5-a38dfaa3c6b3-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-rmdsn\" (UID: \"8cca3fa3-d2b1-4abb-bbb5-a38dfaa3c6b3\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-rmdsn"
Nov 02 15:04:26.052869 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:04:26.052745    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8cca3fa3-d2b1-4abb-bbb5-a38dfaa3c6b3-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-rmdsn\" (UID: \"8cca3fa3-d2b1-4abb-bbb5-a38dfaa3c6b3\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-rmdsn"
Nov 02 15:04:26.052869 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:04:26.052859    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jm8vw\" (UniqueName: \"kubernetes.io/projected/8cca3fa3-d2b1-4abb-bbb5-a38dfaa3c6b3-kube-api-access-jm8vw\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-rmdsn\" (UID: \"8cca3fa3-d2b1-4abb-bbb5-a38dfaa3c6b3\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-rmdsn"
Nov 02 15:04:26.052986 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:04:26.052883    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8cca3fa3-d2b1-4abb-bbb5-a38dfaa3c6b3-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-rmdsn\" (UID: \"8cca3fa3-d2b1-4abb-bbb5-a38dfaa3c6b3\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-rmdsn"
Nov 02 15:04:26.052986 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:04:26.052904    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8cca3fa3-d2b1-4abb-bbb5-a38dfaa3c6b3-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-rmdsn\" (UID: \"8cca3fa3-d2b1-4abb-bbb5-a38dfaa3c6b3\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-rmdsn"
Nov 02 15:04:26.153341 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:04:26.153301    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8cca3fa3-d2b1-4abb-bbb5-a38dfaa3c6b3-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-rmdsn\" (UID: \"8cca3fa3-d2b1-4abb-bbb5-a38dfaa3c6b3\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-rmdsn"
Nov 02 15:04:26.153539 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:04:26.153358    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8cca3fa3-d2b1-4abb-bbb5-a38dfaa3c6b3-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-rmdsn\" (UID: \"8cca3fa3-d2b1-4abb-bbb5-a38dfaa3c6b3\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-rmdsn"
Nov 02 15:04:26.153539 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:04:26.153394    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8cca3fa3-d2b1-4abb-bbb5-a38dfaa3c6b3-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-rmdsn\" (UID: \"8cca3fa3-d2b1-4abb-bbb5-a38dfaa3c6b3\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-rmdsn"
Nov 02 15:04:26.153539 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:04:26.153459    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jm8vw\" (UniqueName: \"kubernetes.io/projected/8cca3fa3-d2b1-4abb-bbb5-a38dfaa3c6b3-kube-api-access-jm8vw\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-rmdsn\" (UID: \"8cca3fa3-d2b1-4abb-bbb5-a38dfaa3c6b3\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-rmdsn"
Nov 02 15:04:26.153539 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:04:26.153498    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8cca3fa3-d2b1-4abb-bbb5-a38dfaa3c6b3-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-rmdsn\" (UID: \"8cca3fa3-d2b1-4abb-bbb5-a38dfaa3c6b3\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-rmdsn"
Nov 02 15:04:26.153539 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:04:26.153530    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8cca3fa3-d2b1-4abb-bbb5-a38dfaa3c6b3-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-rmdsn\" (UID: \"8cca3fa3-d2b1-4abb-bbb5-a38dfaa3c6b3\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-rmdsn"
Nov 02 15:04:26.153978 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:04:26.153948    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8cca3fa3-d2b1-4abb-bbb5-a38dfaa3c6b3-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-rmdsn\" (UID: \"8cca3fa3-d2b1-4abb-bbb5-a38dfaa3c6b3\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-rmdsn"
Nov 02 15:04:26.154265 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:04:26.154037    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8cca3fa3-d2b1-4abb-bbb5-a38dfaa3c6b3-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-rmdsn\" (UID: \"8cca3fa3-d2b1-4abb-bbb5-a38dfaa3c6b3\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-rmdsn"
Nov 02 15:04:26.154265 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:04:26.154224    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8cca3fa3-d2b1-4abb-bbb5-a38dfaa3c6b3-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-rmdsn\" (UID: \"8cca3fa3-d2b1-4abb-bbb5-a38dfaa3c6b3\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-rmdsn"
Nov 02 15:04:26.156430 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:04:26.156405    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8cca3fa3-d2b1-4abb-bbb5-a38dfaa3c6b3-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-rmdsn\" (UID: \"8cca3fa3-d2b1-4abb-bbb5-a38dfaa3c6b3\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-rmdsn"
Nov 02 15:04:26.156542 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:04:26.156428    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8cca3fa3-d2b1-4abb-bbb5-a38dfaa3c6b3-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-rmdsn\" (UID: \"8cca3fa3-d2b1-4abb-bbb5-a38dfaa3c6b3\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-rmdsn"
Nov 02 15:04:26.177864 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:04:26.177837    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jm8vw\" (UniqueName: \"kubernetes.io/projected/8cca3fa3-d2b1-4abb-bbb5-a38dfaa3c6b3-kube-api-access-jm8vw\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-rmdsn\" (UID: \"8cca3fa3-d2b1-4abb-bbb5-a38dfaa3c6b3\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-rmdsn"
Nov 02 15:04:26.248894 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:04:26.248813    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-rmdsn"
Nov 02 15:04:26.547637 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:04:26.547535    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-rmdsn"]
Nov 02 15:04:26.552101 ip-10-0-135-138 kubenswrapper[5304]: W1102 15:04:26.552069    5304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8cca3fa3_d2b1_4abb_bbb5_a38dfaa3c6b3.slice/crio-577688d06d652c7e884d4d874cc78f457f967363c627b1797b485c533af2eaf6 WatchSource:0}: Error finding container 577688d06d652c7e884d4d874cc78f457f967363c627b1797b485c533af2eaf6: Status 404 returned error can't find the container with id 577688d06d652c7e884d4d874cc78f457f967363c627b1797b485c533af2eaf6
Nov 02 15:04:26.717636 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:04:26.717602    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-rmdsn" event={"ID":"8cca3fa3-d2b1-4abb-bbb5-a38dfaa3c6b3","Type":"ContainerStarted","Data":"577688d06d652c7e884d4d874cc78f457f967363c627b1797b485c533af2eaf6"}
Nov 02 15:04:27.434860 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:04:27.434817    5304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/echo-simulator-kserve-64b6f8f96c-t8nst"
Nov 02 15:04:27.448536 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:04:27.448258    5304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/echo-simulator-kserve-64b6f8f96c-t8nst"
Nov 02 15:04:27.723022 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:04:27.722936    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-rmdsn" event={"ID":"8cca3fa3-d2b1-4abb-bbb5-a38dfaa3c6b3","Type":"ContainerStarted","Data":"5122e1bf76f7db0481a9e69a3315abbe38abc7ba9b706862316b5cff47f58e7c"}
Nov 02 15:04:27.738225 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:04:27.738182    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/echo-simulator-kserve-64b6f8f96c-t8nst"
Nov 02 15:04:27.766411 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:04:27.766353    5304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-rmdsn" podStartSLOduration=2.558673662 podStartE2EDuration="2.766333519s" totalImagesPullingTime="207.659857ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:04:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:04:27.765269694 +0000 UTC m=+467.759094104" watchObservedRunningTime="2026-11-02 15:04:27.766333519 +0000 UTC m=+467.760157931"
Nov 02 15:04:28.249652 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:04:28.249616    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-rmdsn"
Nov 02 15:04:29.251382 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:04:29.251341    5304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-56lvt"]
Nov 02 15:04:29.254844 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:04:29.254822    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-56lvt"
Nov 02 15:04:29.269636 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:04:29.269611    5304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\""
Nov 02 15:04:29.283366 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:04:29.283332    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-56lvt"]
Nov 02 15:04:29.379187 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:04:29.379145    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/569cd1bd-5016-4c33-98d7-d9b76c93de43-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-56lvt\" (UID: \"569cd1bd-5016-4c33-98d7-d9b76c93de43\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-56lvt"
Nov 02 15:04:29.379373 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:04:29.379208    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/569cd1bd-5016-4c33-98d7-d9b76c93de43-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-56lvt\" (UID: \"569cd1bd-5016-4c33-98d7-d9b76c93de43\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-56lvt"
Nov 02 15:04:29.379373 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:04:29.379252    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/569cd1bd-5016-4c33-98d7-d9b76c93de43-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-56lvt\" (UID: \"569cd1bd-5016-4c33-98d7-d9b76c93de43\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-56lvt"
Nov 02 15:04:29.379373 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:04:29.379322    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/569cd1bd-5016-4c33-98d7-d9b76c93de43-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-56lvt\" (UID: \"569cd1bd-5016-4c33-98d7-d9b76c93de43\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-56lvt"
Nov 02 15:04:29.379373 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:04:29.379352    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/569cd1bd-5016-4c33-98d7-d9b76c93de43-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-56lvt\" (UID: \"569cd1bd-5016-4c33-98d7-d9b76c93de43\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-56lvt"
Nov 02 15:04:29.379528 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:04:29.379385    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5lbbq\" (UniqueName: \"kubernetes.io/projected/569cd1bd-5016-4c33-98d7-d9b76c93de43-kube-api-access-5lbbq\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-56lvt\" (UID: \"569cd1bd-5016-4c33-98d7-d9b76c93de43\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-56lvt"
Nov 02 15:04:29.480295 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:04:29.480255    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/569cd1bd-5016-4c33-98d7-d9b76c93de43-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-56lvt\" (UID: \"569cd1bd-5016-4c33-98d7-d9b76c93de43\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-56lvt"
Nov 02 15:04:29.480501 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:04:29.480308    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/569cd1bd-5016-4c33-98d7-d9b76c93de43-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-56lvt\" (UID: \"569cd1bd-5016-4c33-98d7-d9b76c93de43\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-56lvt"
Nov 02 15:04:29.480501 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:04:29.480338    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/569cd1bd-5016-4c33-98d7-d9b76c93de43-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-56lvt\" (UID: \"569cd1bd-5016-4c33-98d7-d9b76c93de43\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-56lvt"
Nov 02 15:04:29.480501 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:04:29.480358    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/569cd1bd-5016-4c33-98d7-d9b76c93de43-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-56lvt\" (UID: \"569cd1bd-5016-4c33-98d7-d9b76c93de43\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-56lvt"
Nov 02 15:04:29.480501 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:04:29.480392    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5lbbq\" (UniqueName: \"kubernetes.io/projected/569cd1bd-5016-4c33-98d7-d9b76c93de43-kube-api-access-5lbbq\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-56lvt\" (UID: \"569cd1bd-5016-4c33-98d7-d9b76c93de43\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-56lvt"
Nov 02 15:04:29.480501 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:04:29.480424    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/569cd1bd-5016-4c33-98d7-d9b76c93de43-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-56lvt\" (UID: \"569cd1bd-5016-4c33-98d7-d9b76c93de43\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-56lvt"
Nov 02 15:04:29.480796 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:04:29.480744    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/569cd1bd-5016-4c33-98d7-d9b76c93de43-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-56lvt\" (UID: \"569cd1bd-5016-4c33-98d7-d9b76c93de43\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-56lvt"
Nov 02 15:04:29.480796 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:04:29.480776    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/569cd1bd-5016-4c33-98d7-d9b76c93de43-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-56lvt\" (UID: \"569cd1bd-5016-4c33-98d7-d9b76c93de43\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-56lvt"
Nov 02 15:04:29.480872 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:04:29.480791    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/569cd1bd-5016-4c33-98d7-d9b76c93de43-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-56lvt\" (UID: \"569cd1bd-5016-4c33-98d7-d9b76c93de43\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-56lvt"
Nov 02 15:04:29.482600 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:04:29.482570    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/569cd1bd-5016-4c33-98d7-d9b76c93de43-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-56lvt\" (UID: \"569cd1bd-5016-4c33-98d7-d9b76c93de43\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-56lvt"
Nov 02 15:04:29.482825 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:04:29.482804    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/569cd1bd-5016-4c33-98d7-d9b76c93de43-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-56lvt\" (UID: \"569cd1bd-5016-4c33-98d7-d9b76c93de43\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-56lvt"
Nov 02 15:04:29.503696 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:04:29.503639    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5lbbq\" (UniqueName: \"kubernetes.io/projected/569cd1bd-5016-4c33-98d7-d9b76c93de43-kube-api-access-5lbbq\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-56lvt\" (UID: \"569cd1bd-5016-4c33-98d7-d9b76c93de43\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-56lvt"
Nov 02 15:04:29.567845 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:04:29.567809    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-56lvt"
Nov 02 15:04:29.686993 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:04:29.686950    5304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-dzs48"
Nov 02 15:04:29.699220 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:04:29.699188    5304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-dzs48"
Nov 02 15:04:29.741983 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:04:29.741943    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-dzs48"
Nov 02 15:04:29.865421 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:04:29.865391    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-56lvt"]
Nov 02 15:04:29.870301 ip-10-0-135-138 kubenswrapper[5304]: W1102 15:04:29.870270    5304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod569cd1bd_5016_4c33_98d7_d9b76c93de43.slice/crio-6c6e8f7ad398fcfdccdce7826adb15649d9bcf472b67a1801e9511c5d350b1db WatchSource:0}: Error finding container 6c6e8f7ad398fcfdccdce7826adb15649d9bcf472b67a1801e9511c5d350b1db: Status 404 returned error can't find the container with id 6c6e8f7ad398fcfdccdce7826adb15649d9bcf472b67a1801e9511c5d350b1db
Nov 02 15:04:30.736833 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:04:30.736792    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-56lvt" event={"ID":"569cd1bd-5016-4c33-98d7-d9b76c93de43","Type":"ContainerStarted","Data":"8ce83a73dd8aff501e4adc2a2f77309c6d5aed2c0f237f87fff1fb88362ad44b"}
Nov 02 15:04:30.737266 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:04:30.736859    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-56lvt" event={"ID":"569cd1bd-5016-4c33-98d7-d9b76c93de43","Type":"ContainerStarted","Data":"6c6e8f7ad398fcfdccdce7826adb15649d9bcf472b67a1801e9511c5d350b1db"}
Nov 02 15:04:31.568094 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:04:31.568060    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-56lvt"
Nov 02 15:04:32.062530 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:04:32.062476    5304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nkbrk"
Nov 02 15:04:32.075478 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:04:32.075450    5304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nkbrk"
Nov 02 15:04:32.119577 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:04:32.119528    5304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-56lvt" podStartSLOduration=2.891081087 podStartE2EDuration="3.119514448s" totalImagesPullingTime="228.433361ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:04:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:04:30.780660999 +0000 UTC m=+470.774485410" watchObservedRunningTime="2026-11-02 15:04:32.119514448 +0000 UTC m=+472.113338858"
Nov 02 15:04:32.756709 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:04:32.756681    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nkbrk"
Nov 02 15:04:34.098181 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:04:34.098141    5304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-7t7f8"
Nov 02 15:04:34.110624 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:04:34.110596    5304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-7t7f8"
Nov 02 15:04:34.766549 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:04:34.766513    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-7t7f8"
Nov 02 15:04:36.250034 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:04:36.249994    5304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-rmdsn"
Nov 02 15:04:36.262521 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:04:36.262497    5304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-rmdsn"
Nov 02 15:04:36.772925 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:04:36.772898    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-rmdsn"
Nov 02 15:04:39.568930 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:04:39.568886    5304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-56lvt"
Nov 02 15:04:39.581182 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:04:39.581158    5304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-56lvt"
Nov 02 15:04:39.785441 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:04:39.785412    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-56lvt"
Nov 02 15:04:45.044403 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:04:45.044357    5304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-84d5ffc6f9-rvpgh"]
Nov 02 15:04:45.047964 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:04:45.047946    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-84d5ffc6f9-rvpgh"
Nov 02 15:04:45.066099 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:04:45.066072    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-84d5ffc6f9-rvpgh"]
Nov 02 15:04:45.228283 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:04:45.228234    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/2dd4123d-04a7-4eee-b14e-8c79b45a310d-tls-cert\") pod \"authorino-84d5ffc6f9-rvpgh\" (UID: \"2dd4123d-04a7-4eee-b14e-8c79b45a310d\") " pod="kuadrant-system/authorino-84d5ffc6f9-rvpgh"
Nov 02 15:04:45.228283 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:04:45.228288    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kc5s2\" (UniqueName: \"kubernetes.io/projected/2dd4123d-04a7-4eee-b14e-8c79b45a310d-kube-api-access-kc5s2\") pod \"authorino-84d5ffc6f9-rvpgh\" (UID: \"2dd4123d-04a7-4eee-b14e-8c79b45a310d\") " pod="kuadrant-system/authorino-84d5ffc6f9-rvpgh"
Nov 02 15:04:45.329167 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:04:45.329051    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kc5s2\" (UniqueName: \"kubernetes.io/projected/2dd4123d-04a7-4eee-b14e-8c79b45a310d-kube-api-access-kc5s2\") pod \"authorino-84d5ffc6f9-rvpgh\" (UID: \"2dd4123d-04a7-4eee-b14e-8c79b45a310d\") " pod="kuadrant-system/authorino-84d5ffc6f9-rvpgh"
Nov 02 15:04:45.329167 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:04:45.329159    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/2dd4123d-04a7-4eee-b14e-8c79b45a310d-tls-cert\") pod \"authorino-84d5ffc6f9-rvpgh\" (UID: \"2dd4123d-04a7-4eee-b14e-8c79b45a310d\") " pod="kuadrant-system/authorino-84d5ffc6f9-rvpgh"
Nov 02 15:04:45.331625 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:04:45.331605    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/2dd4123d-04a7-4eee-b14e-8c79b45a310d-tls-cert\") pod \"authorino-84d5ffc6f9-rvpgh\" (UID: \"2dd4123d-04a7-4eee-b14e-8c79b45a310d\") " pod="kuadrant-system/authorino-84d5ffc6f9-rvpgh"
Nov 02 15:04:45.355718 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:04:45.355684    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kc5s2\" (UniqueName: \"kubernetes.io/projected/2dd4123d-04a7-4eee-b14e-8c79b45a310d-kube-api-access-kc5s2\") pod \"authorino-84d5ffc6f9-rvpgh\" (UID: \"2dd4123d-04a7-4eee-b14e-8c79b45a310d\") " pod="kuadrant-system/authorino-84d5ffc6f9-rvpgh"
Nov 02 15:04:45.359479 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:04:45.359462    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-84d5ffc6f9-rvpgh"
Nov 02 15:04:45.654041 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:04:45.654009    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-84d5ffc6f9-rvpgh"]
Nov 02 15:04:45.659042 ip-10-0-135-138 kubenswrapper[5304]: W1102 15:04:45.659010    5304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2dd4123d_04a7_4eee_b14e_8c79b45a310d.slice/crio-f428f06571fd394ff4b871ea4990013c2bcb53bc0ac7e19ad49246a83f723a17 WatchSource:0}: Error finding container f428f06571fd394ff4b871ea4990013c2bcb53bc0ac7e19ad49246a83f723a17: Status 404 returned error can't find the container with id f428f06571fd394ff4b871ea4990013c2bcb53bc0ac7e19ad49246a83f723a17
Nov 02 15:04:45.660424 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:04:45.660401    5304 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 02 15:04:45.801705 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:04:45.801671    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-84d5ffc6f9-rvpgh" event={"ID":"2dd4123d-04a7-4eee-b14e-8c79b45a310d","Type":"ContainerStarted","Data":"f428f06571fd394ff4b871ea4990013c2bcb53bc0ac7e19ad49246a83f723a17"}
Nov 02 15:04:46.808162 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:04:46.808059    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-84d5ffc6f9-rvpgh" event={"ID":"2dd4123d-04a7-4eee-b14e-8c79b45a310d","Type":"ContainerStarted","Data":"cde49ad53f256b687d7f0e003fc29a581eaac5817944bd6aceba77732a03f67d"}
Nov 02 15:04:46.848238 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:04:46.848174    5304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-84d5ffc6f9-rvpgh" podStartSLOduration=1.07753014 podStartE2EDuration="1.848156817s" totalImagesPullingTime="770.626677ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:04:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:04:46.846857943 +0000 UTC m=+486.840682352" watchObservedRunningTime="2026-11-02 15:04:46.848156817 +0000 UTC m=+486.841981224"
Nov 02 15:04:46.876531 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:04:46.875708    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5c48987dbd-bk7v4"]
Nov 02 15:04:46.876531 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:04:46.875928    5304 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-5c48987dbd-bk7v4" podUID="aae93f38-7cf6-465a-8645-e5672b198d53" containerName="authorino" containerID="cri-o://ea81ca00dec69b86596fe0adc276ddee9b9918274f7b398377cd2b7ed588b522" gracePeriod=30
Nov 02 15:04:47.135366 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:04:47.135328    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5c48987dbd-bk7v4"
Nov 02 15:04:47.247507 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:04:47.247469    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aae93f38-7cf6-465a-8645-e5672b198d53-tls-cert\" (UniqueName: \"kubernetes.io/secret/aae93f38-7cf6-465a-8645-e5672b198d53-tls-cert\") pod \"aae93f38-7cf6-465a-8645-e5672b198d53\" (UID: \"aae93f38-7cf6-465a-8645-e5672b198d53\") "
Nov 02 15:04:47.247682 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:04:47.247529    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aae93f38-7cf6-465a-8645-e5672b198d53-kube-api-access-5hfhp\" (UniqueName: \"kubernetes.io/projected/aae93f38-7cf6-465a-8645-e5672b198d53-kube-api-access-5hfhp\") pod \"aae93f38-7cf6-465a-8645-e5672b198d53\" (UID: \"aae93f38-7cf6-465a-8645-e5672b198d53\") "
Nov 02 15:04:47.249837 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:04:47.249808    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aae93f38-7cf6-465a-8645-e5672b198d53-kube-api-access-5hfhp" pod "aae93f38-7cf6-465a-8645-e5672b198d53" (UID: "aae93f38-7cf6-465a-8645-e5672b198d53"). InnerVolumeSpecName "kube-api-access-5hfhp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:04:47.257701 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:04:47.257674    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aae93f38-7cf6-465a-8645-e5672b198d53-tls-cert" pod "aae93f38-7cf6-465a-8645-e5672b198d53" (UID: "aae93f38-7cf6-465a-8645-e5672b198d53"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:04:47.348334 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:04:47.348292    5304 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/aae93f38-7cf6-465a-8645-e5672b198d53-tls-cert\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:04:47.348511 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:04:47.348350    5304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5hfhp\" (UniqueName: \"kubernetes.io/projected/aae93f38-7cf6-465a-8645-e5672b198d53-kube-api-access-5hfhp\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:04:47.813976 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:04:47.813890    5304 generic.go:356] "Generic (PLEG): container finished" podID="aae93f38-7cf6-465a-8645-e5672b198d53" containerID="ea81ca00dec69b86596fe0adc276ddee9b9918274f7b398377cd2b7ed588b522" exitCode=0
Nov 02 15:04:47.813976 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:04:47.813953    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5c48987dbd-bk7v4"
Nov 02 15:04:47.814459 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:04:47.813973    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5c48987dbd-bk7v4" event={"ID":"aae93f38-7cf6-465a-8645-e5672b198d53","Type":"ContainerDied","Data":"ea81ca00dec69b86596fe0adc276ddee9b9918274f7b398377cd2b7ed588b522"}
Nov 02 15:04:47.814459 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:04:47.814019    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5c48987dbd-bk7v4" event={"ID":"aae93f38-7cf6-465a-8645-e5672b198d53","Type":"ContainerDied","Data":"d71f01efa850270e3e8a490350a62bc34a680664db02e1edb1c2217b28506d0e"}
Nov 02 15:04:47.814459 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:04:47.814043    5304 scope.go:122] "RemoveContainer" containerID="ea81ca00dec69b86596fe0adc276ddee9b9918274f7b398377cd2b7ed588b522"
Nov 02 15:04:47.828017 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:04:47.827991    5304 scope.go:122] "RemoveContainer" containerID="ea81ca00dec69b86596fe0adc276ddee9b9918274f7b398377cd2b7ed588b522"
Nov 02 15:04:47.828425 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:04:47.828398    5304 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea81ca00dec69b86596fe0adc276ddee9b9918274f7b398377cd2b7ed588b522\": container with ID starting with ea81ca00dec69b86596fe0adc276ddee9b9918274f7b398377cd2b7ed588b522 not found: ID does not exist" containerID="ea81ca00dec69b86596fe0adc276ddee9b9918274f7b398377cd2b7ed588b522"
Nov 02 15:04:47.828534 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:04:47.828438    5304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea81ca00dec69b86596fe0adc276ddee9b9918274f7b398377cd2b7ed588b522"} err="failed to get container status \"ea81ca00dec69b86596fe0adc276ddee9b9918274f7b398377cd2b7ed588b522\": rpc error: code = NotFound desc = could not find container \"ea81ca00dec69b86596fe0adc276ddee9b9918274f7b398377cd2b7ed588b522\": container with ID starting with ea81ca00dec69b86596fe0adc276ddee9b9918274f7b398377cd2b7ed588b522 not found: ID does not exist"
Nov 02 15:04:47.865376 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:04:47.865341    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5c48987dbd-bk7v4"]
Nov 02 15:04:47.873107 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:04:47.873076    5304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5c48987dbd-bk7v4"]
Nov 02 15:04:48.117546 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:04:48.117505    5304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aae93f38-7cf6-465a-8645-e5672b198d53" path="/var/lib/kubelet/pods/aae93f38-7cf6-465a-8645-e5672b198d53/volumes"
Nov 02 15:04:50.735776 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:04:50.735738    5304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-wxjl7"]
Nov 02 15:04:50.736276 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:04:50.736259    5304 cpu_manager.go:414] "containerMap: removing container" podUID="aae93f38-7cf6-465a-8645-e5672b198d53" containerName="authorino"
Nov 02 15:04:50.736337 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:04:50.736279    5304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aae93f38-7cf6-465a-8645-e5672b198d53" containerName="authorino"
Nov 02 15:04:50.736391 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:04:50.736376    5304 memory_manager.go:365] "RemoveStaleState removing state" podUID="aae93f38-7cf6-465a-8645-e5672b198d53" containerName="authorino"
Nov 02 15:04:50.742932 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:04:50.742911    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-55fbbbf9dc-wxjl7"
Nov 02 15:04:50.759302 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:04:50.757527    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-wxjl7"]
Nov 02 15:04:50.768891 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:04:50.768861    5304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-jzzvg\""
Nov 02 15:04:50.794213 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:04:50.794181    5304 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Nov 02 15:04:50.878678 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:04:50.878637    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f905a4e-b982-4d66-a7bc-171cda1c78dd-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-wxjl7\" (UID: \"2f905a4e-b982-4d66-a7bc-171cda1c78dd\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-wxjl7"
Nov 02 15:04:50.878856 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:04:50.878769    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8l85j\" (UniqueName: \"kubernetes.io/projected/2f905a4e-b982-4d66-a7bc-171cda1c78dd-kube-api-access-8l85j\") pod \"payload-processing-55fbbbf9dc-wxjl7\" (UID: \"2f905a4e-b982-4d66-a7bc-171cda1c78dd\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-wxjl7"
Nov 02 15:04:50.979246 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:04:50.979215    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8l85j\" (UniqueName: \"kubernetes.io/projected/2f905a4e-b982-4d66-a7bc-171cda1c78dd-kube-api-access-8l85j\") pod \"payload-processing-55fbbbf9dc-wxjl7\" (UID: \"2f905a4e-b982-4d66-a7bc-171cda1c78dd\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-wxjl7"
Nov 02 15:04:50.979421 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:04:50.979277    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f905a4e-b982-4d66-a7bc-171cda1c78dd-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-wxjl7\" (UID: \"2f905a4e-b982-4d66-a7bc-171cda1c78dd\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-wxjl7"
Nov 02 15:04:50.979965 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:04:50.979944    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f905a4e-b982-4d66-a7bc-171cda1c78dd-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-wxjl7\" (UID: \"2f905a4e-b982-4d66-a7bc-171cda1c78dd\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-wxjl7"
Nov 02 15:04:51.002940 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:04:51.002874    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8l85j\" (UniqueName: \"kubernetes.io/projected/2f905a4e-b982-4d66-a7bc-171cda1c78dd-kube-api-access-8l85j\") pod \"payload-processing-55fbbbf9dc-wxjl7\" (UID: \"2f905a4e-b982-4d66-a7bc-171cda1c78dd\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-wxjl7"
Nov 02 15:04:51.055755 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:04:51.055714    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-55fbbbf9dc-wxjl7"
Nov 02 15:04:51.089114 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:04:51.089078    5304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-tl8dm"]
Nov 02 15:04:51.094308 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:04:51.094279    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-tl8dm"
Nov 02 15:04:51.115682 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:04:51.115296    5304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-pwrm4\""
Nov 02 15:04:51.115682 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:04:51.115392    5304 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Nov 02 15:04:51.117885 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:04:51.117860    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-tl8dm"]
Nov 02 15:04:51.212152 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:04:51.211763    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a09b277-1d1f-45c3-a18a-d393acfa5882-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-tl8dm\" (UID: \"7a09b277-1d1f-45c3-a18a-d393acfa5882\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-tl8dm"
Nov 02 15:04:51.212152 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:04:51.211824    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a09b277-1d1f-45c3-a18a-d393acfa5882-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-tl8dm\" (UID: \"7a09b277-1d1f-45c3-a18a-d393acfa5882\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-tl8dm"
Nov 02 15:04:51.212152 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:04:51.211865    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a09b277-1d1f-45c3-a18a-d393acfa5882-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-tl8dm\" (UID: \"7a09b277-1d1f-45c3-a18a-d393acfa5882\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-tl8dm"
Nov 02 15:04:51.212152 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:04:51.211887    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a09b277-1d1f-45c3-a18a-d393acfa5882-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-tl8dm\" (UID: \"7a09b277-1d1f-45c3-a18a-d393acfa5882\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-tl8dm"
Nov 02 15:04:51.212152 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:04:51.211914    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a09b277-1d1f-45c3-a18a-d393acfa5882-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-tl8dm\" (UID: \"7a09b277-1d1f-45c3-a18a-d393acfa5882\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-tl8dm"
Nov 02 15:04:51.212152 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:04:51.211938    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jnx24\" (UniqueName: \"kubernetes.io/projected/7a09b277-1d1f-45c3-a18a-d393acfa5882-kube-api-access-jnx24\") pod \"maas-default-gateway-openshift-default-fcb75c45f-tl8dm\" (UID: \"7a09b277-1d1f-45c3-a18a-d393acfa5882\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-tl8dm"
Nov 02 15:04:51.212152 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:04:51.211982    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a09b277-1d1f-45c3-a18a-d393acfa5882-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-tl8dm\" (UID: \"7a09b277-1d1f-45c3-a18a-d393acfa5882\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-tl8dm"
Nov 02 15:04:51.212152 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:04:51.212018    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a09b277-1d1f-45c3-a18a-d393acfa5882-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-tl8dm\" (UID: \"7a09b277-1d1f-45c3-a18a-d393acfa5882\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-tl8dm"
Nov 02 15:04:51.212152 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:04:51.212079    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a09b277-1d1f-45c3-a18a-d393acfa5882-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-tl8dm\" (UID: \"7a09b277-1d1f-45c3-a18a-d393acfa5882\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-tl8dm"
Nov 02 15:04:51.312587 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:04:51.312501    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a09b277-1d1f-45c3-a18a-d393acfa5882-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-tl8dm\" (UID: \"7a09b277-1d1f-45c3-a18a-d393acfa5882\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-tl8dm"
Nov 02 15:04:51.312587 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:04:51.312547    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a09b277-1d1f-45c3-a18a-d393acfa5882-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-tl8dm\" (UID: \"7a09b277-1d1f-45c3-a18a-d393acfa5882\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-tl8dm"
Nov 02 15:04:51.312587 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:04:51.312573    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a09b277-1d1f-45c3-a18a-d393acfa5882-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-tl8dm\" (UID: \"7a09b277-1d1f-45c3-a18a-d393acfa5882\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-tl8dm"
Nov 02 15:04:51.312873 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:04:51.312600    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a09b277-1d1f-45c3-a18a-d393acfa5882-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-tl8dm\" (UID: \"7a09b277-1d1f-45c3-a18a-d393acfa5882\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-tl8dm"
Nov 02 15:04:51.312873 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:04:51.312621    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a09b277-1d1f-45c3-a18a-d393acfa5882-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-tl8dm\" (UID: \"7a09b277-1d1f-45c3-a18a-d393acfa5882\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-tl8dm"
Nov 02 15:04:51.312873 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:04:51.312653    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a09b277-1d1f-45c3-a18a-d393acfa5882-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-tl8dm\" (UID: \"7a09b277-1d1f-45c3-a18a-d393acfa5882\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-tl8dm"
Nov 02 15:04:51.312873 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:04:51.312675    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jnx24\" (UniqueName: \"kubernetes.io/projected/7a09b277-1d1f-45c3-a18a-d393acfa5882-kube-api-access-jnx24\") pod \"maas-default-gateway-openshift-default-fcb75c45f-tl8dm\" (UID: \"7a09b277-1d1f-45c3-a18a-d393acfa5882\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-tl8dm"
Nov 02 15:04:51.312873 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:04:51.312722    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a09b277-1d1f-45c3-a18a-d393acfa5882-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-tl8dm\" (UID: \"7a09b277-1d1f-45c3-a18a-d393acfa5882\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-tl8dm"
Nov 02 15:04:51.312873 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:04:51.312755    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a09b277-1d1f-45c3-a18a-d393acfa5882-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-tl8dm\" (UID: \"7a09b277-1d1f-45c3-a18a-d393acfa5882\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-tl8dm"
Nov 02 15:04:51.313184 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:04:51.313036    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a09b277-1d1f-45c3-a18a-d393acfa5882-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-tl8dm\" (UID: \"7a09b277-1d1f-45c3-a18a-d393acfa5882\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-tl8dm"
Nov 02 15:04:51.313184 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:04:51.313095    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a09b277-1d1f-45c3-a18a-d393acfa5882-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-tl8dm\" (UID: \"7a09b277-1d1f-45c3-a18a-d393acfa5882\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-tl8dm"
Nov 02 15:04:51.313386 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:04:51.313367    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a09b277-1d1f-45c3-a18a-d393acfa5882-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-tl8dm\" (UID: \"7a09b277-1d1f-45c3-a18a-d393acfa5882\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-tl8dm"
Nov 02 15:04:51.313656 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:04:51.313621    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a09b277-1d1f-45c3-a18a-d393acfa5882-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-tl8dm\" (UID: \"7a09b277-1d1f-45c3-a18a-d393acfa5882\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-tl8dm"
Nov 02 15:04:51.313757 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:04:51.313730    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a09b277-1d1f-45c3-a18a-d393acfa5882-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-tl8dm\" (UID: \"7a09b277-1d1f-45c3-a18a-d393acfa5882\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-tl8dm"
Nov 02 15:04:51.315347 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:04:51.315318    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a09b277-1d1f-45c3-a18a-d393acfa5882-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-tl8dm\" (UID: \"7a09b277-1d1f-45c3-a18a-d393acfa5882\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-tl8dm"
Nov 02 15:04:51.315447 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:04:51.315321    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a09b277-1d1f-45c3-a18a-d393acfa5882-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-tl8dm\" (UID: \"7a09b277-1d1f-45c3-a18a-d393acfa5882\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-tl8dm"
Nov 02 15:04:51.343639 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:04:51.343610    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jnx24\" (UniqueName: \"kubernetes.io/projected/7a09b277-1d1f-45c3-a18a-d393acfa5882-kube-api-access-jnx24\") pod \"maas-default-gateway-openshift-default-fcb75c45f-tl8dm\" (UID: \"7a09b277-1d1f-45c3-a18a-d393acfa5882\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-tl8dm"
Nov 02 15:04:51.343785 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:04:51.343678    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a09b277-1d1f-45c3-a18a-d393acfa5882-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-tl8dm\" (UID: \"7a09b277-1d1f-45c3-a18a-d393acfa5882\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-tl8dm"
Nov 02 15:04:51.365592 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:04:51.365562    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-wxjl7"]
Nov 02 15:04:51.370587 ip-10-0-135-138 kubenswrapper[5304]: W1102 15:04:51.370556    5304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f905a4e_b982_4d66_a7bc_171cda1c78dd.slice/crio-df7047d21444022acf9d8f1f35cb300cbf57a41164b856a384c715d6edb645d6 WatchSource:0}: Error finding container df7047d21444022acf9d8f1f35cb300cbf57a41164b856a384c715d6edb645d6: Status 404 returned error can't find the container with id df7047d21444022acf9d8f1f35cb300cbf57a41164b856a384c715d6edb645d6
Nov 02 15:04:51.418455 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:04:51.418422    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-tl8dm"
Nov 02 15:04:51.734955 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:04:51.734917    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-tl8dm"]
Nov 02 15:04:51.739672 ip-10-0-135-138 kubenswrapper[5304]: W1102 15:04:51.739645    5304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a09b277_1d1f_45c3_a18a_d393acfa5882.slice/crio-306e799f8eddb68e1c13320353fa91e151833b88cb1ebd54a9429b0f21d01558 WatchSource:0}: Error finding container 306e799f8eddb68e1c13320353fa91e151833b88cb1ebd54a9429b0f21d01558: Status 404 returned error can't find the container with id 306e799f8eddb68e1c13320353fa91e151833b88cb1ebd54a9429b0f21d01558
Nov 02 15:04:51.835098 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:04:51.835058    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-55fbbbf9dc-wxjl7" event={"ID":"2f905a4e-b982-4d66-a7bc-171cda1c78dd","Type":"ContainerStarted","Data":"df7047d21444022acf9d8f1f35cb300cbf57a41164b856a384c715d6edb645d6"}
Nov 02 15:04:51.836519 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:04:51.836488    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-tl8dm" event={"ID":"7a09b277-1d1f-45c3-a18a-d393acfa5882","Type":"ContainerStarted","Data":"306e799f8eddb68e1c13320353fa91e151833b88cb1ebd54a9429b0f21d01558"}
Nov 02 15:04:53.848911 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:04:53.848872    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-55fbbbf9dc-wxjl7" event={"ID":"2f905a4e-b982-4d66-a7bc-171cda1c78dd","Type":"ContainerStarted","Data":"6d59ae5b6de6db045924c01d589c99fcfb5fc6bdd223bc979793742600cc94b6"}
Nov 02 15:04:53.849425 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:04:53.848963    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-55fbbbf9dc-wxjl7"
Nov 02 15:04:53.889853 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:04:53.889796    5304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-55fbbbf9dc-wxjl7" podStartSLOduration=1.8751640269999998 podStartE2EDuration="3.889778615s" totalImagesPullingTime="2.014614588s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:04:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:04:53.889408187 +0000 UTC m=+493.883232597" watchObservedRunningTime="2026-11-02 15:04:53.889778615 +0000 UTC m=+493.883603026"
Nov 02 15:04:54.801886 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:04:54.801847    5304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k6fjk" containerName="extractor"
Nov 02 15:04:54.951387 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:04:54.951348    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:04:54.951718 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:04:54.951447    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:04:54.951718 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:04:54.951488    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:04:55.860948 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:04:55.860904    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-tl8dm" event={"ID":"7a09b277-1d1f-45c3-a18a-d393acfa5882","Type":"ContainerStarted","Data":"1e9030e051ac48ab0cc6e29ac000907223c70f491d6fdcb0e7bcf3c49454816d"}
Nov 02 15:04:55.918777 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:04:55.918730    5304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-tl8dm" podStartSLOduration=1.710658612 podStartE2EDuration="4.918715741s" totalImagesPullingTime="3.208057129s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:04:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:04:55.917344826 +0000 UTC m=+495.911169235" watchObservedRunningTime="2026-11-02 15:04:55.918715741 +0000 UTC m=+495.912540150"
Nov 02 15:04:56.419438 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:04:56.419398    5304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-tl8dm"
Nov 02 15:04:56.420852 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:04:56.420829    5304 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-tl8dm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" start-of-body=
Nov 02 15:04:56.420907 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:04:56.420875    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-tl8dm" podUID="7a09b277-1d1f-45c3-a18a-d393acfa5882" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused"
Nov 02 15:04:57.419137 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:04:57.419082    5304 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-tl8dm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" start-of-body=
Nov 02 15:04:57.419292 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:04:57.419175    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-tl8dm" podUID="7a09b277-1d1f-45c3-a18a-d393acfa5882" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused"
Nov 02 15:04:58.419315 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:04:58.419278    5304 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-tl8dm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" start-of-body=
Nov 02 15:04:58.419726 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:04:58.419343    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-tl8dm" podUID="7a09b277-1d1f-45c3-a18a-d393acfa5882" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused"
Nov 02 15:04:59.419195 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:04:59.419157    5304 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-tl8dm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" start-of-body=
Nov 02 15:04:59.419411 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:04:59.419235    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-tl8dm" podUID="7a09b277-1d1f-45c3-a18a-d393acfa5882" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused"
Nov 02 15:05:00.172541 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:00.172503    5304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-6d8bf9c9ff-r77kz"]
Nov 02 15:05:00.175251 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:00.175228    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-6d8bf9c9ff-r77kz"
Nov 02 15:05:00.195905 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:00.195879    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0f933e27-7304-44d5-acc5-8089829b1b90-metrics-certs\") pod \"ai-gateway-controller-6d8bf9c9ff-r77kz\" (UID: \"0f933e27-7304-44d5-acc5-8089829b1b90\") " pod="opendatahub/ai-gateway-controller-6d8bf9c9ff-r77kz"
Nov 02 15:05:00.196043 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:00.195922    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fp96f\" (UniqueName: \"kubernetes.io/projected/0f933e27-7304-44d5-acc5-8089829b1b90-kube-api-access-fp96f\") pod \"ai-gateway-controller-6d8bf9c9ff-r77kz\" (UID: \"0f933e27-7304-44d5-acc5-8089829b1b90\") " pod="opendatahub/ai-gateway-controller-6d8bf9c9ff-r77kz"
Nov 02 15:05:00.202115 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:00.202087    5304 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"ai-gateway-controller-parameters\""
Nov 02 15:05:00.202257 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:00.202175    5304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-dockercfg-z7mgl\""
Nov 02 15:05:00.202257 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:00.202229    5304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-metrics-tls\""
Nov 02 15:05:00.203678 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:00.203654    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-6d8bf9c9ff-r77kz"]
Nov 02 15:05:00.296375 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:00.296337    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0f933e27-7304-44d5-acc5-8089829b1b90-metrics-certs\") pod \"ai-gateway-controller-6d8bf9c9ff-r77kz\" (UID: \"0f933e27-7304-44d5-acc5-8089829b1b90\") " pod="opendatahub/ai-gateway-controller-6d8bf9c9ff-r77kz"
Nov 02 15:05:00.296560 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:00.296396    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fp96f\" (UniqueName: \"kubernetes.io/projected/0f933e27-7304-44d5-acc5-8089829b1b90-kube-api-access-fp96f\") pod \"ai-gateway-controller-6d8bf9c9ff-r77kz\" (UID: \"0f933e27-7304-44d5-acc5-8089829b1b90\") " pod="opendatahub/ai-gateway-controller-6d8bf9c9ff-r77kz"
Nov 02 15:05:00.299296 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:00.299261    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0f933e27-7304-44d5-acc5-8089829b1b90-metrics-certs\") pod \"ai-gateway-controller-6d8bf9c9ff-r77kz\" (UID: \"0f933e27-7304-44d5-acc5-8089829b1b90\") " pod="opendatahub/ai-gateway-controller-6d8bf9c9ff-r77kz"
Nov 02 15:05:00.319827 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:00.319801    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fp96f\" (UniqueName: \"kubernetes.io/projected/0f933e27-7304-44d5-acc5-8089829b1b90-kube-api-access-fp96f\") pod \"ai-gateway-controller-6d8bf9c9ff-r77kz\" (UID: \"0f933e27-7304-44d5-acc5-8089829b1b90\") " pod="opendatahub/ai-gateway-controller-6d8bf9c9ff-r77kz"
Nov 02 15:05:00.419241 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:00.419207    5304 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-tl8dm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" start-of-body=
Nov 02 15:05:00.419406 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:00.419260    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-tl8dm" podUID="7a09b277-1d1f-45c3-a18a-d393acfa5882" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused"
Nov 02 15:05:00.486651 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:00.486565    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-6d8bf9c9ff-r77kz"
Nov 02 15:05:00.802699 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:00.802622    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-6d8bf9c9ff-r77kz"]
Nov 02 15:05:00.808283 ip-10-0-135-138 kubenswrapper[5304]: W1102 15:05:00.808248    5304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f933e27_7304_44d5_acc5_8089829b1b90.slice/crio-b9bac8e34e17be105ced1e003a009582965b2af2c32ddb34611960dd0a18d30c WatchSource:0}: Error finding container b9bac8e34e17be105ced1e003a009582965b2af2c32ddb34611960dd0a18d30c: Status 404 returned error can't find the container with id b9bac8e34e17be105ced1e003a009582965b2af2c32ddb34611960dd0a18d30c
Nov 02 15:05:00.886780 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:00.886740    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-6d8bf9c9ff-r77kz" event={"ID":"0f933e27-7304-44d5-acc5-8089829b1b90","Type":"ContainerStarted","Data":"b9bac8e34e17be105ced1e003a009582965b2af2c32ddb34611960dd0a18d30c"}
Nov 02 15:05:01.419775 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:01.419733    5304 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-tl8dm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" start-of-body=
Nov 02 15:05:01.419987 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:01.419797    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-tl8dm" podUID="7a09b277-1d1f-45c3-a18a-d393acfa5882" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused"
Nov 02 15:05:02.418943 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:02.418902    5304 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-tl8dm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" start-of-body=
Nov 02 15:05:02.419394 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:02.418967    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-tl8dm" podUID="7a09b277-1d1f-45c3-a18a-d393acfa5882" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused"
Nov 02 15:05:03.538566 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:03.538471    5304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-tl8dm"
Nov 02 15:05:03.539232 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:03.539194    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-tl8dm"
Nov 02 15:05:03.540211 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:03.540183    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-tl8dm"
Nov 02 15:05:03.902810 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:03.902769    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-6d8bf9c9ff-r77kz" event={"ID":"0f933e27-7304-44d5-acc5-8089829b1b90","Type":"ContainerStarted","Data":"ecbb6cbbaa8c0f570c72fec296b0e64ca22217624669c19f5e07d06d157aa863"}
Nov 02 15:05:03.903038 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:03.903018    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-6d8bf9c9ff-r77kz"
Nov 02 15:05:03.944591 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:03.944542    5304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-6d8bf9c9ff-r77kz" podStartSLOduration=1.627334866 podStartE2EDuration="3.944529614s" totalImagesPullingTime="2.317194748s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:05:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:05:03.944090346 +0000 UTC m=+503.937914754" watchObservedRunningTime="2026-11-02 15:05:03.944529614 +0000 UTC m=+503.938354022"
Nov 02 15:05:04.856856 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:04.856817    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-55fbbbf9dc-wxjl7"
Nov 02 15:05:06.134435 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:06.134401    5304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-r7jjp"]
Nov 02 15:05:06.137636 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:06.137613    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-r7jjp"
Nov 02 15:05:06.165563 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:06.165528    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-r7jjp"]
Nov 02 15:05:06.247296 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:06.247264    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/707b5eda-0575-4b3d-b1d8-f18005ddaa07-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-r7jjp\" (UID: \"707b5eda-0575-4b3d-b1d8-f18005ddaa07\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-r7jjp"
Nov 02 15:05:06.247462 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:06.247299    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/707b5eda-0575-4b3d-b1d8-f18005ddaa07-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-r7jjp\" (UID: \"707b5eda-0575-4b3d-b1d8-f18005ddaa07\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-r7jjp"
Nov 02 15:05:06.247462 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:06.247322    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/707b5eda-0575-4b3d-b1d8-f18005ddaa07-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-r7jjp\" (UID: \"707b5eda-0575-4b3d-b1d8-f18005ddaa07\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-r7jjp"
Nov 02 15:05:06.247462 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:06.247376    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/707b5eda-0575-4b3d-b1d8-f18005ddaa07-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-r7jjp\" (UID: \"707b5eda-0575-4b3d-b1d8-f18005ddaa07\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-r7jjp"
Nov 02 15:05:06.247567 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:06.247465    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/707b5eda-0575-4b3d-b1d8-f18005ddaa07-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-r7jjp\" (UID: \"707b5eda-0575-4b3d-b1d8-f18005ddaa07\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-r7jjp"
Nov 02 15:05:06.247567 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:06.247496    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rmtml\" (UniqueName: \"kubernetes.io/projected/707b5eda-0575-4b3d-b1d8-f18005ddaa07-kube-api-access-rmtml\") pod \"maas-default-gateway-openshift-default-fcb75c45f-r7jjp\" (UID: \"707b5eda-0575-4b3d-b1d8-f18005ddaa07\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-r7jjp"
Nov 02 15:05:06.247627 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:06.247571    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/707b5eda-0575-4b3d-b1d8-f18005ddaa07-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-r7jjp\" (UID: \"707b5eda-0575-4b3d-b1d8-f18005ddaa07\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-r7jjp"
Nov 02 15:05:06.247627 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:06.247599    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/707b5eda-0575-4b3d-b1d8-f18005ddaa07-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-r7jjp\" (UID: \"707b5eda-0575-4b3d-b1d8-f18005ddaa07\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-r7jjp"
Nov 02 15:05:06.247690 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:06.247626    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/707b5eda-0575-4b3d-b1d8-f18005ddaa07-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-r7jjp\" (UID: \"707b5eda-0575-4b3d-b1d8-f18005ddaa07\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-r7jjp"
Nov 02 15:05:06.348782 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:06.348750    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/707b5eda-0575-4b3d-b1d8-f18005ddaa07-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-r7jjp\" (UID: \"707b5eda-0575-4b3d-b1d8-f18005ddaa07\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-r7jjp"
Nov 02 15:05:06.348928 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:06.348796    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/707b5eda-0575-4b3d-b1d8-f18005ddaa07-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-r7jjp\" (UID: \"707b5eda-0575-4b3d-b1d8-f18005ddaa07\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-r7jjp"
Nov 02 15:05:06.348928 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:06.348824    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/707b5eda-0575-4b3d-b1d8-f18005ddaa07-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-r7jjp\" (UID: \"707b5eda-0575-4b3d-b1d8-f18005ddaa07\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-r7jjp"
Nov 02 15:05:06.348928 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:06.348843    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/707b5eda-0575-4b3d-b1d8-f18005ddaa07-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-r7jjp\" (UID: \"707b5eda-0575-4b3d-b1d8-f18005ddaa07\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-r7jjp"
Nov 02 15:05:06.348928 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:06.348859    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/707b5eda-0575-4b3d-b1d8-f18005ddaa07-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-r7jjp\" (UID: \"707b5eda-0575-4b3d-b1d8-f18005ddaa07\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-r7jjp"
Nov 02 15:05:06.348928 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:06.348874    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/707b5eda-0575-4b3d-b1d8-f18005ddaa07-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-r7jjp\" (UID: \"707b5eda-0575-4b3d-b1d8-f18005ddaa07\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-r7jjp"
Nov 02 15:05:06.349196 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:06.348964    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/707b5eda-0575-4b3d-b1d8-f18005ddaa07-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-r7jjp\" (UID: \"707b5eda-0575-4b3d-b1d8-f18005ddaa07\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-r7jjp"
Nov 02 15:05:06.349196 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:06.349017    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rmtml\" (UniqueName: \"kubernetes.io/projected/707b5eda-0575-4b3d-b1d8-f18005ddaa07-kube-api-access-rmtml\") pod \"maas-default-gateway-openshift-default-fcb75c45f-r7jjp\" (UID: \"707b5eda-0575-4b3d-b1d8-f18005ddaa07\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-r7jjp"
Nov 02 15:05:06.349196 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:06.349080    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/707b5eda-0575-4b3d-b1d8-f18005ddaa07-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-r7jjp\" (UID: \"707b5eda-0575-4b3d-b1d8-f18005ddaa07\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-r7jjp"
Nov 02 15:05:06.349362 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:06.349249    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/707b5eda-0575-4b3d-b1d8-f18005ddaa07-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-r7jjp\" (UID: \"707b5eda-0575-4b3d-b1d8-f18005ddaa07\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-r7jjp"
Nov 02 15:05:06.349416 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:06.349376    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/707b5eda-0575-4b3d-b1d8-f18005ddaa07-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-r7jjp\" (UID: \"707b5eda-0575-4b3d-b1d8-f18005ddaa07\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-r7jjp"
Nov 02 15:05:06.349501 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:06.349456    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/707b5eda-0575-4b3d-b1d8-f18005ddaa07-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-r7jjp\" (UID: \"707b5eda-0575-4b3d-b1d8-f18005ddaa07\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-r7jjp"
Nov 02 15:05:06.349627 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:06.349474    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/707b5eda-0575-4b3d-b1d8-f18005ddaa07-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-r7jjp\" (UID: \"707b5eda-0575-4b3d-b1d8-f18005ddaa07\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-r7jjp"
Nov 02 15:05:06.349801 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:06.349781    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/707b5eda-0575-4b3d-b1d8-f18005ddaa07-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-r7jjp\" (UID: \"707b5eda-0575-4b3d-b1d8-f18005ddaa07\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-r7jjp"
Nov 02 15:05:06.351258 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:06.351234    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/707b5eda-0575-4b3d-b1d8-f18005ddaa07-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-r7jjp\" (UID: \"707b5eda-0575-4b3d-b1d8-f18005ddaa07\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-r7jjp"
Nov 02 15:05:06.351394 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:06.351379    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/707b5eda-0575-4b3d-b1d8-f18005ddaa07-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-r7jjp\" (UID: \"707b5eda-0575-4b3d-b1d8-f18005ddaa07\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-r7jjp"
Nov 02 15:05:06.372845 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:06.372822    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/707b5eda-0575-4b3d-b1d8-f18005ddaa07-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-r7jjp\" (UID: \"707b5eda-0575-4b3d-b1d8-f18005ddaa07\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-r7jjp"
Nov 02 15:05:06.372986 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:06.372968    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rmtml\" (UniqueName: \"kubernetes.io/projected/707b5eda-0575-4b3d-b1d8-f18005ddaa07-kube-api-access-rmtml\") pod \"maas-default-gateway-openshift-default-fcb75c45f-r7jjp\" (UID: \"707b5eda-0575-4b3d-b1d8-f18005ddaa07\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-r7jjp"
Nov 02 15:05:06.449907 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:06.449820    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-r7jjp"
Nov 02 15:05:06.760624 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:06.760539    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-r7jjp"]
Nov 02 15:05:06.766181 ip-10-0-135-138 kubenswrapper[5304]: W1102 15:05:06.766150    5304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod707b5eda_0575_4b3d_b1d8_f18005ddaa07.slice/crio-2abd8aaf6940c452481162ce3c00b147c8d0869a6025a9f276a5bef035d47354 WatchSource:0}: Error finding container 2abd8aaf6940c452481162ce3c00b147c8d0869a6025a9f276a5bef035d47354: Status 404 returned error can't find the container with id 2abd8aaf6940c452481162ce3c00b147c8d0869a6025a9f276a5bef035d47354
Nov 02 15:05:06.768648 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:06.768613    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:05:06.768730 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:06.768684    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:05:06.768730 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:06.768710    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:05:06.917212 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:06.917178    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-r7jjp" event={"ID":"707b5eda-0575-4b3d-b1d8-f18005ddaa07","Type":"ContainerStarted","Data":"a5494422ed014731629f8914216f2679ff9aa7d7c0532e007350dab4d603eb5c"}
Nov 02 15:05:06.917375 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:06.917235    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-r7jjp" event={"ID":"707b5eda-0575-4b3d-b1d8-f18005ddaa07","Type":"ContainerStarted","Data":"2abd8aaf6940c452481162ce3c00b147c8d0869a6025a9f276a5bef035d47354"}
Nov 02 15:05:06.963920 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:06.963868    5304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-r7jjp" podStartSLOduration=0.963850967 podStartE2EDuration="963.850967ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:05:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:05:06.962940387 +0000 UTC m=+506.956764796" watchObservedRunningTime="2026-11-02 15:05:06.963850967 +0000 UTC m=+506.957675377"
Nov 02 15:05:07.454161 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:07.452503    5304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-r7jjp"
Nov 02 15:05:07.456353 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:07.456311    5304 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-r7jjp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" start-of-body=
Nov 02 15:05:07.456505 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:07.456423    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-r7jjp" podUID="707b5eda-0575-4b3d-b1d8-f18005ddaa07" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused"
Nov 02 15:05:08.453150 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:08.452180    5304 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-r7jjp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" start-of-body=
Nov 02 15:05:08.453150 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:08.452260    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-r7jjp" podUID="707b5eda-0575-4b3d-b1d8-f18005ddaa07" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused"
Nov 02 15:05:09.450607 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:09.450566    5304 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-r7jjp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" start-of-body=
Nov 02 15:05:09.451237 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:09.451202    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-r7jjp" podUID="707b5eda-0575-4b3d-b1d8-f18005ddaa07" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused"
Nov 02 15:05:10.455606 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:10.455566    5304 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-r7jjp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" start-of-body=
Nov 02 15:05:10.456035 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:10.455625    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-r7jjp" podUID="707b5eda-0575-4b3d-b1d8-f18005ddaa07" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused"
Nov 02 15:05:11.451150 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:11.450529    5304 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-r7jjp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" start-of-body=
Nov 02 15:05:11.451150 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:11.450591    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-r7jjp" podUID="707b5eda-0575-4b3d-b1d8-f18005ddaa07" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused"
Nov 02 15:05:12.454614 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:12.454574    5304 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-r7jjp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" start-of-body=
Nov 02 15:05:12.455251 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:12.455212    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-r7jjp" podUID="707b5eda-0575-4b3d-b1d8-f18005ddaa07" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused"
Nov 02 15:05:13.450884 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:13.450841    5304 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-r7jjp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" start-of-body=
Nov 02 15:05:13.451065 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:13.450910    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-r7jjp" podUID="707b5eda-0575-4b3d-b1d8-f18005ddaa07" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused"
Nov 02 15:05:14.910692 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:14.910659    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-6d8bf9c9ff-r77kz"
Nov 02 15:05:14.968403 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:14.968334    5304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-r7jjp"
Nov 02 15:05:14.968787 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:14.968759    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-r7jjp"
Nov 02 15:05:14.969554 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:14.969532    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-r7jjp"
Nov 02 15:05:24.797152 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:05:24.797098    5304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k6fjk" containerName="extractor"
Nov 02 15:05:52.741779 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:52.741687    5304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-25tqv"]
Nov 02 15:05:52.744991 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:52.744966    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-25tqv"
Nov 02 15:05:52.760805 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:52.760775    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ed68512-6850-45d4-af0d-a941c2bf6ec9-istio-data\") pod \"e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-25tqv\" (UID: \"3ed68512-6850-45d4-af0d-a941c2bf6ec9\") " pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-25tqv"
Nov 02 15:05:52.760967 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:52.760817    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ed68512-6850-45d4-af0d-a941c2bf6ec9-credential-socket\") pod \"e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-25tqv\" (UID: \"3ed68512-6850-45d4-af0d-a941c2bf6ec9\") " pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-25tqv"
Nov 02 15:05:52.760967 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:52.760844    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ed68512-6850-45d4-af0d-a941c2bf6ec9-istio-podinfo\") pod \"e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-25tqv\" (UID: \"3ed68512-6850-45d4-af0d-a941c2bf6ec9\") " pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-25tqv"
Nov 02 15:05:52.760967 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:52.760932    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ed68512-6850-45d4-af0d-a941c2bf6ec9-istiod-ca-cert\") pod \"e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-25tqv\" (UID: \"3ed68512-6850-45d4-af0d-a941c2bf6ec9\") " pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-25tqv"
Nov 02 15:05:52.761155 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:52.760975    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ed68512-6850-45d4-af0d-a941c2bf6ec9-workload-certs\") pod \"e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-25tqv\" (UID: \"3ed68512-6850-45d4-af0d-a941c2bf6ec9\") " pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-25tqv"
Nov 02 15:05:52.761155 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:52.760991    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ed68512-6850-45d4-af0d-a941c2bf6ec9-istio-envoy\") pod \"e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-25tqv\" (UID: \"3ed68512-6850-45d4-af0d-a941c2bf6ec9\") " pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-25tqv"
Nov 02 15:05:52.761155 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:52.761043    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ed68512-6850-45d4-af0d-a941c2bf6ec9-workload-socket\") pod \"e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-25tqv\" (UID: \"3ed68512-6850-45d4-af0d-a941c2bf6ec9\") " pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-25tqv"
Nov 02 15:05:52.761155 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:52.761068    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rwf6k\" (UniqueName: \"kubernetes.io/projected/3ed68512-6850-45d4-af0d-a941c2bf6ec9-kube-api-access-rwf6k\") pod \"e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-25tqv\" (UID: \"3ed68512-6850-45d4-af0d-a941c2bf6ec9\") " pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-25tqv"
Nov 02 15:05:52.761356 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:52.761199    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ed68512-6850-45d4-af0d-a941c2bf6ec9-istio-token\") pod \"e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-25tqv\" (UID: \"3ed68512-6850-45d4-af0d-a941c2bf6ec9\") " pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-25tqv"
Nov 02 15:05:52.764163 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:52.764121    5304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-7ecfe1-openshift-default-dockercfg-gg2h7\""
Nov 02 15:05:52.787365 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:52.787328    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-25tqv"]
Nov 02 15:05:52.861571 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:52.861535    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ed68512-6850-45d4-af0d-a941c2bf6ec9-credential-socket\") pod \"e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-25tqv\" (UID: \"3ed68512-6850-45d4-af0d-a941c2bf6ec9\") " pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-25tqv"
Nov 02 15:05:52.861571 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:52.861574    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ed68512-6850-45d4-af0d-a941c2bf6ec9-istio-podinfo\") pod \"e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-25tqv\" (UID: \"3ed68512-6850-45d4-af0d-a941c2bf6ec9\") " pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-25tqv"
Nov 02 15:05:52.861831 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:52.861595    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ed68512-6850-45d4-af0d-a941c2bf6ec9-istiod-ca-cert\") pod \"e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-25tqv\" (UID: \"3ed68512-6850-45d4-af0d-a941c2bf6ec9\") " pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-25tqv"
Nov 02 15:05:52.861831 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:52.861652    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ed68512-6850-45d4-af0d-a941c2bf6ec9-workload-certs\") pod \"e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-25tqv\" (UID: \"3ed68512-6850-45d4-af0d-a941c2bf6ec9\") " pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-25tqv"
Nov 02 15:05:52.861831 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:52.861690    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ed68512-6850-45d4-af0d-a941c2bf6ec9-istio-envoy\") pod \"e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-25tqv\" (UID: \"3ed68512-6850-45d4-af0d-a941c2bf6ec9\") " pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-25tqv"
Nov 02 15:05:52.861831 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:52.861732    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ed68512-6850-45d4-af0d-a941c2bf6ec9-workload-socket\") pod \"e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-25tqv\" (UID: \"3ed68512-6850-45d4-af0d-a941c2bf6ec9\") " pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-25tqv"
Nov 02 15:05:52.861831 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:52.861756    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rwf6k\" (UniqueName: \"kubernetes.io/projected/3ed68512-6850-45d4-af0d-a941c2bf6ec9-kube-api-access-rwf6k\") pod \"e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-25tqv\" (UID: \"3ed68512-6850-45d4-af0d-a941c2bf6ec9\") " pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-25tqv"
Nov 02 15:05:52.861831 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:52.861799    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ed68512-6850-45d4-af0d-a941c2bf6ec9-istio-token\") pod \"e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-25tqv\" (UID: \"3ed68512-6850-45d4-af0d-a941c2bf6ec9\") " pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-25tqv"
Nov 02 15:05:52.862170 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:52.861880    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ed68512-6850-45d4-af0d-a941c2bf6ec9-istio-data\") pod \"e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-25tqv\" (UID: \"3ed68512-6850-45d4-af0d-a941c2bf6ec9\") " pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-25tqv"
Nov 02 15:05:52.862170 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:52.862042    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ed68512-6850-45d4-af0d-a941c2bf6ec9-credential-socket\") pod \"e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-25tqv\" (UID: \"3ed68512-6850-45d4-af0d-a941c2bf6ec9\") " pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-25tqv"
Nov 02 15:05:52.862170 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:52.862106    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ed68512-6850-45d4-af0d-a941c2bf6ec9-workload-certs\") pod \"e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-25tqv\" (UID: \"3ed68512-6850-45d4-af0d-a941c2bf6ec9\") " pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-25tqv"
Nov 02 15:05:52.862347 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:52.862302    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ed68512-6850-45d4-af0d-a941c2bf6ec9-workload-socket\") pod \"e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-25tqv\" (UID: \"3ed68512-6850-45d4-af0d-a941c2bf6ec9\") " pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-25tqv"
Nov 02 15:05:52.862395 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:52.862377    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ed68512-6850-45d4-af0d-a941c2bf6ec9-istio-data\") pod \"e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-25tqv\" (UID: \"3ed68512-6850-45d4-af0d-a941c2bf6ec9\") " pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-25tqv"
Nov 02 15:05:52.862480 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:52.862459    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ed68512-6850-45d4-af0d-a941c2bf6ec9-istiod-ca-cert\") pod \"e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-25tqv\" (UID: \"3ed68512-6850-45d4-af0d-a941c2bf6ec9\") " pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-25tqv"
Nov 02 15:05:52.864087 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:52.864060    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ed68512-6850-45d4-af0d-a941c2bf6ec9-istio-envoy\") pod \"e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-25tqv\" (UID: \"3ed68512-6850-45d4-af0d-a941c2bf6ec9\") " pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-25tqv"
Nov 02 15:05:52.864212 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:52.864198    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ed68512-6850-45d4-af0d-a941c2bf6ec9-istio-podinfo\") pod \"e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-25tqv\" (UID: \"3ed68512-6850-45d4-af0d-a941c2bf6ec9\") " pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-25tqv"
Nov 02 15:05:52.885702 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:52.885667    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rwf6k\" (UniqueName: \"kubernetes.io/projected/3ed68512-6850-45d4-af0d-a941c2bf6ec9-kube-api-access-rwf6k\") pod \"e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-25tqv\" (UID: \"3ed68512-6850-45d4-af0d-a941c2bf6ec9\") " pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-25tqv"
Nov 02 15:05:52.889290 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:52.889272    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ed68512-6850-45d4-af0d-a941c2bf6ec9-istio-token\") pod \"e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-25tqv\" (UID: \"3ed68512-6850-45d4-af0d-a941c2bf6ec9\") " pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-25tqv"
Nov 02 15:05:52.931094 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:52.931064    5304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-cp5ms"]
Nov 02 15:05:52.934035 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:52.934013    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-cp5ms"
Nov 02 15:05:52.949739 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:52.949716    5304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-624254-openshift-default-dockercfg-c82tf\""
Nov 02 15:05:52.956206 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:52.956181    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-cp5ms"]
Nov 02 15:05:52.962211 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:52.962184    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11a65ba1-0a55-4478-b043-637c0a9d6d64-istio-envoy\") pod \"e2e-worker-w3-624254-openshift-default-7988c9d8b5-cp5ms\" (UID: \"11a65ba1-0a55-4478-b043-637c0a9d6d64\") " pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-cp5ms"
Nov 02 15:05:52.962211 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:52.962212    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11a65ba1-0a55-4478-b043-637c0a9d6d64-istiod-ca-cert\") pod \"e2e-worker-w3-624254-openshift-default-7988c9d8b5-cp5ms\" (UID: \"11a65ba1-0a55-4478-b043-637c0a9d6d64\") " pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-cp5ms"
Nov 02 15:05:52.962415 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:52.962246    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wskwp\" (UniqueName: \"kubernetes.io/projected/11a65ba1-0a55-4478-b043-637c0a9d6d64-kube-api-access-wskwp\") pod \"e2e-worker-w3-624254-openshift-default-7988c9d8b5-cp5ms\" (UID: \"11a65ba1-0a55-4478-b043-637c0a9d6d64\") " pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-cp5ms"
Nov 02 15:05:52.962415 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:52.962346    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11a65ba1-0a55-4478-b043-637c0a9d6d64-workload-certs\") pod \"e2e-worker-w3-624254-openshift-default-7988c9d8b5-cp5ms\" (UID: \"11a65ba1-0a55-4478-b043-637c0a9d6d64\") " pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-cp5ms"
Nov 02 15:05:52.962415 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:52.962382    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11a65ba1-0a55-4478-b043-637c0a9d6d64-workload-socket\") pod \"e2e-worker-w3-624254-openshift-default-7988c9d8b5-cp5ms\" (UID: \"11a65ba1-0a55-4478-b043-637c0a9d6d64\") " pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-cp5ms"
Nov 02 15:05:52.962415 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:52.962399    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11a65ba1-0a55-4478-b043-637c0a9d6d64-credential-socket\") pod \"e2e-worker-w3-624254-openshift-default-7988c9d8b5-cp5ms\" (UID: \"11a65ba1-0a55-4478-b043-637c0a9d6d64\") " pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-cp5ms"
Nov 02 15:05:52.962622 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:52.962447    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11a65ba1-0a55-4478-b043-637c0a9d6d64-istio-podinfo\") pod \"e2e-worker-w3-624254-openshift-default-7988c9d8b5-cp5ms\" (UID: \"11a65ba1-0a55-4478-b043-637c0a9d6d64\") " pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-cp5ms"
Nov 02 15:05:52.962622 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:52.962471    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11a65ba1-0a55-4478-b043-637c0a9d6d64-istio-token\") pod \"e2e-worker-w3-624254-openshift-default-7988c9d8b5-cp5ms\" (UID: \"11a65ba1-0a55-4478-b043-637c0a9d6d64\") " pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-cp5ms"
Nov 02 15:05:52.962622 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:52.962533    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11a65ba1-0a55-4478-b043-637c0a9d6d64-istio-data\") pod \"e2e-worker-w3-624254-openshift-default-7988c9d8b5-cp5ms\" (UID: \"11a65ba1-0a55-4478-b043-637c0a9d6d64\") " pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-cp5ms"
Nov 02 15:05:53.059802 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:53.059707    5304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-g8tjj"]
Nov 02 15:05:53.062497 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:53.062468    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-25tqv"
Nov 02 15:05:53.063399 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:53.063372    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wskwp\" (UniqueName: \"kubernetes.io/projected/11a65ba1-0a55-4478-b043-637c0a9d6d64-kube-api-access-wskwp\") pod \"e2e-worker-w3-624254-openshift-default-7988c9d8b5-cp5ms\" (UID: \"11a65ba1-0a55-4478-b043-637c0a9d6d64\") " pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-cp5ms"
Nov 02 15:05:53.063466 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:53.063443    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11a65ba1-0a55-4478-b043-637c0a9d6d64-workload-certs\") pod \"e2e-worker-w3-624254-openshift-default-7988c9d8b5-cp5ms\" (UID: \"11a65ba1-0a55-4478-b043-637c0a9d6d64\") " pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-cp5ms"
Nov 02 15:05:53.063516 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:53.063489    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11a65ba1-0a55-4478-b043-637c0a9d6d64-workload-socket\") pod \"e2e-worker-w3-624254-openshift-default-7988c9d8b5-cp5ms\" (UID: \"11a65ba1-0a55-4478-b043-637c0a9d6d64\") " pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-cp5ms"
Nov 02 15:05:53.063569 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:53.063514    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11a65ba1-0a55-4478-b043-637c0a9d6d64-credential-socket\") pod \"e2e-worker-w3-624254-openshift-default-7988c9d8b5-cp5ms\" (UID: \"11a65ba1-0a55-4478-b043-637c0a9d6d64\") " pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-cp5ms"
Nov 02 15:05:53.063569 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:53.063542    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11a65ba1-0a55-4478-b043-637c0a9d6d64-istio-podinfo\") pod \"e2e-worker-w3-624254-openshift-default-7988c9d8b5-cp5ms\" (UID: \"11a65ba1-0a55-4478-b043-637c0a9d6d64\") " pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-cp5ms"
Nov 02 15:05:53.063673 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:53.063573    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11a65ba1-0a55-4478-b043-637c0a9d6d64-istio-token\") pod \"e2e-worker-w3-624254-openshift-default-7988c9d8b5-cp5ms\" (UID: \"11a65ba1-0a55-4478-b043-637c0a9d6d64\") " pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-cp5ms"
Nov 02 15:05:53.063673 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:53.063603    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11a65ba1-0a55-4478-b043-637c0a9d6d64-istio-data\") pod \"e2e-worker-w3-624254-openshift-default-7988c9d8b5-cp5ms\" (UID: \"11a65ba1-0a55-4478-b043-637c0a9d6d64\") " pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-cp5ms"
Nov 02 15:05:53.063673 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:53.063612    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-g8tjj"
Nov 02 15:05:53.063673 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:53.063647    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11a65ba1-0a55-4478-b043-637c0a9d6d64-istio-envoy\") pod \"e2e-worker-w3-624254-openshift-default-7988c9d8b5-cp5ms\" (UID: \"11a65ba1-0a55-4478-b043-637c0a9d6d64\") " pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-cp5ms"
Nov 02 15:05:53.063673 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:53.063671    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11a65ba1-0a55-4478-b043-637c0a9d6d64-istiod-ca-cert\") pod \"e2e-worker-w3-624254-openshift-default-7988c9d8b5-cp5ms\" (UID: \"11a65ba1-0a55-4478-b043-637c0a9d6d64\") " pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-cp5ms"
Nov 02 15:05:53.064010 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:53.063986    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11a65ba1-0a55-4478-b043-637c0a9d6d64-workload-certs\") pod \"e2e-worker-w3-624254-openshift-default-7988c9d8b5-cp5ms\" (UID: \"11a65ba1-0a55-4478-b043-637c0a9d6d64\") " pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-cp5ms"
Nov 02 15:05:53.064077 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:53.064026    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11a65ba1-0a55-4478-b043-637c0a9d6d64-workload-socket\") pod \"e2e-worker-w3-624254-openshift-default-7988c9d8b5-cp5ms\" (UID: \"11a65ba1-0a55-4478-b043-637c0a9d6d64\") " pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-cp5ms"
Nov 02 15:05:53.064301 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:53.064273    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11a65ba1-0a55-4478-b043-637c0a9d6d64-credential-socket\") pod \"e2e-worker-w3-624254-openshift-default-7988c9d8b5-cp5ms\" (UID: \"11a65ba1-0a55-4478-b043-637c0a9d6d64\") " pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-cp5ms"
Nov 02 15:05:53.065157 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:53.064675    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11a65ba1-0a55-4478-b043-637c0a9d6d64-istio-data\") pod \"e2e-worker-w3-624254-openshift-default-7988c9d8b5-cp5ms\" (UID: \"11a65ba1-0a55-4478-b043-637c0a9d6d64\") " pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-cp5ms"
Nov 02 15:05:53.065157 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:53.065022    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11a65ba1-0a55-4478-b043-637c0a9d6d64-istiod-ca-cert\") pod \"e2e-worker-w3-624254-openshift-default-7988c9d8b5-cp5ms\" (UID: \"11a65ba1-0a55-4478-b043-637c0a9d6d64\") " pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-cp5ms"
Nov 02 15:05:53.066566 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:53.066545    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11a65ba1-0a55-4478-b043-637c0a9d6d64-istio-envoy\") pod \"e2e-worker-w3-624254-openshift-default-7988c9d8b5-cp5ms\" (UID: \"11a65ba1-0a55-4478-b043-637c0a9d6d64\") " pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-cp5ms"
Nov 02 15:05:53.067061 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:53.067043    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11a65ba1-0a55-4478-b043-637c0a9d6d64-istio-podinfo\") pod \"e2e-worker-w3-624254-openshift-default-7988c9d8b5-cp5ms\" (UID: \"11a65ba1-0a55-4478-b043-637c0a9d6d64\") " pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-cp5ms"
Nov 02 15:05:53.085983 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:53.085903    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-g8tjj"]
Nov 02 15:05:53.086330 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:53.086291    5304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-1cc1b2-openshift-default-dockercfg-vsxtg\""
Nov 02 15:05:53.090821 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:53.090789    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11a65ba1-0a55-4478-b043-637c0a9d6d64-istio-token\") pod \"e2e-worker-w3-624254-openshift-default-7988c9d8b5-cp5ms\" (UID: \"11a65ba1-0a55-4478-b043-637c0a9d6d64\") " pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-cp5ms"
Nov 02 15:05:53.090947 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:53.090926    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wskwp\" (UniqueName: \"kubernetes.io/projected/11a65ba1-0a55-4478-b043-637c0a9d6d64-kube-api-access-wskwp\") pod \"e2e-worker-w3-624254-openshift-default-7988c9d8b5-cp5ms\" (UID: \"11a65ba1-0a55-4478-b043-637c0a9d6d64\") " pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-cp5ms"
Nov 02 15:05:53.164198 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:53.164162    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b48fb40-a77c-4345-a9d4-776bcdabb078-istiod-ca-cert\") pod \"e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-g8tjj\" (UID: \"0b48fb40-a77c-4345-a9d4-776bcdabb078\") " pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-g8tjj"
Nov 02 15:05:53.164198 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:53.164203    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b48fb40-a77c-4345-a9d4-776bcdabb078-istio-envoy\") pod \"e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-g8tjj\" (UID: \"0b48fb40-a77c-4345-a9d4-776bcdabb078\") " pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-g8tjj"
Nov 02 15:05:53.164400 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:53.164219    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xmffj\" (UniqueName: \"kubernetes.io/projected/0b48fb40-a77c-4345-a9d4-776bcdabb078-kube-api-access-xmffj\") pod \"e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-g8tjj\" (UID: \"0b48fb40-a77c-4345-a9d4-776bcdabb078\") " pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-g8tjj"
Nov 02 15:05:53.164400 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:53.164343    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b48fb40-a77c-4345-a9d4-776bcdabb078-workload-certs\") pod \"e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-g8tjj\" (UID: \"0b48fb40-a77c-4345-a9d4-776bcdabb078\") " pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-g8tjj"
Nov 02 15:05:53.164400 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:53.164391    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b48fb40-a77c-4345-a9d4-776bcdabb078-credential-socket\") pod \"e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-g8tjj\" (UID: \"0b48fb40-a77c-4345-a9d4-776bcdabb078\") " pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-g8tjj"
Nov 02 15:05:53.164510 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:53.164443    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b48fb40-a77c-4345-a9d4-776bcdabb078-istio-data\") pod \"e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-g8tjj\" (UID: \"0b48fb40-a77c-4345-a9d4-776bcdabb078\") " pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-g8tjj"
Nov 02 15:05:53.164510 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:53.164489    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b48fb40-a77c-4345-a9d4-776bcdabb078-istio-podinfo\") pod \"e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-g8tjj\" (UID: \"0b48fb40-a77c-4345-a9d4-776bcdabb078\") " pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-g8tjj"
Nov 02 15:05:53.164575 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:53.164525    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b48fb40-a77c-4345-a9d4-776bcdabb078-istio-token\") pod \"e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-g8tjj\" (UID: \"0b48fb40-a77c-4345-a9d4-776bcdabb078\") " pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-g8tjj"
Nov 02 15:05:53.164575 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:53.164544    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b48fb40-a77c-4345-a9d4-776bcdabb078-workload-socket\") pod \"e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-g8tjj\" (UID: \"0b48fb40-a77c-4345-a9d4-776bcdabb078\") " pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-g8tjj"
Nov 02 15:05:53.222545 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:53.222510    5304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-ldcsg"]
Nov 02 15:05:53.225056 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:53.225033    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-ldcsg"
Nov 02 15:05:53.246279 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:53.246254    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-cp5ms"
Nov 02 15:05:53.249936 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:53.249907    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-ldcsg"]
Nov 02 15:05:53.265405 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:53.265380    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b48fb40-a77c-4345-a9d4-776bcdabb078-workload-certs\") pod \"e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-g8tjj\" (UID: \"0b48fb40-a77c-4345-a9d4-776bcdabb078\") " pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-g8tjj"
Nov 02 15:05:53.265529 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:53.265415    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27105db3-fb1c-4da0-8cf0-5f301ac6967f-workload-socket\") pod \"e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-ldcsg\" (UID: \"27105db3-fb1c-4da0-8cf0-5f301ac6967f\") " pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-ldcsg"
Nov 02 15:05:53.265529 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:53.265438    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27105db3-fb1c-4da0-8cf0-5f301ac6967f-istio-envoy\") pod \"e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-ldcsg\" (UID: \"27105db3-fb1c-4da0-8cf0-5f301ac6967f\") " pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-ldcsg"
Nov 02 15:05:53.265529 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:53.265462    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b48fb40-a77c-4345-a9d4-776bcdabb078-istio-token\") pod \"e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-g8tjj\" (UID: \"0b48fb40-a77c-4345-a9d4-776bcdabb078\") " pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-g8tjj"
Nov 02 15:05:53.265529 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:53.265482    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27105db3-fb1c-4da0-8cf0-5f301ac6967f-credential-socket\") pod \"e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-ldcsg\" (UID: \"27105db3-fb1c-4da0-8cf0-5f301ac6967f\") " pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-ldcsg"
Nov 02 15:05:53.265723 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:53.265625    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b48fb40-a77c-4345-a9d4-776bcdabb078-credential-socket\") pod \"e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-g8tjj\" (UID: \"0b48fb40-a77c-4345-a9d4-776bcdabb078\") " pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-g8tjj"
Nov 02 15:05:53.265778 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:53.265713    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b48fb40-a77c-4345-a9d4-776bcdabb078-istiod-ca-cert\") pod \"e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-g8tjj\" (UID: \"0b48fb40-a77c-4345-a9d4-776bcdabb078\") " pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-g8tjj"
Nov 02 15:05:53.265778 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:53.265754    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/27105db3-fb1c-4da0-8cf0-5f301ac6967f-istio-token\") pod \"e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-ldcsg\" (UID: \"27105db3-fb1c-4da0-8cf0-5f301ac6967f\") " pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-ldcsg"
Nov 02 15:05:53.265872 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:53.265787    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b48fb40-a77c-4345-a9d4-776bcdabb078-istio-envoy\") pod \"e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-g8tjj\" (UID: \"0b48fb40-a77c-4345-a9d4-776bcdabb078\") " pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-g8tjj"
Nov 02 15:05:53.265872 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:53.265813    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xmffj\" (UniqueName: \"kubernetes.io/projected/0b48fb40-a77c-4345-a9d4-776bcdabb078-kube-api-access-xmffj\") pod \"e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-g8tjj\" (UID: \"0b48fb40-a77c-4345-a9d4-776bcdabb078\") " pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-g8tjj"
Nov 02 15:05:53.265970 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:53.265864    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b48fb40-a77c-4345-a9d4-776bcdabb078-workload-certs\") pod \"e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-g8tjj\" (UID: \"0b48fb40-a77c-4345-a9d4-776bcdabb078\") " pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-g8tjj"
Nov 02 15:05:53.265970 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:53.265874    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6696p\" (UniqueName: \"kubernetes.io/projected/27105db3-fb1c-4da0-8cf0-5f301ac6967f-kube-api-access-6696p\") pod \"e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-ldcsg\" (UID: \"27105db3-fb1c-4da0-8cf0-5f301ac6967f\") " pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-ldcsg"
Nov 02 15:05:53.265970 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:53.265917    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b48fb40-a77c-4345-a9d4-776bcdabb078-istio-podinfo\") pod \"e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-g8tjj\" (UID: \"0b48fb40-a77c-4345-a9d4-776bcdabb078\") " pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-g8tjj"
Nov 02 15:05:53.265970 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:53.265954    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b48fb40-a77c-4345-a9d4-776bcdabb078-workload-socket\") pod \"e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-g8tjj\" (UID: \"0b48fb40-a77c-4345-a9d4-776bcdabb078\") " pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-g8tjj"
Nov 02 15:05:53.266210 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:53.265980    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b48fb40-a77c-4345-a9d4-776bcdabb078-credential-socket\") pod \"e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-g8tjj\" (UID: \"0b48fb40-a77c-4345-a9d4-776bcdabb078\") " pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-g8tjj"
Nov 02 15:05:53.266210 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:53.265982    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b48fb40-a77c-4345-a9d4-776bcdabb078-istio-data\") pod \"e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-g8tjj\" (UID: \"0b48fb40-a77c-4345-a9d4-776bcdabb078\") " pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-g8tjj"
Nov 02 15:05:53.266210 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:53.266046    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27105db3-fb1c-4da0-8cf0-5f301ac6967f-istio-data\") pod \"e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-ldcsg\" (UID: \"27105db3-fb1c-4da0-8cf0-5f301ac6967f\") " pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-ldcsg"
Nov 02 15:05:53.266210 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:53.266092    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27105db3-fb1c-4da0-8cf0-5f301ac6967f-istio-podinfo\") pod \"e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-ldcsg\" (UID: \"27105db3-fb1c-4da0-8cf0-5f301ac6967f\") " pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-ldcsg"
Nov 02 15:05:53.266210 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:53.266120    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27105db3-fb1c-4da0-8cf0-5f301ac6967f-istiod-ca-cert\") pod \"e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-ldcsg\" (UID: \"27105db3-fb1c-4da0-8cf0-5f301ac6967f\") " pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-ldcsg"
Nov 02 15:05:53.266210 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:53.266176    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27105db3-fb1c-4da0-8cf0-5f301ac6967f-workload-certs\") pod \"e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-ldcsg\" (UID: \"27105db3-fb1c-4da0-8cf0-5f301ac6967f\") " pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-ldcsg"
Nov 02 15:05:53.266531 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:53.266300    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b48fb40-a77c-4345-a9d4-776bcdabb078-istiod-ca-cert\") pod \"e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-g8tjj\" (UID: \"0b48fb40-a77c-4345-a9d4-776bcdabb078\") " pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-g8tjj"
Nov 02 15:05:53.266838 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:53.266792    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b48fb40-a77c-4345-a9d4-776bcdabb078-istio-data\") pod \"e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-g8tjj\" (UID: \"0b48fb40-a77c-4345-a9d4-776bcdabb078\") " pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-g8tjj"
Nov 02 15:05:53.267043 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:53.266996    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b48fb40-a77c-4345-a9d4-776bcdabb078-workload-socket\") pod \"e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-g8tjj\" (UID: \"0b48fb40-a77c-4345-a9d4-776bcdabb078\") " pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-g8tjj"
Nov 02 15:05:53.268391 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:53.268357    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b48fb40-a77c-4345-a9d4-776bcdabb078-istio-envoy\") pod \"e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-g8tjj\" (UID: \"0b48fb40-a77c-4345-a9d4-776bcdabb078\") " pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-g8tjj"
Nov 02 15:05:53.268723 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:53.268703    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b48fb40-a77c-4345-a9d4-776bcdabb078-istio-podinfo\") pod \"e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-g8tjj\" (UID: \"0b48fb40-a77c-4345-a9d4-776bcdabb078\") " pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-g8tjj"
Nov 02 15:05:53.300575 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:53.300509    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xmffj\" (UniqueName: \"kubernetes.io/projected/0b48fb40-a77c-4345-a9d4-776bcdabb078-kube-api-access-xmffj\") pod \"e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-g8tjj\" (UID: \"0b48fb40-a77c-4345-a9d4-776bcdabb078\") " pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-g8tjj"
Nov 02 15:05:53.301240 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:53.301101    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b48fb40-a77c-4345-a9d4-776bcdabb078-istio-token\") pod \"e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-g8tjj\" (UID: \"0b48fb40-a77c-4345-a9d4-776bcdabb078\") " pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-g8tjj"
Nov 02 15:05:53.302552 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:53.302525    5304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-5dd8e9-openshift-default-dockercfg-fthlq\""
Nov 02 15:05:53.366939 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:53.366899    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27105db3-fb1c-4da0-8cf0-5f301ac6967f-workload-socket\") pod \"e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-ldcsg\" (UID: \"27105db3-fb1c-4da0-8cf0-5f301ac6967f\") " pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-ldcsg"
Nov 02 15:05:53.366939 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:53.366941    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27105db3-fb1c-4da0-8cf0-5f301ac6967f-istio-envoy\") pod \"e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-ldcsg\" (UID: \"27105db3-fb1c-4da0-8cf0-5f301ac6967f\") " pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-ldcsg"
Nov 02 15:05:53.367222 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:53.366970    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27105db3-fb1c-4da0-8cf0-5f301ac6967f-credential-socket\") pod \"e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-ldcsg\" (UID: \"27105db3-fb1c-4da0-8cf0-5f301ac6967f\") " pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-ldcsg"
Nov 02 15:05:53.367222 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:53.367014    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/27105db3-fb1c-4da0-8cf0-5f301ac6967f-istio-token\") pod \"e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-ldcsg\" (UID: \"27105db3-fb1c-4da0-8cf0-5f301ac6967f\") " pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-ldcsg"
Nov 02 15:05:53.367222 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:53.367076    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6696p\" (UniqueName: \"kubernetes.io/projected/27105db3-fb1c-4da0-8cf0-5f301ac6967f-kube-api-access-6696p\") pod \"e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-ldcsg\" (UID: \"27105db3-fb1c-4da0-8cf0-5f301ac6967f\") " pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-ldcsg"
Nov 02 15:05:53.367222 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:53.367154    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27105db3-fb1c-4da0-8cf0-5f301ac6967f-istio-data\") pod \"e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-ldcsg\" (UID: \"27105db3-fb1c-4da0-8cf0-5f301ac6967f\") " pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-ldcsg"
Nov 02 15:05:53.367222 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:53.367190    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27105db3-fb1c-4da0-8cf0-5f301ac6967f-istio-podinfo\") pod \"e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-ldcsg\" (UID: \"27105db3-fb1c-4da0-8cf0-5f301ac6967f\") " pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-ldcsg"
Nov 02 15:05:53.367222 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:53.367213    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27105db3-fb1c-4da0-8cf0-5f301ac6967f-istiod-ca-cert\") pod \"e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-ldcsg\" (UID: \"27105db3-fb1c-4da0-8cf0-5f301ac6967f\") " pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-ldcsg"
Nov 02 15:05:53.367535 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:53.367241    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27105db3-fb1c-4da0-8cf0-5f301ac6967f-workload-certs\") pod \"e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-ldcsg\" (UID: \"27105db3-fb1c-4da0-8cf0-5f301ac6967f\") " pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-ldcsg"
Nov 02 15:05:53.367535 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:53.367481    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27105db3-fb1c-4da0-8cf0-5f301ac6967f-credential-socket\") pod \"e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-ldcsg\" (UID: \"27105db3-fb1c-4da0-8cf0-5f301ac6967f\") " pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-ldcsg"
Nov 02 15:05:53.367644 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:53.367551    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27105db3-fb1c-4da0-8cf0-5f301ac6967f-workload-socket\") pod \"e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-ldcsg\" (UID: \"27105db3-fb1c-4da0-8cf0-5f301ac6967f\") " pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-ldcsg"
Nov 02 15:05:53.367705 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:53.367691    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27105db3-fb1c-4da0-8cf0-5f301ac6967f-workload-certs\") pod \"e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-ldcsg\" (UID: \"27105db3-fb1c-4da0-8cf0-5f301ac6967f\") " pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-ldcsg"
Nov 02 15:05:53.367775 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:53.367749    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27105db3-fb1c-4da0-8cf0-5f301ac6967f-istio-data\") pod \"e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-ldcsg\" (UID: \"27105db3-fb1c-4da0-8cf0-5f301ac6967f\") " pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-ldcsg"
Nov 02 15:05:53.367870 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:53.367852    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27105db3-fb1c-4da0-8cf0-5f301ac6967f-istiod-ca-cert\") pod \"e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-ldcsg\" (UID: \"27105db3-fb1c-4da0-8cf0-5f301ac6967f\") " pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-ldcsg"
Nov 02 15:05:53.369274 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:53.369258    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27105db3-fb1c-4da0-8cf0-5f301ac6967f-istio-envoy\") pod \"e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-ldcsg\" (UID: \"27105db3-fb1c-4da0-8cf0-5f301ac6967f\") " pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-ldcsg"
Nov 02 15:05:53.369589 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:53.369570    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27105db3-fb1c-4da0-8cf0-5f301ac6967f-istio-podinfo\") pod \"e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-ldcsg\" (UID: \"27105db3-fb1c-4da0-8cf0-5f301ac6967f\") " pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-ldcsg"
Nov 02 15:05:53.398226 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:53.398192    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-g8tjj"
Nov 02 15:05:53.399660 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:53.399605    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-25tqv"]
Nov 02 15:05:53.403051 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:53.401413    5304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-pshqj"]
Nov 02 15:05:53.403051 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:53.402875    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/27105db3-fb1c-4da0-8cf0-5f301ac6967f-istio-token\") pod \"e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-ldcsg\" (UID: \"27105db3-fb1c-4da0-8cf0-5f301ac6967f\") " pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-ldcsg"
Nov 02 15:05:53.404223 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:53.404199    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6696p\" (UniqueName: \"kubernetes.io/projected/27105db3-fb1c-4da0-8cf0-5f301ac6967f-kube-api-access-6696p\") pod \"e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-ldcsg\" (UID: \"27105db3-fb1c-4da0-8cf0-5f301ac6967f\") " pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-ldcsg"
Nov 02 15:05:53.404347 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:53.404272    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-pshqj"
Nov 02 15:05:53.405916 ip-10-0-135-138 kubenswrapper[5304]: W1102 15:05:53.405889    5304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ed68512_6850_45d4_af0d_a941c2bf6ec9.slice/crio-ed3cfb3ddf9d5584473a5939c1c4f17ae2ee1220e947a2364f1106d9120fcf13 WatchSource:0}: Error finding container ed3cfb3ddf9d5584473a5939c1c4f17ae2ee1220e947a2364f1106d9120fcf13: Status 404 returned error can't find the container with id ed3cfb3ddf9d5584473a5939c1c4f17ae2ee1220e947a2364f1106d9120fcf13
Nov 02 15:05:53.409689 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:53.409659    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:05:53.409802 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:53.409748    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:05:53.409867 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:53.409814    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:05:53.428721 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:53.427830    5304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-739bf1-openshift-default-dockercfg-dm45l\""
Nov 02 15:05:53.429245 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:53.429220    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-pshqj"]
Nov 02 15:05:53.467915 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:53.467886    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89710086-0e78-4395-b8f9-e47cce4f7dfe-istio-envoy\") pod \"e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-pshqj\" (UID: \"89710086-0e78-4395-b8f9-e47cce4f7dfe\") " pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-pshqj"
Nov 02 15:05:53.468102 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:53.467950    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mfhk7\" (UniqueName: \"kubernetes.io/projected/89710086-0e78-4395-b8f9-e47cce4f7dfe-kube-api-access-mfhk7\") pod \"e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-pshqj\" (UID: \"89710086-0e78-4395-b8f9-e47cce4f7dfe\") " pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-pshqj"
Nov 02 15:05:53.468102 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:53.467984    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89710086-0e78-4395-b8f9-e47cce4f7dfe-istiod-ca-cert\") pod \"e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-pshqj\" (UID: \"89710086-0e78-4395-b8f9-e47cce4f7dfe\") " pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-pshqj"
Nov 02 15:05:53.468102 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:53.468045    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89710086-0e78-4395-b8f9-e47cce4f7dfe-workload-socket\") pod \"e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-pshqj\" (UID: \"89710086-0e78-4395-b8f9-e47cce4f7dfe\") " pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-pshqj"
Nov 02 15:05:53.468102 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:53.468090    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/89710086-0e78-4395-b8f9-e47cce4f7dfe-istio-token\") pod \"e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-pshqj\" (UID: \"89710086-0e78-4395-b8f9-e47cce4f7dfe\") " pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-pshqj"
Nov 02 15:05:53.468362 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:53.468158    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89710086-0e78-4395-b8f9-e47cce4f7dfe-istio-data\") pod \"e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-pshqj\" (UID: \"89710086-0e78-4395-b8f9-e47cce4f7dfe\") " pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-pshqj"
Nov 02 15:05:53.468362 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:53.468257    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89710086-0e78-4395-b8f9-e47cce4f7dfe-istio-podinfo\") pod \"e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-pshqj\" (UID: \"89710086-0e78-4395-b8f9-e47cce4f7dfe\") " pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-pshqj"
Nov 02 15:05:53.468362 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:53.468299    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89710086-0e78-4395-b8f9-e47cce4f7dfe-credential-socket\") pod \"e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-pshqj\" (UID: \"89710086-0e78-4395-b8f9-e47cce4f7dfe\") " pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-pshqj"
Nov 02 15:05:53.468362 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:53.468323    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89710086-0e78-4395-b8f9-e47cce4f7dfe-workload-certs\") pod \"e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-pshqj\" (UID: \"89710086-0e78-4395-b8f9-e47cce4f7dfe\") " pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-pshqj"
Nov 02 15:05:53.537868 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:53.537836    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-ldcsg"
Nov 02 15:05:53.573439 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:53.563583    5304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-nrs9j"]
Nov 02 15:05:53.585201 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:53.585165    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89710086-0e78-4395-b8f9-e47cce4f7dfe-workload-socket\") pod \"e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-pshqj\" (UID: \"89710086-0e78-4395-b8f9-e47cce4f7dfe\") " pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-pshqj"
Nov 02 15:05:53.585437 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:53.585419    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/89710086-0e78-4395-b8f9-e47cce4f7dfe-istio-token\") pod \"e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-pshqj\" (UID: \"89710086-0e78-4395-b8f9-e47cce4f7dfe\") " pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-pshqj"
Nov 02 15:05:53.585553 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:53.585539    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89710086-0e78-4395-b8f9-e47cce4f7dfe-istio-data\") pod \"e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-pshqj\" (UID: \"89710086-0e78-4395-b8f9-e47cce4f7dfe\") " pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-pshqj"
Nov 02 15:05:53.585664 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:53.585649    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89710086-0e78-4395-b8f9-e47cce4f7dfe-istio-podinfo\") pod \"e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-pshqj\" (UID: \"89710086-0e78-4395-b8f9-e47cce4f7dfe\") " pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-pshqj"
Nov 02 15:05:53.585765 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:53.585753    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89710086-0e78-4395-b8f9-e47cce4f7dfe-credential-socket\") pod \"e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-pshqj\" (UID: \"89710086-0e78-4395-b8f9-e47cce4f7dfe\") " pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-pshqj"
Nov 02 15:05:53.585860 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:53.585848    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89710086-0e78-4395-b8f9-e47cce4f7dfe-workload-certs\") pod \"e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-pshqj\" (UID: \"89710086-0e78-4395-b8f9-e47cce4f7dfe\") " pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-pshqj"
Nov 02 15:05:53.598160 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:53.588421    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89710086-0e78-4395-b8f9-e47cce4f7dfe-istio-envoy\") pod \"e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-pshqj\" (UID: \"89710086-0e78-4395-b8f9-e47cce4f7dfe\") " pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-pshqj"
Nov 02 15:05:53.598160 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:53.588491    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89710086-0e78-4395-b8f9-e47cce4f7dfe-workload-socket\") pod \"e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-pshqj\" (UID: \"89710086-0e78-4395-b8f9-e47cce4f7dfe\") " pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-pshqj"
Nov 02 15:05:53.598160 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:53.588498    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mfhk7\" (UniqueName: \"kubernetes.io/projected/89710086-0e78-4395-b8f9-e47cce4f7dfe-kube-api-access-mfhk7\") pod \"e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-pshqj\" (UID: \"89710086-0e78-4395-b8f9-e47cce4f7dfe\") " pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-pshqj"
Nov 02 15:05:53.598160 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:53.588543    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89710086-0e78-4395-b8f9-e47cce4f7dfe-istiod-ca-cert\") pod \"e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-pshqj\" (UID: \"89710086-0e78-4395-b8f9-e47cce4f7dfe\") " pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-pshqj"
Nov 02 15:05:53.598160 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:53.588844    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89710086-0e78-4395-b8f9-e47cce4f7dfe-workload-certs\") pod \"e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-pshqj\" (UID: \"89710086-0e78-4395-b8f9-e47cce4f7dfe\") " pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-pshqj"
Nov 02 15:05:53.598160 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:53.589026    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89710086-0e78-4395-b8f9-e47cce4f7dfe-credential-socket\") pod \"e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-pshqj\" (UID: \"89710086-0e78-4395-b8f9-e47cce4f7dfe\") " pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-pshqj"
Nov 02 15:05:53.598160 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:53.589291    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89710086-0e78-4395-b8f9-e47cce4f7dfe-istiod-ca-cert\") pod \"e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-pshqj\" (UID: \"89710086-0e78-4395-b8f9-e47cce4f7dfe\") " pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-pshqj"
Nov 02 15:05:53.598160 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:53.589983    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89710086-0e78-4395-b8f9-e47cce4f7dfe-istio-data\") pod \"e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-pshqj\" (UID: \"89710086-0e78-4395-b8f9-e47cce4f7dfe\") " pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-pshqj"
Nov 02 15:05:53.598160 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:53.590068    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89710086-0e78-4395-b8f9-e47cce4f7dfe-istio-podinfo\") pod \"e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-pshqj\" (UID: \"89710086-0e78-4395-b8f9-e47cce4f7dfe\") " pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-pshqj"
Nov 02 15:05:53.598160 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:53.593228    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:05:53.598160 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:53.593312    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:05:53.598160 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:53.593358    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:05:53.598160 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:53.595482    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89710086-0e78-4395-b8f9-e47cce4f7dfe-istio-envoy\") pod \"e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-pshqj\" (UID: \"89710086-0e78-4395-b8f9-e47cce4f7dfe\") " pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-pshqj"
Nov 02 15:05:53.602696 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:53.600065    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-cp5ms"]
Nov 02 15:05:53.602696 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:53.600101    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-nrs9j"]
Nov 02 15:05:53.602696 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:53.600204    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-nrs9j"
Nov 02 15:05:53.614868 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:53.614834    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/89710086-0e78-4395-b8f9-e47cce4f7dfe-istio-token\") pod \"e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-pshqj\" (UID: \"89710086-0e78-4395-b8f9-e47cce4f7dfe\") " pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-pshqj"
Nov 02 15:05:53.616833 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:53.616800    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mfhk7\" (UniqueName: \"kubernetes.io/projected/89710086-0e78-4395-b8f9-e47cce4f7dfe-kube-api-access-mfhk7\") pod \"e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-pshqj\" (UID: \"89710086-0e78-4395-b8f9-e47cce4f7dfe\") " pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-pshqj"
Nov 02 15:05:53.627095 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:53.625344    5304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-6af50f-openshift-default-dockercfg-n5jqm\""
Nov 02 15:05:53.689164 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:53.689093    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1051c4a-02ca-40b8-b8de-180bd2ca598b-istio-podinfo\") pod \"e2e-worker-w0-6af50f-openshift-default-8545fb68f-nrs9j\" (UID: \"c1051c4a-02ca-40b8-b8de-180bd2ca598b\") " pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-nrs9j"
Nov 02 15:05:53.689164 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:53.689148    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1051c4a-02ca-40b8-b8de-180bd2ca598b-istio-envoy\") pod \"e2e-worker-w0-6af50f-openshift-default-8545fb68f-nrs9j\" (UID: \"c1051c4a-02ca-40b8-b8de-180bd2ca598b\") " pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-nrs9j"
Nov 02 15:05:53.689332 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:53.689232    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1051c4a-02ca-40b8-b8de-180bd2ca598b-workload-socket\") pod \"e2e-worker-w0-6af50f-openshift-default-8545fb68f-nrs9j\" (UID: \"c1051c4a-02ca-40b8-b8de-180bd2ca598b\") " pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-nrs9j"
Nov 02 15:05:53.689332 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:53.689269    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1051c4a-02ca-40b8-b8de-180bd2ca598b-credential-socket\") pod \"e2e-worker-w0-6af50f-openshift-default-8545fb68f-nrs9j\" (UID: \"c1051c4a-02ca-40b8-b8de-180bd2ca598b\") " pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-nrs9j"
Nov 02 15:05:53.689332 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:53.689309    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1051c4a-02ca-40b8-b8de-180bd2ca598b-istiod-ca-cert\") pod \"e2e-worker-w0-6af50f-openshift-default-8545fb68f-nrs9j\" (UID: \"c1051c4a-02ca-40b8-b8de-180bd2ca598b\") " pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-nrs9j"
Nov 02 15:05:53.689440 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:53.689407    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1051c4a-02ca-40b8-b8de-180bd2ca598b-istio-token\") pod \"e2e-worker-w0-6af50f-openshift-default-8545fb68f-nrs9j\" (UID: \"c1051c4a-02ca-40b8-b8de-180bd2ca598b\") " pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-nrs9j"
Nov 02 15:05:53.689483 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:53.689440    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1051c4a-02ca-40b8-b8de-180bd2ca598b-istio-data\") pod \"e2e-worker-w0-6af50f-openshift-default-8545fb68f-nrs9j\" (UID: \"c1051c4a-02ca-40b8-b8de-180bd2ca598b\") " pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-nrs9j"
Nov 02 15:05:53.689526 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:53.689498    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h46cm\" (UniqueName: \"kubernetes.io/projected/c1051c4a-02ca-40b8-b8de-180bd2ca598b-kube-api-access-h46cm\") pod \"e2e-worker-w0-6af50f-openshift-default-8545fb68f-nrs9j\" (UID: \"c1051c4a-02ca-40b8-b8de-180bd2ca598b\") " pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-nrs9j"
Nov 02 15:05:53.689599 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:53.689582    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1051c4a-02ca-40b8-b8de-180bd2ca598b-workload-certs\") pod \"e2e-worker-w0-6af50f-openshift-default-8545fb68f-nrs9j\" (UID: \"c1051c4a-02ca-40b8-b8de-180bd2ca598b\") " pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-nrs9j"
Nov 02 15:05:53.714997 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:53.714963    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-g8tjj"]
Nov 02 15:05:53.718152 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:53.717883    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-pshqj"
Nov 02 15:05:53.730480 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:53.729399    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:05:53.730480 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:53.729519    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:05:53.730480 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:53.729564    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:05:53.792512 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:53.791384    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1051c4a-02ca-40b8-b8de-180bd2ca598b-istio-token\") pod \"e2e-worker-w0-6af50f-openshift-default-8545fb68f-nrs9j\" (UID: \"c1051c4a-02ca-40b8-b8de-180bd2ca598b\") " pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-nrs9j"
Nov 02 15:05:53.792512 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:53.791436    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1051c4a-02ca-40b8-b8de-180bd2ca598b-istio-data\") pod \"e2e-worker-w0-6af50f-openshift-default-8545fb68f-nrs9j\" (UID: \"c1051c4a-02ca-40b8-b8de-180bd2ca598b\") " pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-nrs9j"
Nov 02 15:05:53.792512 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:53.791463    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h46cm\" (UniqueName: \"kubernetes.io/projected/c1051c4a-02ca-40b8-b8de-180bd2ca598b-kube-api-access-h46cm\") pod \"e2e-worker-w0-6af50f-openshift-default-8545fb68f-nrs9j\" (UID: \"c1051c4a-02ca-40b8-b8de-180bd2ca598b\") " pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-nrs9j"
Nov 02 15:05:53.792512 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:53.791501    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1051c4a-02ca-40b8-b8de-180bd2ca598b-workload-certs\") pod \"e2e-worker-w0-6af50f-openshift-default-8545fb68f-nrs9j\" (UID: \"c1051c4a-02ca-40b8-b8de-180bd2ca598b\") " pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-nrs9j"
Nov 02 15:05:53.792512 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:53.791539    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1051c4a-02ca-40b8-b8de-180bd2ca598b-istio-podinfo\") pod \"e2e-worker-w0-6af50f-openshift-default-8545fb68f-nrs9j\" (UID: \"c1051c4a-02ca-40b8-b8de-180bd2ca598b\") " pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-nrs9j"
Nov 02 15:05:53.792512 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:53.791567    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1051c4a-02ca-40b8-b8de-180bd2ca598b-istio-envoy\") pod \"e2e-worker-w0-6af50f-openshift-default-8545fb68f-nrs9j\" (UID: \"c1051c4a-02ca-40b8-b8de-180bd2ca598b\") " pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-nrs9j"
Nov 02 15:05:53.792512 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:53.791608    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1051c4a-02ca-40b8-b8de-180bd2ca598b-workload-socket\") pod \"e2e-worker-w0-6af50f-openshift-default-8545fb68f-nrs9j\" (UID: \"c1051c4a-02ca-40b8-b8de-180bd2ca598b\") " pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-nrs9j"
Nov 02 15:05:53.792512 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:53.791655    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1051c4a-02ca-40b8-b8de-180bd2ca598b-credential-socket\") pod \"e2e-worker-w0-6af50f-openshift-default-8545fb68f-nrs9j\" (UID: \"c1051c4a-02ca-40b8-b8de-180bd2ca598b\") " pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-nrs9j"
Nov 02 15:05:53.792512 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:53.791703    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1051c4a-02ca-40b8-b8de-180bd2ca598b-istiod-ca-cert\") pod \"e2e-worker-w0-6af50f-openshift-default-8545fb68f-nrs9j\" (UID: \"c1051c4a-02ca-40b8-b8de-180bd2ca598b\") " pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-nrs9j"
Nov 02 15:05:53.795108 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:53.793676    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1051c4a-02ca-40b8-b8de-180bd2ca598b-workload-certs\") pod \"e2e-worker-w0-6af50f-openshift-default-8545fb68f-nrs9j\" (UID: \"c1051c4a-02ca-40b8-b8de-180bd2ca598b\") " pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-nrs9j"
Nov 02 15:05:53.795108 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:53.794290    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1051c4a-02ca-40b8-b8de-180bd2ca598b-workload-socket\") pod \"e2e-worker-w0-6af50f-openshift-default-8545fb68f-nrs9j\" (UID: \"c1051c4a-02ca-40b8-b8de-180bd2ca598b\") " pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-nrs9j"
Nov 02 15:05:53.795108 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:53.794323    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1051c4a-02ca-40b8-b8de-180bd2ca598b-credential-socket\") pod \"e2e-worker-w0-6af50f-openshift-default-8545fb68f-nrs9j\" (UID: \"c1051c4a-02ca-40b8-b8de-180bd2ca598b\") " pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-nrs9j"
Nov 02 15:05:53.795108 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:53.794820    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1051c4a-02ca-40b8-b8de-180bd2ca598b-istio-data\") pod \"e2e-worker-w0-6af50f-openshift-default-8545fb68f-nrs9j\" (UID: \"c1051c4a-02ca-40b8-b8de-180bd2ca598b\") " pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-nrs9j"
Nov 02 15:05:53.795457 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:53.795152    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1051c4a-02ca-40b8-b8de-180bd2ca598b-istiod-ca-cert\") pod \"e2e-worker-w0-6af50f-openshift-default-8545fb68f-nrs9j\" (UID: \"c1051c4a-02ca-40b8-b8de-180bd2ca598b\") " pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-nrs9j"
Nov 02 15:05:53.797178 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:53.796849    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1051c4a-02ca-40b8-b8de-180bd2ca598b-istio-envoy\") pod \"e2e-worker-w0-6af50f-openshift-default-8545fb68f-nrs9j\" (UID: \"c1051c4a-02ca-40b8-b8de-180bd2ca598b\") " pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-nrs9j"
Nov 02 15:05:53.797449 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:53.797429    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1051c4a-02ca-40b8-b8de-180bd2ca598b-istio-podinfo\") pod \"e2e-worker-w0-6af50f-openshift-default-8545fb68f-nrs9j\" (UID: \"c1051c4a-02ca-40b8-b8de-180bd2ca598b\") " pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-nrs9j"
Nov 02 15:05:53.824475 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:53.824428    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h46cm\" (UniqueName: \"kubernetes.io/projected/c1051c4a-02ca-40b8-b8de-180bd2ca598b-kube-api-access-h46cm\") pod \"e2e-worker-w0-6af50f-openshift-default-8545fb68f-nrs9j\" (UID: \"c1051c4a-02ca-40b8-b8de-180bd2ca598b\") " pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-nrs9j"
Nov 02 15:05:53.824999 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:53.824940    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1051c4a-02ca-40b8-b8de-180bd2ca598b-istio-token\") pod \"e2e-worker-w0-6af50f-openshift-default-8545fb68f-nrs9j\" (UID: \"c1051c4a-02ca-40b8-b8de-180bd2ca598b\") " pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-nrs9j"
Nov 02 15:05:53.913166 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:53.913070    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-ldcsg"]
Nov 02 15:05:53.921203 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:53.920610    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:05:53.921203 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:53.920719    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:05:53.921203 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:53.920762    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:05:53.926657 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:53.926443    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-nrs9j"
Nov 02 15:05:54.091007 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:54.088248    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-pshqj"]
Nov 02 15:05:54.094477 ip-10-0-135-138 kubenswrapper[5304]: W1102 15:05:54.094432    5304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89710086_0e78_4395_b8f9_e47cce4f7dfe.slice/crio-b22b8b1d23aa8eac8a94fe384a915c78536066ad85d43a9e083e3c5d3d36208d WatchSource:0}: Error finding container b22b8b1d23aa8eac8a94fe384a915c78536066ad85d43a9e083e3c5d3d36208d: Status 404 returned error can't find the container with id b22b8b1d23aa8eac8a94fe384a915c78536066ad85d43a9e083e3c5d3d36208d
Nov 02 15:05:54.097450 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:54.097407    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:05:54.097546 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:54.097499    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:05:54.097546 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:54.097540    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:05:54.129843 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:54.129812    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-cp5ms" event={"ID":"11a65ba1-0a55-4478-b043-637c0a9d6d64","Type":"ContainerStarted","Data":"66e57b8107829805af57f0382cb5ffde3b62067aae95273892bb44947fb1304d"}
Nov 02 15:05:54.129972 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:54.129868    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-cp5ms" event={"ID":"11a65ba1-0a55-4478-b043-637c0a9d6d64","Type":"ContainerStarted","Data":"25c68a804ac50dc74ee65fde529de4a6579b9bc0b46de19a466accbee2bfef04"}
Nov 02 15:05:54.131714 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:54.131684    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-g8tjj" event={"ID":"0b48fb40-a77c-4345-a9d4-776bcdabb078","Type":"ContainerStarted","Data":"925c184dcf230cd74bfef28c14af0b95596ad6cd7a419c7948359b1478c56603"}
Nov 02 15:05:54.131929 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:54.131895    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-g8tjj" event={"ID":"0b48fb40-a77c-4345-a9d4-776bcdabb078","Type":"ContainerStarted","Data":"a5cadc32746496ae272409ec178c19410b6bbf9db2d7a5753600207f243cf8c2"}
Nov 02 15:05:54.133632 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:54.133393    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-25tqv" event={"ID":"3ed68512-6850-45d4-af0d-a941c2bf6ec9","Type":"ContainerStarted","Data":"eecc968f16d732bb7fb9155081d87e858ec89a885a7242f9502bda03fad23c51"}
Nov 02 15:05:54.133632 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:54.133433    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-25tqv" event={"ID":"3ed68512-6850-45d4-af0d-a941c2bf6ec9","Type":"ContainerStarted","Data":"ed3cfb3ddf9d5584473a5939c1c4f17ae2ee1220e947a2364f1106d9120fcf13"}
Nov 02 15:05:54.134976 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:54.134952    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-pshqj" event={"ID":"89710086-0e78-4395-b8f9-e47cce4f7dfe","Type":"ContainerStarted","Data":"b22b8b1d23aa8eac8a94fe384a915c78536066ad85d43a9e083e3c5d3d36208d"}
Nov 02 15:05:54.136687 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:54.136337    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-ldcsg" event={"ID":"27105db3-fb1c-4da0-8cf0-5f301ac6967f","Type":"ContainerStarted","Data":"a942baeee15555237950f168b16307153d6ecf18f87153f75833ffd30e255eb1"}
Nov 02 15:05:54.136687 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:54.136376    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-ldcsg" event={"ID":"27105db3-fb1c-4da0-8cf0-5f301ac6967f","Type":"ContainerStarted","Data":"945eddde5e4cf858141f8ed45ad6794ad808f8ca82fa6f9edd55c94c2fb70e67"}
Nov 02 15:05:54.231758 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:54.231703    5304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-25tqv" podStartSLOduration=2.231689039 podStartE2EDuration="2.231689039s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:05:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:05:54.23090872 +0000 UTC m=+554.224733133" watchObservedRunningTime="2026-11-02 15:05:54.231689039 +0000 UTC m=+554.225513470"
Nov 02 15:05:54.232840 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:54.232802    5304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-cp5ms" podStartSLOduration=2.232791992 podStartE2EDuration="2.232791992s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:05:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:05:54.182452558 +0000 UTC m=+554.176276968" watchObservedRunningTime="2026-11-02 15:05:54.232791992 +0000 UTC m=+554.226616400"
Nov 02 15:05:54.246609 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:54.246573    5304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-cp5ms"
Nov 02 15:05:54.251999 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:54.251964    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-nrs9j"]
Nov 02 15:05:54.253898 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:54.253875    5304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-cp5ms"
Nov 02 15:05:54.256416 ip-10-0-135-138 kubenswrapper[5304]: W1102 15:05:54.256370    5304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1051c4a_02ca_40b8_b8de_180bd2ca598b.slice/crio-d36f3b581eeaaa3d3dd863e265ff6ed4a3920d387341834892edded20fdd5f99 WatchSource:0}: Error finding container d36f3b581eeaaa3d3dd863e265ff6ed4a3920d387341834892edded20fdd5f99: Status 404 returned error can't find the container with id d36f3b581eeaaa3d3dd863e265ff6ed4a3920d387341834892edded20fdd5f99
Nov 02 15:05:54.259294 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:54.259266    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:05:54.259350 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:54.259327    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:05:54.259391 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:54.259356    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:05:54.282509 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:54.282426    5304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-g8tjj" podStartSLOduration=1.2824126169999999 podStartE2EDuration="1.282412617s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:05:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:05:54.281044267 +0000 UTC m=+554.274868676" watchObservedRunningTime="2026-11-02 15:05:54.282412617 +0000 UTC m=+554.276237026"
Nov 02 15:05:54.331010 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:54.330964    5304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-ldcsg" podStartSLOduration=1.330949607 podStartE2EDuration="1.330949607s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:05:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:05:54.330194293 +0000 UTC m=+554.324018701" watchObservedRunningTime="2026-11-02 15:05:54.330949607 +0000 UTC m=+554.324774015"
Nov 02 15:05:54.399163 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:54.399114    5304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-g8tjj"
Nov 02 15:05:54.404418 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:54.404394    5304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-g8tjj"
Nov 02 15:05:54.538913 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:54.538833    5304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-ldcsg"
Nov 02 15:05:54.797845 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:05:54.797765    5304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k6fjk" containerName="extractor"
Nov 02 15:05:55.063254 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:55.063172    5304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-25tqv"
Nov 02 15:05:55.068188 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:55.068165    5304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-25tqv"
Nov 02 15:05:55.142386 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:55.142344    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-pshqj" event={"ID":"89710086-0e78-4395-b8f9-e47cce4f7dfe","Type":"ContainerStarted","Data":"6dd484c90f59525171dfb12122eb7a12a797fd5f1bf98e31a7ef3f7c6170893b"}
Nov 02 15:05:55.144008 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:55.143981    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-nrs9j" event={"ID":"c1051c4a-02ca-40b8-b8de-180bd2ca598b","Type":"ContainerStarted","Data":"9fafab017e9dff6e5bce71d3792068134589f9b46bfe53dbbefbeaefabbdb83c"}
Nov 02 15:05:55.144227 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:55.144022    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-nrs9j" event={"ID":"c1051c4a-02ca-40b8-b8de-180bd2ca598b","Type":"ContainerStarted","Data":"d36f3b581eeaaa3d3dd863e265ff6ed4a3920d387341834892edded20fdd5f99"}
Nov 02 15:05:55.144305 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:55.144262    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-cp5ms"
Nov 02 15:05:55.145044 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:55.144628    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-g8tjj"
Nov 02 15:05:55.145044 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:55.144790    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-25tqv"
Nov 02 15:05:55.145417 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:55.145398    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-cp5ms"
Nov 02 15:05:55.145486 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:55.145445    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-g8tjj"
Nov 02 15:05:55.145486 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:55.145473    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-25tqv"
Nov 02 15:05:55.193701 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:55.193646    5304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-pshqj" podStartSLOduration=2.193631609 podStartE2EDuration="2.193631609s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:05:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:05:55.190101768 +0000 UTC m=+555.183926177" watchObservedRunningTime="2026-11-02 15:05:55.193631609 +0000 UTC m=+555.187456018"
Nov 02 15:05:55.305678 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:55.305625    5304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-nrs9j" podStartSLOduration=2.305609452 podStartE2EDuration="2.305609452s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:05:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:05:55.304021844 +0000 UTC m=+555.297846255" watchObservedRunningTime="2026-11-02 15:05:55.305609452 +0000 UTC m=+555.299433861"
Nov 02 15:05:55.552587 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:55.552550    5304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-ldcsg"
Nov 02 15:05:55.720212 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:55.720170    5304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-pshqj"
Nov 02 15:05:55.725537 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:55.725508    5304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-pshqj"
Nov 02 15:05:55.927633 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:55.927600    5304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-nrs9j"
Nov 02 15:05:55.932707 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:55.932683    5304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-nrs9j"
Nov 02 15:05:56.148628 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:56.148592    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-pshqj"
Nov 02 15:05:56.148829 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:56.148804    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-ldcsg"
Nov 02 15:05:56.149431 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:56.149407    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-nrs9j"
Nov 02 15:05:56.149746 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:56.149719    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-pshqj"
Nov 02 15:05:56.149846 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:56.149811    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-ldcsg"
Nov 02 15:05:56.150320 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:05:56.150301    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-nrs9j"
Nov 02 15:06:02.187683 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:02.187647    5304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-5dd8e9-68fff466d7-7dgm5"]
Nov 02 15:06:02.191344 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:02.191328    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-5dd8e9-68fff466d7-7dgm5"
Nov 02 15:06:02.212339 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:02.212312    5304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w1-5dd8e9-serving-cert\""
Nov 02 15:06:02.229730 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:02.229705    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-5dd8e9-68fff466d7-7dgm5"]
Nov 02 15:06:02.280651 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:02.280619    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kh64b\" (UniqueName: \"kubernetes.io/projected/f5ef30f3-a503-42ee-a484-caf583967d39-kube-api-access-kh64b\") pod \"maas-api-e2e-worker-w1-5dd8e9-68fff466d7-7dgm5\" (UID: \"f5ef30f3-a503-42ee-a484-caf583967d39\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-5dd8e9-68fff466d7-7dgm5"
Nov 02 15:06:02.280819 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:02.280658    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f5ef30f3-a503-42ee-a484-caf583967d39-maas-api-tls\") pod \"maas-api-e2e-worker-w1-5dd8e9-68fff466d7-7dgm5\" (UID: \"f5ef30f3-a503-42ee-a484-caf583967d39\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-5dd8e9-68fff466d7-7dgm5"
Nov 02 15:06:02.280819 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:02.280677    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f5ef30f3-a503-42ee-a484-caf583967d39-metrics-certs\") pod \"maas-api-e2e-worker-w1-5dd8e9-68fff466d7-7dgm5\" (UID: \"f5ef30f3-a503-42ee-a484-caf583967d39\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-5dd8e9-68fff466d7-7dgm5"
Nov 02 15:06:02.280819 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:02.280806    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5ef30f3-a503-42ee-a484-caf583967d39-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-5dd8e9-68fff466d7-7dgm5\" (UID: \"f5ef30f3-a503-42ee-a484-caf583967d39\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-5dd8e9-68fff466d7-7dgm5"
Nov 02 15:06:02.381717 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:02.381680    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5ef30f3-a503-42ee-a484-caf583967d39-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-5dd8e9-68fff466d7-7dgm5\" (UID: \"f5ef30f3-a503-42ee-a484-caf583967d39\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-5dd8e9-68fff466d7-7dgm5"
Nov 02 15:06:02.381914 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:02.381785    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kh64b\" (UniqueName: \"kubernetes.io/projected/f5ef30f3-a503-42ee-a484-caf583967d39-kube-api-access-kh64b\") pod \"maas-api-e2e-worker-w1-5dd8e9-68fff466d7-7dgm5\" (UID: \"f5ef30f3-a503-42ee-a484-caf583967d39\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-5dd8e9-68fff466d7-7dgm5"
Nov 02 15:06:02.381914 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:02.381806    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f5ef30f3-a503-42ee-a484-caf583967d39-maas-api-tls\") pod \"maas-api-e2e-worker-w1-5dd8e9-68fff466d7-7dgm5\" (UID: \"f5ef30f3-a503-42ee-a484-caf583967d39\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-5dd8e9-68fff466d7-7dgm5"
Nov 02 15:06:02.381914 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:02.381821    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f5ef30f3-a503-42ee-a484-caf583967d39-metrics-certs\") pod \"maas-api-e2e-worker-w1-5dd8e9-68fff466d7-7dgm5\" (UID: \"f5ef30f3-a503-42ee-a484-caf583967d39\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-5dd8e9-68fff466d7-7dgm5"
Nov 02 15:06:02.382445 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:02.382418    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5ef30f3-a503-42ee-a484-caf583967d39-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-5dd8e9-68fff466d7-7dgm5\" (UID: \"f5ef30f3-a503-42ee-a484-caf583967d39\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-5dd8e9-68fff466d7-7dgm5"
Nov 02 15:06:02.384313 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:02.384288    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f5ef30f3-a503-42ee-a484-caf583967d39-metrics-certs\") pod \"maas-api-e2e-worker-w1-5dd8e9-68fff466d7-7dgm5\" (UID: \"f5ef30f3-a503-42ee-a484-caf583967d39\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-5dd8e9-68fff466d7-7dgm5"
Nov 02 15:06:02.384424 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:02.384331    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f5ef30f3-a503-42ee-a484-caf583967d39-maas-api-tls\") pod \"maas-api-e2e-worker-w1-5dd8e9-68fff466d7-7dgm5\" (UID: \"f5ef30f3-a503-42ee-a484-caf583967d39\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-5dd8e9-68fff466d7-7dgm5"
Nov 02 15:06:02.405564 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:02.405536    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kh64b\" (UniqueName: \"kubernetes.io/projected/f5ef30f3-a503-42ee-a484-caf583967d39-kube-api-access-kh64b\") pod \"maas-api-e2e-worker-w1-5dd8e9-68fff466d7-7dgm5\" (UID: \"f5ef30f3-a503-42ee-a484-caf583967d39\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-5dd8e9-68fff466d7-7dgm5"
Nov 02 15:06:02.504425 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:02.504340    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-5dd8e9-68fff466d7-7dgm5"
Nov 02 15:06:02.801699 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:02.801596    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-5dd8e9-68fff466d7-7dgm5"]
Nov 02 15:06:02.805512 ip-10-0-135-138 kubenswrapper[5304]: W1102 15:06:02.805484    5304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5ef30f3_a503_42ee_a484_caf583967d39.slice/crio-32c5f15f5496e2d0861e2df17e31b1019d0c34db2a02f363d15868ee92b6e9cb WatchSource:0}: Error finding container 32c5f15f5496e2d0861e2df17e31b1019d0c34db2a02f363d15868ee92b6e9cb: Status 404 returned error can't find the container with id 32c5f15f5496e2d0861e2df17e31b1019d0c34db2a02f363d15868ee92b6e9cb
Nov 02 15:06:03.180004 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:03.179976    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-5dd8e9-68fff466d7-7dgm5" event={"ID":"f5ef30f3-a503-42ee-a484-caf583967d39","Type":"ContainerStarted","Data":"32c5f15f5496e2d0861e2df17e31b1019d0c34db2a02f363d15868ee92b6e9cb"}
Nov 02 15:06:04.053150 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:04.053099    5304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-624254-576d847589-4jfld"]
Nov 02 15:06:04.060806 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:04.060775    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-624254-576d847589-4jfld"
Nov 02 15:06:04.092436 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:04.092402    5304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w3-624254-serving-cert\""
Nov 02 15:06:04.094343 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:04.094313    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-624254-576d847589-4jfld"]
Nov 02 15:06:04.099887 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:04.099697    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e311645-6f4b-4169-8c27-8ad75ac36da2-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-624254-576d847589-4jfld\" (UID: \"2e311645-6f4b-4169-8c27-8ad75ac36da2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-624254-576d847589-4jfld"
Nov 02 15:06:04.099887 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:04.099781    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2e311645-6f4b-4169-8c27-8ad75ac36da2-maas-api-tls\") pod \"maas-api-e2e-worker-w3-624254-576d847589-4jfld\" (UID: \"2e311645-6f4b-4169-8c27-8ad75ac36da2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-624254-576d847589-4jfld"
Nov 02 15:06:04.099887 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:04.099809    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jxhpr\" (UniqueName: \"kubernetes.io/projected/2e311645-6f4b-4169-8c27-8ad75ac36da2-kube-api-access-jxhpr\") pod \"maas-api-e2e-worker-w3-624254-576d847589-4jfld\" (UID: \"2e311645-6f4b-4169-8c27-8ad75ac36da2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-624254-576d847589-4jfld"
Nov 02 15:06:04.099887 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:04.099874    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2e311645-6f4b-4169-8c27-8ad75ac36da2-metrics-certs\") pod \"maas-api-e2e-worker-w3-624254-576d847589-4jfld\" (UID: \"2e311645-6f4b-4169-8c27-8ad75ac36da2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-624254-576d847589-4jfld"
Nov 02 15:06:04.186845 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:04.186804    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-5dd8e9-68fff466d7-7dgm5" event={"ID":"f5ef30f3-a503-42ee-a484-caf583967d39","Type":"ContainerStarted","Data":"e67722e309f649301fc7d99b2b5087cd346d4a3f9ed6b37b31aa90db878b5dc2"}
Nov 02 15:06:04.187016 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:04.186935    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-5dd8e9-68fff466d7-7dgm5"
Nov 02 15:06:04.200616 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:04.200584    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e311645-6f4b-4169-8c27-8ad75ac36da2-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-624254-576d847589-4jfld\" (UID: \"2e311645-6f4b-4169-8c27-8ad75ac36da2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-624254-576d847589-4jfld"
Nov 02 15:06:04.200777 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:04.200656    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2e311645-6f4b-4169-8c27-8ad75ac36da2-maas-api-tls\") pod \"maas-api-e2e-worker-w3-624254-576d847589-4jfld\" (UID: \"2e311645-6f4b-4169-8c27-8ad75ac36da2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-624254-576d847589-4jfld"
Nov 02 15:06:04.201057 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:06:04.201032    5304 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w3-624254-serving-cert: secret "maas-api-e2e-worker-w3-624254-serving-cert" not found
Nov 02 15:06:04.201186 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:04.201108    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jxhpr\" (UniqueName: \"kubernetes.io/projected/2e311645-6f4b-4169-8c27-8ad75ac36da2-kube-api-access-jxhpr\") pod \"maas-api-e2e-worker-w3-624254-576d847589-4jfld\" (UID: \"2e311645-6f4b-4169-8c27-8ad75ac36da2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-624254-576d847589-4jfld"
Nov 02 15:06:04.201186 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:06:04.201114    5304 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2e311645-6f4b-4169-8c27-8ad75ac36da2-maas-api-tls podName:2e311645-6f4b-4169-8c27-8ad75ac36da2 nodeName:}" failed. No retries permitted until 2026-11-02 15:06:04.701093871 +0000 UTC m=+564.694918293 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/2e311645-6f4b-4169-8c27-8ad75ac36da2-maas-api-tls") pod "maas-api-e2e-worker-w3-624254-576d847589-4jfld" (UID: "2e311645-6f4b-4169-8c27-8ad75ac36da2") : secret "maas-api-e2e-worker-w3-624254-serving-cert" not found
Nov 02 15:06:04.201294 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:04.201269    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2e311645-6f4b-4169-8c27-8ad75ac36da2-metrics-certs\") pod \"maas-api-e2e-worker-w3-624254-576d847589-4jfld\" (UID: \"2e311645-6f4b-4169-8c27-8ad75ac36da2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-624254-576d847589-4jfld"
Nov 02 15:06:04.201502 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:04.201482    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e311645-6f4b-4169-8c27-8ad75ac36da2-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-624254-576d847589-4jfld\" (UID: \"2e311645-6f4b-4169-8c27-8ad75ac36da2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-624254-576d847589-4jfld"
Nov 02 15:06:04.203860 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:04.203834    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2e311645-6f4b-4169-8c27-8ad75ac36da2-metrics-certs\") pod \"maas-api-e2e-worker-w3-624254-576d847589-4jfld\" (UID: \"2e311645-6f4b-4169-8c27-8ad75ac36da2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-624254-576d847589-4jfld"
Nov 02 15:06:04.224770 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:04.224743    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jxhpr\" (UniqueName: \"kubernetes.io/projected/2e311645-6f4b-4169-8c27-8ad75ac36da2-kube-api-access-jxhpr\") pod \"maas-api-e2e-worker-w3-624254-576d847589-4jfld\" (UID: \"2e311645-6f4b-4169-8c27-8ad75ac36da2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-624254-576d847589-4jfld"
Nov 02 15:06:04.229925 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:04.229729    5304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-5dd8e9-68fff466d7-7dgm5" podStartSLOduration=1.8745167139999999 podStartE2EDuration="2.229714721s" totalImagesPullingTime="355.198007ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:06:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:06:04.229171523 +0000 UTC m=+564.222995933" watchObservedRunningTime="2026-11-02 15:06:04.229714721 +0000 UTC m=+564.223539130"
Nov 02 15:06:04.707300 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:04.707265    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2e311645-6f4b-4169-8c27-8ad75ac36da2-maas-api-tls\") pod \"maas-api-e2e-worker-w3-624254-576d847589-4jfld\" (UID: \"2e311645-6f4b-4169-8c27-8ad75ac36da2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-624254-576d847589-4jfld"
Nov 02 15:06:04.709828 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:04.709781    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2e311645-6f4b-4169-8c27-8ad75ac36da2-maas-api-tls\") pod \"maas-api-e2e-worker-w3-624254-576d847589-4jfld\" (UID: \"2e311645-6f4b-4169-8c27-8ad75ac36da2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-624254-576d847589-4jfld"
Nov 02 15:06:04.985320 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:04.985162    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-624254-576d847589-4jfld"
Nov 02 15:06:05.295765 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:05.295669    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-624254-576d847589-4jfld"]
Nov 02 15:06:05.308079 ip-10-0-135-138 kubenswrapper[5304]: W1102 15:06:05.308035    5304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e311645_6f4b_4169_8c27_8ad75ac36da2.slice/crio-d4549008cd70092ec77708be5f081cbead433f2366d0a1f421dac2aa44fc636a WatchSource:0}: Error finding container d4549008cd70092ec77708be5f081cbead433f2366d0a1f421dac2aa44fc636a: Status 404 returned error can't find the container with id d4549008cd70092ec77708be5f081cbead433f2366d0a1f421dac2aa44fc636a
Nov 02 15:06:06.197355 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:06.197317    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-624254-576d847589-4jfld" event={"ID":"2e311645-6f4b-4169-8c27-8ad75ac36da2","Type":"ContainerStarted","Data":"2d5898fc59bae3612aa31c443a2dac401dd48e79dd43ef4c9dc0903bf2dcae78"}
Nov 02 15:06:06.197548 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:06.197376    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-624254-576d847589-4jfld" event={"ID":"2e311645-6f4b-4169-8c27-8ad75ac36da2","Type":"ContainerStarted","Data":"d4549008cd70092ec77708be5f081cbead433f2366d0a1f421dac2aa44fc636a"}
Nov 02 15:06:06.197548 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:06.197403    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-624254-576d847589-4jfld"
Nov 02 15:06:10.197373 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:10.197343    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-5dd8e9-68fff466d7-7dgm5"
Nov 02 15:06:10.226864 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:10.226556    5304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-624254-576d847589-4jfld" podStartSLOduration=5.846739992 podStartE2EDuration="6.226540135s" totalImagesPullingTime="379.800143ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:06:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:06:06.241404028 +0000 UTC m=+566.235228438" watchObservedRunningTime="2026-11-02 15:06:10.226540135 +0000 UTC m=+570.220364544"
Nov 02 15:06:10.228367 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:10.228343    5304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-6af50f-865547d8c5-dvg56"]
Nov 02 15:06:10.257286 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:10.256461    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-6af50f-865547d8c5-dvg56"
Nov 02 15:06:10.277444 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:10.277412    5304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w0-6af50f-serving-cert\""
Nov 02 15:06:10.290182 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:10.290153    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-6af50f-865547d8c5-dvg56"]
Nov 02 15:06:10.406928 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:10.406894    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5t56c\" (UniqueName: \"kubernetes.io/projected/ab7216bc-cf46-4280-acd1-3a15d1ead1c0-kube-api-access-5t56c\") pod \"maas-api-e2e-worker-w0-6af50f-865547d8c5-dvg56\" (UID: \"ab7216bc-cf46-4280-acd1-3a15d1ead1c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-6af50f-865547d8c5-dvg56"
Nov 02 15:06:10.407086 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:10.406943    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab7216bc-cf46-4280-acd1-3a15d1ead1c0-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-6af50f-865547d8c5-dvg56\" (UID: \"ab7216bc-cf46-4280-acd1-3a15d1ead1c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-6af50f-865547d8c5-dvg56"
Nov 02 15:06:10.407086 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:10.407040    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ab7216bc-cf46-4280-acd1-3a15d1ead1c0-maas-api-tls\") pod \"maas-api-e2e-worker-w0-6af50f-865547d8c5-dvg56\" (UID: \"ab7216bc-cf46-4280-acd1-3a15d1ead1c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-6af50f-865547d8c5-dvg56"
Nov 02 15:06:10.407086 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:10.407084    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ab7216bc-cf46-4280-acd1-3a15d1ead1c0-metrics-certs\") pod \"maas-api-e2e-worker-w0-6af50f-865547d8c5-dvg56\" (UID: \"ab7216bc-cf46-4280-acd1-3a15d1ead1c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-6af50f-865547d8c5-dvg56"
Nov 02 15:06:10.508592 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:10.508514    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5t56c\" (UniqueName: \"kubernetes.io/projected/ab7216bc-cf46-4280-acd1-3a15d1ead1c0-kube-api-access-5t56c\") pod \"maas-api-e2e-worker-w0-6af50f-865547d8c5-dvg56\" (UID: \"ab7216bc-cf46-4280-acd1-3a15d1ead1c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-6af50f-865547d8c5-dvg56"
Nov 02 15:06:10.508592 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:10.508571    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab7216bc-cf46-4280-acd1-3a15d1ead1c0-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-6af50f-865547d8c5-dvg56\" (UID: \"ab7216bc-cf46-4280-acd1-3a15d1ead1c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-6af50f-865547d8c5-dvg56"
Nov 02 15:06:10.508768 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:10.508605    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ab7216bc-cf46-4280-acd1-3a15d1ead1c0-maas-api-tls\") pod \"maas-api-e2e-worker-w0-6af50f-865547d8c5-dvg56\" (UID: \"ab7216bc-cf46-4280-acd1-3a15d1ead1c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-6af50f-865547d8c5-dvg56"
Nov 02 15:06:10.508768 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:10.508637    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ab7216bc-cf46-4280-acd1-3a15d1ead1c0-metrics-certs\") pod \"maas-api-e2e-worker-w0-6af50f-865547d8c5-dvg56\" (UID: \"ab7216bc-cf46-4280-acd1-3a15d1ead1c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-6af50f-865547d8c5-dvg56"
Nov 02 15:06:10.509579 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:10.509555    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab7216bc-cf46-4280-acd1-3a15d1ead1c0-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-6af50f-865547d8c5-dvg56\" (UID: \"ab7216bc-cf46-4280-acd1-3a15d1ead1c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-6af50f-865547d8c5-dvg56"
Nov 02 15:06:10.511067 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:10.511047    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ab7216bc-cf46-4280-acd1-3a15d1ead1c0-metrics-certs\") pod \"maas-api-e2e-worker-w0-6af50f-865547d8c5-dvg56\" (UID: \"ab7216bc-cf46-4280-acd1-3a15d1ead1c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-6af50f-865547d8c5-dvg56"
Nov 02 15:06:10.515831 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:10.515808    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ab7216bc-cf46-4280-acd1-3a15d1ead1c0-maas-api-tls\") pod \"maas-api-e2e-worker-w0-6af50f-865547d8c5-dvg56\" (UID: \"ab7216bc-cf46-4280-acd1-3a15d1ead1c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-6af50f-865547d8c5-dvg56"
Nov 02 15:06:10.548161 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:10.544039    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5t56c\" (UniqueName: \"kubernetes.io/projected/ab7216bc-cf46-4280-acd1-3a15d1ead1c0-kube-api-access-5t56c\") pod \"maas-api-e2e-worker-w0-6af50f-865547d8c5-dvg56\" (UID: \"ab7216bc-cf46-4280-acd1-3a15d1ead1c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-6af50f-865547d8c5-dvg56"
Nov 02 15:06:10.604495 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:10.604460    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-6af50f-865547d8c5-dvg56"
Nov 02 15:06:10.963365 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:10.963332    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-6af50f-865547d8c5-dvg56"]
Nov 02 15:06:10.967440 ip-10-0-135-138 kubenswrapper[5304]: W1102 15:06:10.967409    5304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab7216bc_cf46_4280_acd1_3a15d1ead1c0.slice/crio-b7d1e39ea100be76085e97ac54b814764583c82231ab512544018c306af2d345 WatchSource:0}: Error finding container b7d1e39ea100be76085e97ac54b814764583c82231ab512544018c306af2d345: Status 404 returned error can't find the container with id b7d1e39ea100be76085e97ac54b814764583c82231ab512544018c306af2d345
Nov 02 15:06:11.225081 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:11.224995    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-6af50f-865547d8c5-dvg56" event={"ID":"ab7216bc-cf46-4280-acd1-3a15d1ead1c0","Type":"ContainerStarted","Data":"b7d1e39ea100be76085e97ac54b814764583c82231ab512544018c306af2d345"}
Nov 02 15:06:12.210251 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:12.210215    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-624254-576d847589-4jfld"
Nov 02 15:06:12.258245 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:12.258211    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-6af50f-865547d8c5-dvg56" event={"ID":"ab7216bc-cf46-4280-acd1-3a15d1ead1c0","Type":"ContainerStarted","Data":"f5c343e6eb800855b293ef6faec0fcf5f4c6b3b43409fb692d685bae2fffa4e1"}
Nov 02 15:06:12.258765 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:12.258346    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-6af50f-865547d8c5-dvg56"
Nov 02 15:06:12.359289 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:12.348565    5304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-6af50f-865547d8c5-dvg56" podStartSLOduration=1.940049707 podStartE2EDuration="2.348542273s" totalImagesPullingTime="408.492566ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:06:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:06:12.337434547 +0000 UTC m=+572.331258958" watchObservedRunningTime="2026-11-02 15:06:12.348542273 +0000 UTC m=+572.342366682"
Nov 02 15:06:16.302439 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:16.296711    5304 generic.go:356] "Generic (PLEG): container finished" podID="0b48fb40-a77c-4345-a9d4-776bcdabb078" containerID="925c184dcf230cd74bfef28c14af0b95596ad6cd7a419c7948359b1478c56603" exitCode=0
Nov 02 15:06:16.302439 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:16.296866    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-g8tjj" event={"ID":"0b48fb40-a77c-4345-a9d4-776bcdabb078","Type":"ContainerDied","Data":"925c184dcf230cd74bfef28c14af0b95596ad6cd7a419c7948359b1478c56603"}
Nov 02 15:06:16.303391 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:16.303356    5304 scope.go:122] "RemoveContainer" containerID="925c184dcf230cd74bfef28c14af0b95596ad6cd7a419c7948359b1478c56603"
Nov 02 15:06:16.315148 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:16.312200    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:06:16.315148 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:16.312340    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:06:16.315148 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:16.312409    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:06:17.328152 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:17.327846    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-g8tjj" event={"ID":"0b48fb40-a77c-4345-a9d4-776bcdabb078","Type":"ContainerStarted","Data":"8dae976028ac237c8abcbbaf74aab1520943f99f8ed0e7c8875888f531c144c2"}
Nov 02 15:06:17.410151 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:17.408550    5304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-g8tjj"
Nov 02 15:06:17.410151 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:17.408679    5304 patch_prober.go:29] interesting pod/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-g8tjj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body=
Nov 02 15:06:17.410151 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:17.408739    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-g8tjj" podUID="0b48fb40-a77c-4345-a9d4-776bcdabb078" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused"
Nov 02 15:06:18.309417 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:18.309381    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-6af50f-865547d8c5-dvg56"
Nov 02 15:06:18.399588 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:18.399552    5304 patch_prober.go:29] interesting pod/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-g8tjj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body=
Nov 02 15:06:18.446162 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:18.443224    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-g8tjj" podUID="0b48fb40-a77c-4345-a9d4-776bcdabb078" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused"
Nov 02 15:06:19.403152 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:19.401553    5304 patch_prober.go:29] interesting pod/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-g8tjj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body=
Nov 02 15:06:19.403152 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:19.401632    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-g8tjj" podUID="0b48fb40-a77c-4345-a9d4-776bcdabb078" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused"
Nov 02 15:06:20.467149 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:20.448177    5304 patch_prober.go:29] interesting pod/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-g8tjj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body=
Nov 02 15:06:20.467149 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:20.449440    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-g8tjj" podUID="0b48fb40-a77c-4345-a9d4-776bcdabb078" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused"
Nov 02 15:06:20.799379 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:20.799280    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nkbrk" podUID="1f68d837-fff2-418b-84d7-c2d7eb21a57e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.24:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:06:21.403147 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:21.402855    5304 patch_prober.go:29] interesting pod/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-g8tjj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body=
Nov 02 15:06:21.403147 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:21.403081    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-g8tjj" podUID="0b48fb40-a77c-4345-a9d4-776bcdabb078" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused"
Nov 02 15:06:21.869150 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:21.868245    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nkbrk" podUID="1f68d837-fff2-418b-84d7-c2d7eb21a57e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.24:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:06:21.869150 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:21.869020    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nkbrk"
Nov 02 15:06:22.127800 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:22.026872    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nkbrk"
Nov 02 15:06:22.399705 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:22.399251    5304 patch_prober.go:29] interesting pod/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-g8tjj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body=
Nov 02 15:06:22.399705 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:22.399324    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-g8tjj" podUID="0b48fb40-a77c-4345-a9d4-776bcdabb078" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused"
Nov 02 15:06:23.405883 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:23.405445    5304 patch_prober.go:29] interesting pod/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-g8tjj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body=
Nov 02 15:06:23.405883 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:23.405509    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-g8tjj" podUID="0b48fb40-a77c-4345-a9d4-776bcdabb078" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused"
Nov 02 15:06:24.774478 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:24.773221    5304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-g8tjj"
Nov 02 15:06:24.775282 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:24.775176    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-g8tjj"
Nov 02 15:06:24.794001 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:24.793926    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-g8tjj"
Nov 02 15:06:24.863760 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:06:24.863715    5304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k6fjk" containerName="extractor"
Nov 02 15:06:26.203290 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:26.203249    5304 patch_prober.go:29] interesting pod/e2e-worker-w3-624254-openshift-default-7988c9d8b5-cp5ms container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 15:06:26.203805 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:26.203314    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-cp5ms" podUID="11a65ba1-0a55-4478-b043-637c0a9d6d64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:06:27.204368 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:27.204277    5304 patch_prober.go:29] interesting pod/e2e-worker-w0-6af50f-openshift-default-8545fb68f-nrs9j container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 15:06:27.370706 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:27.204365    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-nrs9j" podUID="c1051c4a-02ca-40b8-b8de-180bd2ca598b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:06:27.370706 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:27.204278    5304 patch_prober.go:29] interesting pod/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-pshqj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 15:06:27.370706 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:27.204450    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-pshqj" podUID="89710086-0e78-4395-b8f9-e47cce4f7dfe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:06:27.370706 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:27.204297    5304 patch_prober.go:29] interesting pod/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-ldcsg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 15:06:27.370706 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:27.204498    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-ldcsg" podUID="27105db3-fb1c-4da0-8cf0-5f301ac6967f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:06:27.875398 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:27.875366    5304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-5dd8e9/simulator-w1-5dd8e9-kserve-6f9f755d6f-gnw5d"]
Nov 02 15:06:27.911259 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:27.911222    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-5dd8e9/simulator-w1-5dd8e9-kserve-6f9f755d6f-gnw5d"]
Nov 02 15:06:27.911421 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:27.911357    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-5dd8e9/simulator-w1-5dd8e9-kserve-6f9f755d6f-gnw5d"
Nov 02 15:06:27.956385 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:27.956356    5304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-5dd8e9\"/\"default-dockercfg-ccmk6\""
Nov 02 15:06:27.956607 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:27.956384    5304 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-5dd8e9\"/\"openshift-service-ca.crt\""
Nov 02 15:06:27.956721 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:27.956464    5304 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-5dd8e9\"/\"kube-root-ca.crt\""
Nov 02 15:06:27.956819 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:27.956610    5304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-5dd8e9\"/\"simulator-w1-5dd8e9-kserve-self-signed-certs\""
Nov 02 15:06:27.989046 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:27.989011    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/66b75e64-cb24-40be-9076-4984c132e208-dshm\") pod \"simulator-w1-5dd8e9-kserve-6f9f755d6f-gnw5d\" (UID: \"66b75e64-cb24-40be-9076-4984c132e208\") " pod="e2e-models-e2e-worker-w1-5dd8e9/simulator-w1-5dd8e9-kserve-6f9f755d6f-gnw5d"
Nov 02 15:06:27.989260 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:27.989157    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/66b75e64-cb24-40be-9076-4984c132e208-model-cache\") pod \"simulator-w1-5dd8e9-kserve-6f9f755d6f-gnw5d\" (UID: \"66b75e64-cb24-40be-9076-4984c132e208\") " pod="e2e-models-e2e-worker-w1-5dd8e9/simulator-w1-5dd8e9-kserve-6f9f755d6f-gnw5d"
Nov 02 15:06:27.989260 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:27.989229    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/66b75e64-cb24-40be-9076-4984c132e208-home\") pod \"simulator-w1-5dd8e9-kserve-6f9f755d6f-gnw5d\" (UID: \"66b75e64-cb24-40be-9076-4984c132e208\") " pod="e2e-models-e2e-worker-w1-5dd8e9/simulator-w1-5dd8e9-kserve-6f9f755d6f-gnw5d"
Nov 02 15:06:27.989409 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:27.989294    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-br92q\" (UniqueName: \"kubernetes.io/projected/66b75e64-cb24-40be-9076-4984c132e208-kube-api-access-br92q\") pod \"simulator-w1-5dd8e9-kserve-6f9f755d6f-gnw5d\" (UID: \"66b75e64-cb24-40be-9076-4984c132e208\") " pod="e2e-models-e2e-worker-w1-5dd8e9/simulator-w1-5dd8e9-kserve-6f9f755d6f-gnw5d"
Nov 02 15:06:27.989409 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:27.989350    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/66b75e64-cb24-40be-9076-4984c132e208-tmp-dir\") pod \"simulator-w1-5dd8e9-kserve-6f9f755d6f-gnw5d\" (UID: \"66b75e64-cb24-40be-9076-4984c132e208\") " pod="e2e-models-e2e-worker-w1-5dd8e9/simulator-w1-5dd8e9-kserve-6f9f755d6f-gnw5d"
Nov 02 15:06:27.989495 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:27.989413    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/66b75e64-cb24-40be-9076-4984c132e208-tls-certs\") pod \"simulator-w1-5dd8e9-kserve-6f9f755d6f-gnw5d\" (UID: \"66b75e64-cb24-40be-9076-4984c132e208\") " pod="e2e-models-e2e-worker-w1-5dd8e9/simulator-w1-5dd8e9-kserve-6f9f755d6f-gnw5d"
Nov 02 15:06:28.090951 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:28.090358    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/66b75e64-cb24-40be-9076-4984c132e208-dshm\") pod \"simulator-w1-5dd8e9-kserve-6f9f755d6f-gnw5d\" (UID: \"66b75e64-cb24-40be-9076-4984c132e208\") " pod="e2e-models-e2e-worker-w1-5dd8e9/simulator-w1-5dd8e9-kserve-6f9f755d6f-gnw5d"
Nov 02 15:06:28.090951 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:28.090393    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/66b75e64-cb24-40be-9076-4984c132e208-model-cache\") pod \"simulator-w1-5dd8e9-kserve-6f9f755d6f-gnw5d\" (UID: \"66b75e64-cb24-40be-9076-4984c132e208\") " pod="e2e-models-e2e-worker-w1-5dd8e9/simulator-w1-5dd8e9-kserve-6f9f755d6f-gnw5d"
Nov 02 15:06:28.090951 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:28.090436    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/66b75e64-cb24-40be-9076-4984c132e208-home\") pod \"simulator-w1-5dd8e9-kserve-6f9f755d6f-gnw5d\" (UID: \"66b75e64-cb24-40be-9076-4984c132e208\") " pod="e2e-models-e2e-worker-w1-5dd8e9/simulator-w1-5dd8e9-kserve-6f9f755d6f-gnw5d"
Nov 02 15:06:28.090951 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:28.090475    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-br92q\" (UniqueName: \"kubernetes.io/projected/66b75e64-cb24-40be-9076-4984c132e208-kube-api-access-br92q\") pod \"simulator-w1-5dd8e9-kserve-6f9f755d6f-gnw5d\" (UID: \"66b75e64-cb24-40be-9076-4984c132e208\") " pod="e2e-models-e2e-worker-w1-5dd8e9/simulator-w1-5dd8e9-kserve-6f9f755d6f-gnw5d"
Nov 02 15:06:28.090951 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:28.090502    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/66b75e64-cb24-40be-9076-4984c132e208-tmp-dir\") pod \"simulator-w1-5dd8e9-kserve-6f9f755d6f-gnw5d\" (UID: \"66b75e64-cb24-40be-9076-4984c132e208\") " pod="e2e-models-e2e-worker-w1-5dd8e9/simulator-w1-5dd8e9-kserve-6f9f755d6f-gnw5d"
Nov 02 15:06:28.090951 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:28.090544    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/66b75e64-cb24-40be-9076-4984c132e208-tls-certs\") pod \"simulator-w1-5dd8e9-kserve-6f9f755d6f-gnw5d\" (UID: \"66b75e64-cb24-40be-9076-4984c132e208\") " pod="e2e-models-e2e-worker-w1-5dd8e9/simulator-w1-5dd8e9-kserve-6f9f755d6f-gnw5d"
Nov 02 15:06:28.091395 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:28.091155    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/66b75e64-cb24-40be-9076-4984c132e208-home\") pod \"simulator-w1-5dd8e9-kserve-6f9f755d6f-gnw5d\" (UID: \"66b75e64-cb24-40be-9076-4984c132e208\") " pod="e2e-models-e2e-worker-w1-5dd8e9/simulator-w1-5dd8e9-kserve-6f9f755d6f-gnw5d"
Nov 02 15:06:28.091500 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:28.091448    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/66b75e64-cb24-40be-9076-4984c132e208-model-cache\") pod \"simulator-w1-5dd8e9-kserve-6f9f755d6f-gnw5d\" (UID: \"66b75e64-cb24-40be-9076-4984c132e208\") " pod="e2e-models-e2e-worker-w1-5dd8e9/simulator-w1-5dd8e9-kserve-6f9f755d6f-gnw5d"
Nov 02 15:06:28.091576 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:28.091557    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/66b75e64-cb24-40be-9076-4984c132e208-tmp-dir\") pod \"simulator-w1-5dd8e9-kserve-6f9f755d6f-gnw5d\" (UID: \"66b75e64-cb24-40be-9076-4984c132e208\") " pod="e2e-models-e2e-worker-w1-5dd8e9/simulator-w1-5dd8e9-kserve-6f9f755d6f-gnw5d"
Nov 02 15:06:28.093277 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:28.093239    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/66b75e64-cb24-40be-9076-4984c132e208-tls-certs\") pod \"simulator-w1-5dd8e9-kserve-6f9f755d6f-gnw5d\" (UID: \"66b75e64-cb24-40be-9076-4984c132e208\") " pod="e2e-models-e2e-worker-w1-5dd8e9/simulator-w1-5dd8e9-kserve-6f9f755d6f-gnw5d"
Nov 02 15:06:28.093459 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:28.093442    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/66b75e64-cb24-40be-9076-4984c132e208-dshm\") pod \"simulator-w1-5dd8e9-kserve-6f9f755d6f-gnw5d\" (UID: \"66b75e64-cb24-40be-9076-4984c132e208\") " pod="e2e-models-e2e-worker-w1-5dd8e9/simulator-w1-5dd8e9-kserve-6f9f755d6f-gnw5d"
Nov 02 15:06:28.116488 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:28.115756    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-br92q\" (UniqueName: \"kubernetes.io/projected/66b75e64-cb24-40be-9076-4984c132e208-kube-api-access-br92q\") pod \"simulator-w1-5dd8e9-kserve-6f9f755d6f-gnw5d\" (UID: \"66b75e64-cb24-40be-9076-4984c132e208\") " pod="e2e-models-e2e-worker-w1-5dd8e9/simulator-w1-5dd8e9-kserve-6f9f755d6f-gnw5d"
Nov 02 15:06:28.256520 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:28.256481    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-5dd8e9/simulator-w1-5dd8e9-kserve-6f9f755d6f-gnw5d"
Nov 02 15:06:28.696637 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:28.647159    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-5dd8e9/simulator-w1-5dd8e9-kserve-6f9f755d6f-gnw5d"]
Nov 02 15:06:28.696637 ip-10-0-135-138 kubenswrapper[5304]: W1102 15:06:28.658819    5304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod66b75e64_cb24_40be_9076_4984c132e208.slice/crio-27ac563a147095630ec60ec88f6fc5d9de9ef241d07a24c9812e1df4ed350cba WatchSource:0}: Error finding container 27ac563a147095630ec60ec88f6fc5d9de9ef241d07a24c9812e1df4ed350cba: Status 404 returned error can't find the container with id 27ac563a147095630ec60ec88f6fc5d9de9ef241d07a24c9812e1df4ed350cba
Nov 02 15:06:29.535639 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:29.535558    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-5dd8e9/simulator-w1-5dd8e9-kserve-6f9f755d6f-gnw5d" event={"ID":"66b75e64-cb24-40be-9076-4984c132e208","Type":"ContainerStarted","Data":"b4b6aedcdb640eb49cd010d0c3dbddcb760257857b2386f94572d9a21537ea2c"}
Nov 02 15:06:29.562679 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:29.536090    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-5dd8e9/simulator-w1-5dd8e9-kserve-6f9f755d6f-gnw5d" event={"ID":"66b75e64-cb24-40be-9076-4984c132e208","Type":"ContainerStarted","Data":"27ac563a147095630ec60ec88f6fc5d9de9ef241d07a24c9812e1df4ed350cba"}
Nov 02 15:06:29.562679 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:29.537328    5304 generic.go:356] "Generic (PLEG): container finished" podID="11a65ba1-0a55-4478-b043-637c0a9d6d64" containerID="66e57b8107829805af57f0382cb5ffde3b62067aae95273892bb44947fb1304d" exitCode=0
Nov 02 15:06:29.562679 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:29.537394    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-cp5ms" event={"ID":"11a65ba1-0a55-4478-b043-637c0a9d6d64","Type":"ContainerDied","Data":"66e57b8107829805af57f0382cb5ffde3b62067aae95273892bb44947fb1304d"}
Nov 02 15:06:29.562679 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:29.537766    5304 scope.go:122] "RemoveContainer" containerID="66e57b8107829805af57f0382cb5ffde3b62067aae95273892bb44947fb1304d"
Nov 02 15:06:29.562679 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:29.539023    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:06:29.562679 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:29.539089    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:06:29.562679 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:29.539138    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:06:29.583058 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:29.582247    5304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-5dd8e9/simulator-w1-5dd8e9-kserve-6f9f755d6f-gnw5d" podStartSLOduration=2.582228761 podStartE2EDuration="2.582228761s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:06:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:06:29.581670263 +0000 UTC m=+589.575494667" watchObservedRunningTime="2026-11-02 15:06:29.582228761 +0000 UTC m=+589.576053171"
Nov 02 15:06:30.256972 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:30.256941    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5dd8e9/simulator-w1-5dd8e9-kserve-6f9f755d6f-gnw5d"
Nov 02 15:06:30.545687 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:30.544674    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-cp5ms" event={"ID":"11a65ba1-0a55-4478-b043-637c0a9d6d64","Type":"ContainerStarted","Data":"f6c95516a630ebb1446e90f41aa47193036cdcdc56b49b12fa665ec5574f07af"}
Nov 02 15:06:30.548870 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:30.548052    5304 generic.go:356] "Generic (PLEG): container finished" podID="89710086-0e78-4395-b8f9-e47cce4f7dfe" containerID="6dd484c90f59525171dfb12122eb7a12a797fd5f1bf98e31a7ef3f7c6170893b" exitCode=0
Nov 02 15:06:30.548870 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:30.548140    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-pshqj" event={"ID":"89710086-0e78-4395-b8f9-e47cce4f7dfe","Type":"ContainerDied","Data":"6dd484c90f59525171dfb12122eb7a12a797fd5f1bf98e31a7ef3f7c6170893b"}
Nov 02 15:06:30.548870 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:30.548527    5304 scope.go:122] "RemoveContainer" containerID="6dd484c90f59525171dfb12122eb7a12a797fd5f1bf98e31a7ef3f7c6170893b"
Nov 02 15:06:30.551310 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:30.550319    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:06:30.551310 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:30.550413    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:06:30.551310 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:30.550478    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:06:30.555477 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:30.555105    5304 generic.go:356] "Generic (PLEG): container finished" podID="27105db3-fb1c-4da0-8cf0-5f301ac6967f" containerID="a942baeee15555237950f168b16307153d6ecf18f87153f75833ffd30e255eb1" exitCode=0
Nov 02 15:06:30.555477 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:30.555168    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-ldcsg" event={"ID":"27105db3-fb1c-4da0-8cf0-5f301ac6967f","Type":"ContainerDied","Data":"a942baeee15555237950f168b16307153d6ecf18f87153f75833ffd30e255eb1"}
Nov 02 15:06:30.555995 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:30.555879    5304 scope.go:122] "RemoveContainer" containerID="a942baeee15555237950f168b16307153d6ecf18f87153f75833ffd30e255eb1"
Nov 02 15:06:30.557291 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:30.557258    5304 generic.go:356] "Generic (PLEG): container finished" podID="c1051c4a-02ca-40b8-b8de-180bd2ca598b" containerID="9fafab017e9dff6e5bce71d3792068134589f9b46bfe53dbbefbeaefabbdb83c" exitCode=0
Nov 02 15:06:30.557942 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:30.557920    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-nrs9j" event={"ID":"c1051c4a-02ca-40b8-b8de-180bd2ca598b","Type":"ContainerDied","Data":"9fafab017e9dff6e5bce71d3792068134589f9b46bfe53dbbefbeaefabbdb83c"}
Nov 02 15:06:30.558281 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:30.558254    5304 scope.go:122] "RemoveContainer" containerID="9fafab017e9dff6e5bce71d3792068134589f9b46bfe53dbbefbeaefabbdb83c"
Nov 02 15:06:30.568240 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:30.561642    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:06:30.568240 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:30.561731    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:06:30.568240 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:30.561778    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:06:30.568240 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:30.561895    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:06:30.568240 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:30.561989    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:06:30.568240 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:30.562038    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:06:30.761418 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:30.761032    5304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-624254/simulator-w3-624254-kserve-56866b7699-z47sd"]
Nov 02 15:06:30.803661 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:30.801291    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-624254/simulator-w3-624254-kserve-56866b7699-z47sd"
Nov 02 15:06:30.808089 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:30.808042    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-624254/simulator-w3-624254-kserve-56866b7699-z47sd"]
Nov 02 15:06:30.821697 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:30.821671    5304 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-624254\"/\"openshift-service-ca.crt\""
Nov 02 15:06:30.880433 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:30.880384    5304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-624254\"/\"simulator-w3-624254-kserve-self-signed-certs\""
Nov 02 15:06:30.890208 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:30.890172    5304 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-624254\"/\"kube-root-ca.crt\""
Nov 02 15:06:30.891156 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:30.891114    5304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-624254\"/\"default-dockercfg-4m4cf\""
Nov 02 15:06:30.922268 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:30.922232    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/345c710f-b414-4e62-86c7-d51ff4e5be15-model-cache\") pod \"simulator-w3-624254-kserve-56866b7699-z47sd\" (UID: \"345c710f-b414-4e62-86c7-d51ff4e5be15\") " pod="e2e-models-e2e-worker-w3-624254/simulator-w3-624254-kserve-56866b7699-z47sd"
Nov 02 15:06:30.922459 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:30.922278    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/345c710f-b414-4e62-86c7-d51ff4e5be15-dshm\") pod \"simulator-w3-624254-kserve-56866b7699-z47sd\" (UID: \"345c710f-b414-4e62-86c7-d51ff4e5be15\") " pod="e2e-models-e2e-worker-w3-624254/simulator-w3-624254-kserve-56866b7699-z47sd"
Nov 02 15:06:30.922459 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:30.922331    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/345c710f-b414-4e62-86c7-d51ff4e5be15-tls-certs\") pod \"simulator-w3-624254-kserve-56866b7699-z47sd\" (UID: \"345c710f-b414-4e62-86c7-d51ff4e5be15\") " pod="e2e-models-e2e-worker-w3-624254/simulator-w3-624254-kserve-56866b7699-z47sd"
Nov 02 15:06:30.922459 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:30.922350    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5rl6k\" (UniqueName: \"kubernetes.io/projected/345c710f-b414-4e62-86c7-d51ff4e5be15-kube-api-access-5rl6k\") pod \"simulator-w3-624254-kserve-56866b7699-z47sd\" (UID: \"345c710f-b414-4e62-86c7-d51ff4e5be15\") " pod="e2e-models-e2e-worker-w3-624254/simulator-w3-624254-kserve-56866b7699-z47sd"
Nov 02 15:06:30.922459 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:30.922400    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/345c710f-b414-4e62-86c7-d51ff4e5be15-tmp-dir\") pod \"simulator-w3-624254-kserve-56866b7699-z47sd\" (UID: \"345c710f-b414-4e62-86c7-d51ff4e5be15\") " pod="e2e-models-e2e-worker-w3-624254/simulator-w3-624254-kserve-56866b7699-z47sd"
Nov 02 15:06:30.922459 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:30.922429    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/345c710f-b414-4e62-86c7-d51ff4e5be15-home\") pod \"simulator-w3-624254-kserve-56866b7699-z47sd\" (UID: \"345c710f-b414-4e62-86c7-d51ff4e5be15\") " pod="e2e-models-e2e-worker-w3-624254/simulator-w3-624254-kserve-56866b7699-z47sd"
Nov 02 15:06:31.023304 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:31.023257    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/345c710f-b414-4e62-86c7-d51ff4e5be15-model-cache\") pod \"simulator-w3-624254-kserve-56866b7699-z47sd\" (UID: \"345c710f-b414-4e62-86c7-d51ff4e5be15\") " pod="e2e-models-e2e-worker-w3-624254/simulator-w3-624254-kserve-56866b7699-z47sd"
Nov 02 15:06:31.024110 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:31.024086    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/345c710f-b414-4e62-86c7-d51ff4e5be15-dshm\") pod \"simulator-w3-624254-kserve-56866b7699-z47sd\" (UID: \"345c710f-b414-4e62-86c7-d51ff4e5be15\") " pod="e2e-models-e2e-worker-w3-624254/simulator-w3-624254-kserve-56866b7699-z47sd"
Nov 02 15:06:31.024404 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:31.024378    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/345c710f-b414-4e62-86c7-d51ff4e5be15-tls-certs\") pod \"simulator-w3-624254-kserve-56866b7699-z47sd\" (UID: \"345c710f-b414-4e62-86c7-d51ff4e5be15\") " pod="e2e-models-e2e-worker-w3-624254/simulator-w3-624254-kserve-56866b7699-z47sd"
Nov 02 15:06:31.024961 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:31.024930    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5rl6k\" (UniqueName: \"kubernetes.io/projected/345c710f-b414-4e62-86c7-d51ff4e5be15-kube-api-access-5rl6k\") pod \"simulator-w3-624254-kserve-56866b7699-z47sd\" (UID: \"345c710f-b414-4e62-86c7-d51ff4e5be15\") " pod="e2e-models-e2e-worker-w3-624254/simulator-w3-624254-kserve-56866b7699-z47sd"
Nov 02 15:06:31.025578 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:31.023972    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/345c710f-b414-4e62-86c7-d51ff4e5be15-model-cache\") pod \"simulator-w3-624254-kserve-56866b7699-z47sd\" (UID: \"345c710f-b414-4e62-86c7-d51ff4e5be15\") " pod="e2e-models-e2e-worker-w3-624254/simulator-w3-624254-kserve-56866b7699-z47sd"
Nov 02 15:06:31.025810 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:31.025791    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/345c710f-b414-4e62-86c7-d51ff4e5be15-tmp-dir\") pod \"simulator-w3-624254-kserve-56866b7699-z47sd\" (UID: \"345c710f-b414-4e62-86c7-d51ff4e5be15\") " pod="e2e-models-e2e-worker-w3-624254/simulator-w3-624254-kserve-56866b7699-z47sd"
Nov 02 15:06:31.026320 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:31.026202    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/345c710f-b414-4e62-86c7-d51ff4e5be15-tmp-dir\") pod \"simulator-w3-624254-kserve-56866b7699-z47sd\" (UID: \"345c710f-b414-4e62-86c7-d51ff4e5be15\") " pod="e2e-models-e2e-worker-w3-624254/simulator-w3-624254-kserve-56866b7699-z47sd"
Nov 02 15:06:31.026468 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:31.026452    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/345c710f-b414-4e62-86c7-d51ff4e5be15-home\") pod \"simulator-w3-624254-kserve-56866b7699-z47sd\" (UID: \"345c710f-b414-4e62-86c7-d51ff4e5be15\") " pod="e2e-models-e2e-worker-w3-624254/simulator-w3-624254-kserve-56866b7699-z47sd"
Nov 02 15:06:31.027667 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:31.026986    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/345c710f-b414-4e62-86c7-d51ff4e5be15-home\") pod \"simulator-w3-624254-kserve-56866b7699-z47sd\" (UID: \"345c710f-b414-4e62-86c7-d51ff4e5be15\") " pod="e2e-models-e2e-worker-w3-624254/simulator-w3-624254-kserve-56866b7699-z47sd"
Nov 02 15:06:31.027667 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:31.027452    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/345c710f-b414-4e62-86c7-d51ff4e5be15-dshm\") pod \"simulator-w3-624254-kserve-56866b7699-z47sd\" (UID: \"345c710f-b414-4e62-86c7-d51ff4e5be15\") " pod="e2e-models-e2e-worker-w3-624254/simulator-w3-624254-kserve-56866b7699-z47sd"
Nov 02 15:06:31.028192 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:31.028145    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/345c710f-b414-4e62-86c7-d51ff4e5be15-tls-certs\") pod \"simulator-w3-624254-kserve-56866b7699-z47sd\" (UID: \"345c710f-b414-4e62-86c7-d51ff4e5be15\") " pod="e2e-models-e2e-worker-w3-624254/simulator-w3-624254-kserve-56866b7699-z47sd"
Nov 02 15:06:31.074949 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:31.074857    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5rl6k\" (UniqueName: \"kubernetes.io/projected/345c710f-b414-4e62-86c7-d51ff4e5be15-kube-api-access-5rl6k\") pod \"simulator-w3-624254-kserve-56866b7699-z47sd\" (UID: \"345c710f-b414-4e62-86c7-d51ff4e5be15\") " pod="e2e-models-e2e-worker-w3-624254/simulator-w3-624254-kserve-56866b7699-z47sd"
Nov 02 15:06:31.175606 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:31.175568    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-624254/simulator-w3-624254-kserve-56866b7699-z47sd"
Nov 02 15:06:31.249967 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:31.249533    5304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-cp5ms"
Nov 02 15:06:31.249967 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:31.249663    5304 patch_prober.go:29] interesting pod/e2e-worker-w3-624254-openshift-default-7988c9d8b5-cp5ms container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body=
Nov 02 15:06:31.249967 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:31.249706    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-cp5ms" podUID="11a65ba1-0a55-4478-b043-637c0a9d6d64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused"
Nov 02 15:06:31.812015 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:31.806500    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-pshqj" event={"ID":"89710086-0e78-4395-b8f9-e47cce4f7dfe","Type":"ContainerStarted","Data":"7141638a35a6e8d0eeea3c7ac8e21defc3c8db61481fb5e2a002c417397cc317"}
Nov 02 15:06:31.838152 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:31.829252    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-624254/simulator-w3-624254-kserve-56866b7699-z47sd"]
Nov 02 15:06:31.905163 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:31.890790    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-ldcsg" event={"ID":"27105db3-fb1c-4da0-8cf0-5f301ac6967f","Type":"ContainerStarted","Data":"e216b1873ddeba69a9821db8f47b65bc4c0a00632d1182c921c472fa261d9a4a"}
Nov 02 15:06:31.911193 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:31.910808    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-nrs9j" event={"ID":"c1051c4a-02ca-40b8-b8de-180bd2ca598b","Type":"ContainerStarted","Data":"47952bf72c2463b71b7b3c41deda3dfc9230b87048e3357cd2c0eae2fd88a945"}
Nov 02 15:06:31.934955 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:31.934165    5304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-nrs9j"
Nov 02 15:06:31.945331 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:31.944989    5304 patch_prober.go:29] interesting pod/e2e-worker-w0-6af50f-openshift-default-8545fb68f-nrs9j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body=
Nov 02 15:06:31.945331 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:31.945047    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-nrs9j" podUID="c1051c4a-02ca-40b8-b8de-180bd2ca598b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused"
Nov 02 15:06:31.946335 ip-10-0-135-138 kubenswrapper[5304]: W1102 15:06:31.946299    5304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod345c710f_b414_4e62_86c7_d51ff4e5be15.slice/crio-7e0d992ea9e5c666ebacb9dea2ee81a51e686dc31e30fed251c58d7a190cad00 WatchSource:0}: Error finding container 7e0d992ea9e5c666ebacb9dea2ee81a51e686dc31e30fed251c58d7a190cad00: Status 404 returned error can't find the container with id 7e0d992ea9e5c666ebacb9dea2ee81a51e686dc31e30fed251c58d7a190cad00
Nov 02 15:06:32.260152 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:32.258862    5304 patch_prober.go:29] interesting pod/e2e-worker-w3-624254-openshift-default-7988c9d8b5-cp5ms container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body=
Nov 02 15:06:32.260152 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:32.258925    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-cp5ms" podUID="11a65ba1-0a55-4478-b043-637c0a9d6d64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused"
Nov 02 15:06:32.543212 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:32.539618    5304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-ldcsg"
Nov 02 15:06:32.567150 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:32.566380    5304 patch_prober.go:29] interesting pod/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-ldcsg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body=
Nov 02 15:06:32.567150 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:32.566453    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-ldcsg" podUID="27105db3-fb1c-4da0-8cf0-5f301ac6967f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused"
Nov 02 15:06:32.721151 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:32.720272    5304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-pshqj"
Nov 02 15:06:32.721151 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:32.720394    5304 patch_prober.go:29] interesting pod/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-pshqj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body=
Nov 02 15:06:32.721151 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:32.720447    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-pshqj" podUID="89710086-0e78-4395-b8f9-e47cce4f7dfe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused"
Nov 02 15:06:32.929389 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:32.929309    5304 patch_prober.go:29] interesting pod/e2e-worker-w0-6af50f-openshift-default-8545fb68f-nrs9j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body=
Nov 02 15:06:32.929389 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:32.929377    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-nrs9j" podUID="c1051c4a-02ca-40b8-b8de-180bd2ca598b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused"
Nov 02 15:06:32.951152 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:32.932554    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-624254/simulator-w3-624254-kserve-56866b7699-z47sd" event={"ID":"345c710f-b414-4e62-86c7-d51ff4e5be15","Type":"ContainerStarted","Data":"7e0d992ea9e5c666ebacb9dea2ee81a51e686dc31e30fed251c58d7a190cad00"}
Nov 02 15:06:33.250065 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:33.249548    5304 patch_prober.go:29] interesting pod/e2e-worker-w3-624254-openshift-default-7988c9d8b5-cp5ms container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body=
Nov 02 15:06:33.250065 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:33.249613    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-cp5ms" podUID="11a65ba1-0a55-4478-b043-637c0a9d6d64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused"
Nov 02 15:06:33.557203 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:33.543779    5304 patch_prober.go:29] interesting pod/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-ldcsg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body=
Nov 02 15:06:33.557203 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:33.543865    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-ldcsg" podUID="27105db3-fb1c-4da0-8cf0-5f301ac6967f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused"
Nov 02 15:06:33.727151 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:33.721612    5304 patch_prober.go:29] interesting pod/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-pshqj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body=
Nov 02 15:06:33.727151 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:33.721691    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-pshqj" podUID="89710086-0e78-4395-b8f9-e47cce4f7dfe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused"
Nov 02 15:06:33.930158 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:33.929567    5304 patch_prober.go:29] interesting pod/e2e-worker-w0-6af50f-openshift-default-8545fb68f-nrs9j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body=
Nov 02 15:06:33.930158 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:33.929634    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-nrs9j" podUID="c1051c4a-02ca-40b8-b8de-180bd2ca598b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused"
Nov 02 15:06:33.952324 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:33.952285    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-624254/simulator-w3-624254-kserve-56866b7699-z47sd" event={"ID":"345c710f-b414-4e62-86c7-d51ff4e5be15","Type":"ContainerStarted","Data":"9f9134166e5dd2fc996adf28f0f4cf600048bbae39ae4cc4cf8e9a0a8d7bd87f"}
Nov 02 15:06:34.040216 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:34.040138    5304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-624254/simulator-w3-624254-kserve-56866b7699-z47sd" podStartSLOduration=4.04009275 podStartE2EDuration="4.04009275s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:06:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:06:34.023780332 +0000 UTC m=+594.017604741" watchObservedRunningTime="2026-11-02 15:06:34.04009275 +0000 UTC m=+594.033917161"
Nov 02 15:06:34.082921 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:34.082876    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nkbrk" podUID="1f68d837-fff2-418b-84d7-c2d7eb21a57e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.24:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:06:34.205253 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:34.198400    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-624254/simulator-w3-624254-kserve-56866b7699-z47sd"
Nov 02 15:06:34.253150 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:34.252615    5304 patch_prober.go:29] interesting pod/e2e-worker-w3-624254-openshift-default-7988c9d8b5-cp5ms container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body=
Nov 02 15:06:34.253150 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:34.252677    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-cp5ms" podUID="11a65ba1-0a55-4478-b043-637c0a9d6d64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused"
Nov 02 15:06:34.546208 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:34.545568    5304 patch_prober.go:29] interesting pod/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-ldcsg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body=
Nov 02 15:06:34.546208 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:34.545633    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-ldcsg" podUID="27105db3-fb1c-4da0-8cf0-5f301ac6967f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused"
Nov 02 15:06:34.722197 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:34.721778    5304 patch_prober.go:29] interesting pod/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-pshqj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body=
Nov 02 15:06:34.722197 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:34.721847    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-pshqj" podUID="89710086-0e78-4395-b8f9-e47cce4f7dfe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused"
Nov 02 15:06:34.928285 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:34.927827    5304 patch_prober.go:29] interesting pod/e2e-worker-w0-6af50f-openshift-default-8545fb68f-nrs9j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body=
Nov 02 15:06:34.928285 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:34.927890    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-nrs9j" podUID="c1051c4a-02ca-40b8-b8de-180bd2ca598b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused"
Nov 02 15:06:35.249026 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:35.248513    5304 patch_prober.go:29] interesting pod/e2e-worker-w3-624254-openshift-default-7988c9d8b5-cp5ms container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body=
Nov 02 15:06:35.249026 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:35.248574    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-cp5ms" podUID="11a65ba1-0a55-4478-b043-637c0a9d6d64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused"
Nov 02 15:06:35.543207 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:35.538558    5304 patch_prober.go:29] interesting pod/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-ldcsg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body=
Nov 02 15:06:35.543207 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:35.538634    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-ldcsg" podUID="27105db3-fb1c-4da0-8cf0-5f301ac6967f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused"
Nov 02 15:06:35.722593 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:35.722547    5304 patch_prober.go:29] interesting pod/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-pshqj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body=
Nov 02 15:06:35.722787 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:35.722617    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-pshqj" podUID="89710086-0e78-4395-b8f9-e47cce4f7dfe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused"
Nov 02 15:06:35.927924 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:35.927887    5304 patch_prober.go:29] interesting pod/e2e-worker-w0-6af50f-openshift-default-8545fb68f-nrs9j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body=
Nov 02 15:06:35.928250 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:35.928216    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-nrs9j" podUID="c1051c4a-02ca-40b8-b8de-180bd2ca598b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused"
Nov 02 15:06:36.248629 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:36.248532    5304 patch_prober.go:29] interesting pod/e2e-worker-w3-624254-openshift-default-7988c9d8b5-cp5ms container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body=
Nov 02 15:06:36.248903 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:36.248874    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-cp5ms" podUID="11a65ba1-0a55-4478-b043-637c0a9d6d64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused"
Nov 02 15:06:36.538655 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:36.538561    5304 patch_prober.go:29] interesting pod/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-ldcsg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body=
Nov 02 15:06:36.539260 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:36.539227    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-ldcsg" podUID="27105db3-fb1c-4da0-8cf0-5f301ac6967f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused"
Nov 02 15:06:36.729296 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:36.729260    5304 patch_prober.go:29] interesting pod/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-pshqj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body=
Nov 02 15:06:36.729587 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:36.729559    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-pshqj" podUID="89710086-0e78-4395-b8f9-e47cce4f7dfe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused"
Nov 02 15:06:36.930957 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:36.930505    5304 patch_prober.go:29] interesting pod/e2e-worker-w0-6af50f-openshift-default-8545fb68f-nrs9j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body=
Nov 02 15:06:36.930957 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:36.930569    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-nrs9j" podUID="c1051c4a-02ca-40b8-b8de-180bd2ca598b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused"
Nov 02 15:06:37.539523 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:37.538521    5304 patch_prober.go:29] interesting pod/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-ldcsg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body=
Nov 02 15:06:37.539523 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:37.538586    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-ldcsg" podUID="27105db3-fb1c-4da0-8cf0-5f301ac6967f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused"
Nov 02 15:06:37.550290 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:37.548701    5304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-cp5ms"
Nov 02 15:06:37.550290 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:37.550081    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-cp5ms"
Nov 02 15:06:37.553594 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:37.553524    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-cp5ms"
Nov 02 15:06:37.720498 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:37.720456    5304 patch_prober.go:29] interesting pod/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-pshqj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body=
Nov 02 15:06:37.720684 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:37.720531    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-pshqj" podUID="89710086-0e78-4395-b8f9-e47cce4f7dfe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused"
Nov 02 15:06:38.021701 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:38.021655    5304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-nrs9j"
Nov 02 15:06:38.022692 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:38.022549    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-nrs9j"
Nov 02 15:06:38.024373 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:38.024350    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-nrs9j"
Nov 02 15:06:38.257417 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:38.257377    5304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-5dd8e9/simulator-w1-5dd8e9-kserve-6f9f755d6f-gnw5d"
Nov 02 15:06:38.276450 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:38.276339    5304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-5dd8e9/simulator-w1-5dd8e9-kserve-6f9f755d6f-gnw5d"
Nov 02 15:06:38.724846 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:38.724813    5304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-pshqj"
Nov 02 15:06:38.725259 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:38.725145    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-pshqj"
Nov 02 15:06:38.726015 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:38.725993    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-pshqj"
Nov 02 15:06:38.794043 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:38.794011    5304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-ldcsg"
Nov 02 15:06:38.794365 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:38.794340    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-ldcsg"
Nov 02 15:06:38.795505 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:38.795482    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-ldcsg"
Nov 02 15:06:39.010466 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:39.010384    5304 generic.go:356] "Generic (PLEG): container finished" podID="3ed68512-6850-45d4-af0d-a941c2bf6ec9" containerID="eecc968f16d732bb7fb9155081d87e858ec89a885a7242f9502bda03fad23c51" exitCode=0
Nov 02 15:06:39.010651 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:39.010461    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-25tqv" event={"ID":"3ed68512-6850-45d4-af0d-a941c2bf6ec9","Type":"ContainerDied","Data":"eecc968f16d732bb7fb9155081d87e858ec89a885a7242f9502bda03fad23c51"}
Nov 02 15:06:39.011568 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:39.011546    5304 scope.go:122] "RemoveContainer" containerID="eecc968f16d732bb7fb9155081d87e858ec89a885a7242f9502bda03fad23c51"
Nov 02 15:06:39.012774 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:39.012743    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:06:39.012835 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:39.012812    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:06:39.012875 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:39.012848    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:06:39.023508 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:39.023485    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5dd8e9/simulator-w1-5dd8e9-kserve-6f9f755d6f-gnw5d"
Nov 02 15:06:40.021793 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:40.021761    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-25tqv" event={"ID":"3ed68512-6850-45d4-af0d-a941c2bf6ec9","Type":"ContainerStarted","Data":"1442a2c5a420e08de2915db274b4fc9b201da678242c7a4d4e215143733d6a93"}
Nov 02 15:06:40.063598 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:40.063562    5304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-25tqv"
Nov 02 15:06:40.064926 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:40.064902    5304 patch_prober.go:29] interesting pod/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-25tqv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body=
Nov 02 15:06:40.064999 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:40.064962    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-25tqv" podUID="3ed68512-6850-45d4-af0d-a941c2bf6ec9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused"
Nov 02 15:06:40.163824 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:40.162819    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-25tqv"
Nov 02 15:06:41.063062 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:41.063022    5304 patch_prober.go:29] interesting pod/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-25tqv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body=
Nov 02 15:06:41.063481 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:41.063077    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-25tqv" podUID="3ed68512-6850-45d4-af0d-a941c2bf6ec9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused"
Nov 02 15:06:41.176389 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:41.176354    5304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-624254/simulator-w3-624254-kserve-56866b7699-z47sd"
Nov 02 15:06:41.188859 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:41.188831    5304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-624254/simulator-w3-624254-kserve-56866b7699-z47sd"
Nov 02 15:06:42.062819 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:42.062780    5304 patch_prober.go:29] interesting pod/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-25tqv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body=
Nov 02 15:06:42.063022 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:42.062848    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-25tqv" podUID="3ed68512-6850-45d4-af0d-a941c2bf6ec9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused"
Nov 02 15:06:42.065910 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:42.065888    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-624254/simulator-w3-624254-kserve-56866b7699-z47sd"
Nov 02 15:06:43.063026 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:43.062985    5304 patch_prober.go:29] interesting pod/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-25tqv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body=
Nov 02 15:06:43.063225 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:43.063049    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-25tqv" podUID="3ed68512-6850-45d4-af0d-a941c2bf6ec9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused"
Nov 02 15:06:44.062952 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:44.062894    5304 patch_prober.go:29] interesting pod/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-25tqv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body=
Nov 02 15:06:44.063462 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:44.062959    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-25tqv" podUID="3ed68512-6850-45d4-af0d-a941c2bf6ec9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused"
Nov 02 15:06:44.561444 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:44.561397    5304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-739bf1/simulator-w2-739bf1-kserve-7c74f85465-ltk4t"]
Nov 02 15:06:44.565120 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:44.565099    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-739bf1/simulator-w2-739bf1-kserve-7c74f85465-ltk4t"
Nov 02 15:06:44.584067 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:44.584037    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-739bf1/simulator-w2-739bf1-kserve-7c74f85465-ltk4t"]
Nov 02 15:06:44.594217 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:44.594186    5304 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-739bf1\"/\"kube-root-ca.crt\""
Nov 02 15:06:44.594349 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:44.594213    5304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-739bf1\"/\"default-dockercfg-q4dms\""
Nov 02 15:06:44.594349 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:44.594223    5304 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-739bf1\"/\"openshift-service-ca.crt\""
Nov 02 15:06:44.594457 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:44.594376    5304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-739bf1\"/\"simulator-w2-739bf1-kserve-self-signed-certs\""
Nov 02 15:06:44.622546 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:44.622519    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cf6a1ded-28a3-49eb-878b-b4ad77a0bb5c-tls-certs\") pod \"simulator-w2-739bf1-kserve-7c74f85465-ltk4t\" (UID: \"cf6a1ded-28a3-49eb-878b-b4ad77a0bb5c\") " pod="e2e-models-e2e-worker-w2-739bf1/simulator-w2-739bf1-kserve-7c74f85465-ltk4t"
Nov 02 15:06:44.622662 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:44.622551    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cf6a1ded-28a3-49eb-878b-b4ad77a0bb5c-model-cache\") pod \"simulator-w2-739bf1-kserve-7c74f85465-ltk4t\" (UID: \"cf6a1ded-28a3-49eb-878b-b4ad77a0bb5c\") " pod="e2e-models-e2e-worker-w2-739bf1/simulator-w2-739bf1-kserve-7c74f85465-ltk4t"
Nov 02 15:06:44.622662 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:44.622588    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-92w48\" (UniqueName: \"kubernetes.io/projected/cf6a1ded-28a3-49eb-878b-b4ad77a0bb5c-kube-api-access-92w48\") pod \"simulator-w2-739bf1-kserve-7c74f85465-ltk4t\" (UID: \"cf6a1ded-28a3-49eb-878b-b4ad77a0bb5c\") " pod="e2e-models-e2e-worker-w2-739bf1/simulator-w2-739bf1-kserve-7c74f85465-ltk4t"
Nov 02 15:06:44.622758 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:44.622712    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cf6a1ded-28a3-49eb-878b-b4ad77a0bb5c-dshm\") pod \"simulator-w2-739bf1-kserve-7c74f85465-ltk4t\" (UID: \"cf6a1ded-28a3-49eb-878b-b4ad77a0bb5c\") " pod="e2e-models-e2e-worker-w2-739bf1/simulator-w2-739bf1-kserve-7c74f85465-ltk4t"
Nov 02 15:06:44.622758 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:44.622748    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cf6a1ded-28a3-49eb-878b-b4ad77a0bb5c-home\") pod \"simulator-w2-739bf1-kserve-7c74f85465-ltk4t\" (UID: \"cf6a1ded-28a3-49eb-878b-b4ad77a0bb5c\") " pod="e2e-models-e2e-worker-w2-739bf1/simulator-w2-739bf1-kserve-7c74f85465-ltk4t"
Nov 02 15:06:44.622878 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:44.622858    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cf6a1ded-28a3-49eb-878b-b4ad77a0bb5c-tmp-dir\") pod \"simulator-w2-739bf1-kserve-7c74f85465-ltk4t\" (UID: \"cf6a1ded-28a3-49eb-878b-b4ad77a0bb5c\") " pod="e2e-models-e2e-worker-w2-739bf1/simulator-w2-739bf1-kserve-7c74f85465-ltk4t"
Nov 02 15:06:44.724276 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:44.724241    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cf6a1ded-28a3-49eb-878b-b4ad77a0bb5c-tmp-dir\") pod \"simulator-w2-739bf1-kserve-7c74f85465-ltk4t\" (UID: \"cf6a1ded-28a3-49eb-878b-b4ad77a0bb5c\") " pod="e2e-models-e2e-worker-w2-739bf1/simulator-w2-739bf1-kserve-7c74f85465-ltk4t"
Nov 02 15:06:44.724276 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:44.724281    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cf6a1ded-28a3-49eb-878b-b4ad77a0bb5c-tls-certs\") pod \"simulator-w2-739bf1-kserve-7c74f85465-ltk4t\" (UID: \"cf6a1ded-28a3-49eb-878b-b4ad77a0bb5c\") " pod="e2e-models-e2e-worker-w2-739bf1/simulator-w2-739bf1-kserve-7c74f85465-ltk4t"
Nov 02 15:06:44.724529 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:44.724305    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cf6a1ded-28a3-49eb-878b-b4ad77a0bb5c-model-cache\") pod \"simulator-w2-739bf1-kserve-7c74f85465-ltk4t\" (UID: \"cf6a1ded-28a3-49eb-878b-b4ad77a0bb5c\") " pod="e2e-models-e2e-worker-w2-739bf1/simulator-w2-739bf1-kserve-7c74f85465-ltk4t"
Nov 02 15:06:44.724529 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:44.724435    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-92w48\" (UniqueName: \"kubernetes.io/projected/cf6a1ded-28a3-49eb-878b-b4ad77a0bb5c-kube-api-access-92w48\") pod \"simulator-w2-739bf1-kserve-7c74f85465-ltk4t\" (UID: \"cf6a1ded-28a3-49eb-878b-b4ad77a0bb5c\") " pod="e2e-models-e2e-worker-w2-739bf1/simulator-w2-739bf1-kserve-7c74f85465-ltk4t"
Nov 02 15:06:44.724645 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:44.724536    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cf6a1ded-28a3-49eb-878b-b4ad77a0bb5c-dshm\") pod \"simulator-w2-739bf1-kserve-7c74f85465-ltk4t\" (UID: \"cf6a1ded-28a3-49eb-878b-b4ad77a0bb5c\") " pod="e2e-models-e2e-worker-w2-739bf1/simulator-w2-739bf1-kserve-7c74f85465-ltk4t"
Nov 02 15:06:44.724645 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:44.724575    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cf6a1ded-28a3-49eb-878b-b4ad77a0bb5c-home\") pod \"simulator-w2-739bf1-kserve-7c74f85465-ltk4t\" (UID: \"cf6a1ded-28a3-49eb-878b-b4ad77a0bb5c\") " pod="e2e-models-e2e-worker-w2-739bf1/simulator-w2-739bf1-kserve-7c74f85465-ltk4t"
Nov 02 15:06:44.724746 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:44.724722    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cf6a1ded-28a3-49eb-878b-b4ad77a0bb5c-model-cache\") pod \"simulator-w2-739bf1-kserve-7c74f85465-ltk4t\" (UID: \"cf6a1ded-28a3-49eb-878b-b4ad77a0bb5c\") " pod="e2e-models-e2e-worker-w2-739bf1/simulator-w2-739bf1-kserve-7c74f85465-ltk4t"
Nov 02 15:06:44.724807 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:44.724744    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cf6a1ded-28a3-49eb-878b-b4ad77a0bb5c-tmp-dir\") pod \"simulator-w2-739bf1-kserve-7c74f85465-ltk4t\" (UID: \"cf6a1ded-28a3-49eb-878b-b4ad77a0bb5c\") " pod="e2e-models-e2e-worker-w2-739bf1/simulator-w2-739bf1-kserve-7c74f85465-ltk4t"
Nov 02 15:06:44.724991 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:44.724971    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cf6a1ded-28a3-49eb-878b-b4ad77a0bb5c-home\") pod \"simulator-w2-739bf1-kserve-7c74f85465-ltk4t\" (UID: \"cf6a1ded-28a3-49eb-878b-b4ad77a0bb5c\") " pod="e2e-models-e2e-worker-w2-739bf1/simulator-w2-739bf1-kserve-7c74f85465-ltk4t"
Nov 02 15:06:44.726771 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:44.726747    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cf6a1ded-28a3-49eb-878b-b4ad77a0bb5c-dshm\") pod \"simulator-w2-739bf1-kserve-7c74f85465-ltk4t\" (UID: \"cf6a1ded-28a3-49eb-878b-b4ad77a0bb5c\") " pod="e2e-models-e2e-worker-w2-739bf1/simulator-w2-739bf1-kserve-7c74f85465-ltk4t"
Nov 02 15:06:44.726887 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:44.726856    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cf6a1ded-28a3-49eb-878b-b4ad77a0bb5c-tls-certs\") pod \"simulator-w2-739bf1-kserve-7c74f85465-ltk4t\" (UID: \"cf6a1ded-28a3-49eb-878b-b4ad77a0bb5c\") " pod="e2e-models-e2e-worker-w2-739bf1/simulator-w2-739bf1-kserve-7c74f85465-ltk4t"
Nov 02 15:06:44.762191 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:44.762158    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-92w48\" (UniqueName: \"kubernetes.io/projected/cf6a1ded-28a3-49eb-878b-b4ad77a0bb5c-kube-api-access-92w48\") pod \"simulator-w2-739bf1-kserve-7c74f85465-ltk4t\" (UID: \"cf6a1ded-28a3-49eb-878b-b4ad77a0bb5c\") " pod="e2e-models-e2e-worker-w2-739bf1/simulator-w2-739bf1-kserve-7c74f85465-ltk4t"
Nov 02 15:06:44.878726 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:44.878689    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-739bf1/simulator-w2-739bf1-kserve-7c74f85465-ltk4t"
Nov 02 15:06:45.063500 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:45.063464    5304 patch_prober.go:29] interesting pod/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-25tqv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body=
Nov 02 15:06:45.063922 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:45.063518    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-25tqv" podUID="3ed68512-6850-45d4-af0d-a941c2bf6ec9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused"
Nov 02 15:06:45.177226 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:45.177149    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-739bf1/simulator-w2-739bf1-kserve-7c74f85465-ltk4t"]
Nov 02 15:06:45.180194 ip-10-0-135-138 kubenswrapper[5304]: W1102 15:06:45.180164    5304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcf6a1ded_28a3_49eb_878b_b4ad77a0bb5c.slice/crio-b47f48bf334bbae1e02521b22179b9069b7e76b7a1cf8ccae1e93dab301c0c6a WatchSource:0}: Error finding container b47f48bf334bbae1e02521b22179b9069b7e76b7a1cf8ccae1e93dab301c0c6a: Status 404 returned error can't find the container with id b47f48bf334bbae1e02521b22179b9069b7e76b7a1cf8ccae1e93dab301c0c6a
Nov 02 15:06:46.068416 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:46.068377    5304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-25tqv"
Nov 02 15:06:46.069486 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:46.069457    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-25tqv"
Nov 02 15:06:46.074807 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:46.074770    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-739bf1/simulator-w2-739bf1-kserve-7c74f85465-ltk4t" event={"ID":"cf6a1ded-28a3-49eb-878b-b4ad77a0bb5c","Type":"ContainerStarted","Data":"5475afc0733bb8493b073d13d49dd895ba6cba428a1433374b0572ec4b1c5ed9"}
Nov 02 15:06:46.074923 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:46.074813    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-739bf1/simulator-w2-739bf1-kserve-7c74f85465-ltk4t" event={"ID":"cf6a1ded-28a3-49eb-878b-b4ad77a0bb5c","Type":"ContainerStarted","Data":"b47f48bf334bbae1e02521b22179b9069b7e76b7a1cf8ccae1e93dab301c0c6a"}
Nov 02 15:06:46.194981 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:46.194913    5304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-739bf1/simulator-w2-739bf1-kserve-7c74f85465-ltk4t" podStartSLOduration=2.194894491 podStartE2EDuration="2.194894491s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:06:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:06:46.193423443 +0000 UTC m=+606.187247853" watchObservedRunningTime="2026-11-02 15:06:46.194894491 +0000 UTC m=+606.188718901"
Nov 02 15:06:46.879311 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:46.879280    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-739bf1/simulator-w2-739bf1-kserve-7c74f85465-ltk4t"
Nov 02 15:06:54.799871 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:06:54.799834    5304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k6fjk" containerName="extractor"
Nov 02 15:06:54.879334 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:54.879302    5304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-739bf1/simulator-w2-739bf1-kserve-7c74f85465-ltk4t"
Nov 02 15:06:54.891942 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:54.891914    5304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-739bf1/simulator-w2-739bf1-kserve-7c74f85465-ltk4t"
Nov 02 15:06:55.135750 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:55.135723    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-739bf1/simulator-w2-739bf1-kserve-7c74f85465-ltk4t"
Nov 02 15:06:59.787938 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:59.787897    5304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-1cc1b2/premium-w5-1cc1b2-kserve-58db46f97c-cgdqg"]
Nov 02 15:06:59.790708 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:59.790688    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-1cc1b2/premium-w5-1cc1b2-kserve-58db46f97c-cgdqg"
Nov 02 15:06:59.810524 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:59.810499    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-1cc1b2/premium-w5-1cc1b2-kserve-58db46f97c-cgdqg"]
Nov 02 15:06:59.820172 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:59.820111    5304 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-1cc1b2\"/\"kube-root-ca.crt\""
Nov 02 15:06:59.820363 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:59.820258    5304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-1cc1b2\"/\"default-dockercfg-l4n54\""
Nov 02 15:06:59.820363 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:59.820288    5304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-1cc1b2\"/\"premium-w5-1cc1b2-kserve-self-signed-certs\""
Nov 02 15:06:59.820537 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:59.820453    5304 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-1cc1b2\"/\"openshift-service-ca.crt\""
Nov 02 15:06:59.899342 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:59.899308    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/210f620a-8ae7-4e4c-acc3-46f75ddc6105-home\") pod \"premium-w5-1cc1b2-kserve-58db46f97c-cgdqg\" (UID: \"210f620a-8ae7-4e4c-acc3-46f75ddc6105\") " pod="e2e-models-e2e-worker-w5-1cc1b2/premium-w5-1cc1b2-kserve-58db46f97c-cgdqg"
Nov 02 15:06:59.899518 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:59.899350    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/210f620a-8ae7-4e4c-acc3-46f75ddc6105-model-cache\") pod \"premium-w5-1cc1b2-kserve-58db46f97c-cgdqg\" (UID: \"210f620a-8ae7-4e4c-acc3-46f75ddc6105\") " pod="e2e-models-e2e-worker-w5-1cc1b2/premium-w5-1cc1b2-kserve-58db46f97c-cgdqg"
Nov 02 15:06:59.899518 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:59.899425    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v2cc5\" (UniqueName: \"kubernetes.io/projected/210f620a-8ae7-4e4c-acc3-46f75ddc6105-kube-api-access-v2cc5\") pod \"premium-w5-1cc1b2-kserve-58db46f97c-cgdqg\" (UID: \"210f620a-8ae7-4e4c-acc3-46f75ddc6105\") " pod="e2e-models-e2e-worker-w5-1cc1b2/premium-w5-1cc1b2-kserve-58db46f97c-cgdqg"
Nov 02 15:06:59.899518 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:59.899473    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/210f620a-8ae7-4e4c-acc3-46f75ddc6105-tmp-dir\") pod \"premium-w5-1cc1b2-kserve-58db46f97c-cgdqg\" (UID: \"210f620a-8ae7-4e4c-acc3-46f75ddc6105\") " pod="e2e-models-e2e-worker-w5-1cc1b2/premium-w5-1cc1b2-kserve-58db46f97c-cgdqg"
Nov 02 15:06:59.899623 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:59.899538    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/210f620a-8ae7-4e4c-acc3-46f75ddc6105-tls-certs\") pod \"premium-w5-1cc1b2-kserve-58db46f97c-cgdqg\" (UID: \"210f620a-8ae7-4e4c-acc3-46f75ddc6105\") " pod="e2e-models-e2e-worker-w5-1cc1b2/premium-w5-1cc1b2-kserve-58db46f97c-cgdqg"
Nov 02 15:06:59.899623 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:06:59.899590    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/210f620a-8ae7-4e4c-acc3-46f75ddc6105-dshm\") pod \"premium-w5-1cc1b2-kserve-58db46f97c-cgdqg\" (UID: \"210f620a-8ae7-4e4c-acc3-46f75ddc6105\") " pod="e2e-models-e2e-worker-w5-1cc1b2/premium-w5-1cc1b2-kserve-58db46f97c-cgdqg"
Nov 02 15:07:00.001082 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:07:00.001031    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2cc5\" (UniqueName: \"kubernetes.io/projected/210f620a-8ae7-4e4c-acc3-46f75ddc6105-kube-api-access-v2cc5\") pod \"premium-w5-1cc1b2-kserve-58db46f97c-cgdqg\" (UID: \"210f620a-8ae7-4e4c-acc3-46f75ddc6105\") " pod="e2e-models-e2e-worker-w5-1cc1b2/premium-w5-1cc1b2-kserve-58db46f97c-cgdqg"
Nov 02 15:07:00.001354 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:07:00.001161    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/210f620a-8ae7-4e4c-acc3-46f75ddc6105-tmp-dir\") pod \"premium-w5-1cc1b2-kserve-58db46f97c-cgdqg\" (UID: \"210f620a-8ae7-4e4c-acc3-46f75ddc6105\") " pod="e2e-models-e2e-worker-w5-1cc1b2/premium-w5-1cc1b2-kserve-58db46f97c-cgdqg"
Nov 02 15:07:00.001354 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:07:00.001193    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/210f620a-8ae7-4e4c-acc3-46f75ddc6105-tls-certs\") pod \"premium-w5-1cc1b2-kserve-58db46f97c-cgdqg\" (UID: \"210f620a-8ae7-4e4c-acc3-46f75ddc6105\") " pod="e2e-models-e2e-worker-w5-1cc1b2/premium-w5-1cc1b2-kserve-58db46f97c-cgdqg"
Nov 02 15:07:00.001354 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:07:00.001241    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/210f620a-8ae7-4e4c-acc3-46f75ddc6105-dshm\") pod \"premium-w5-1cc1b2-kserve-58db46f97c-cgdqg\" (UID: \"210f620a-8ae7-4e4c-acc3-46f75ddc6105\") " pod="e2e-models-e2e-worker-w5-1cc1b2/premium-w5-1cc1b2-kserve-58db46f97c-cgdqg"
Nov 02 15:07:00.001354 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:07:00.001344    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/210f620a-8ae7-4e4c-acc3-46f75ddc6105-home\") pod \"premium-w5-1cc1b2-kserve-58db46f97c-cgdqg\" (UID: \"210f620a-8ae7-4e4c-acc3-46f75ddc6105\") " pod="e2e-models-e2e-worker-w5-1cc1b2/premium-w5-1cc1b2-kserve-58db46f97c-cgdqg"
Nov 02 15:07:00.001603 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:07:00.001377    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/210f620a-8ae7-4e4c-acc3-46f75ddc6105-model-cache\") pod \"premium-w5-1cc1b2-kserve-58db46f97c-cgdqg\" (UID: \"210f620a-8ae7-4e4c-acc3-46f75ddc6105\") " pod="e2e-models-e2e-worker-w5-1cc1b2/premium-w5-1cc1b2-kserve-58db46f97c-cgdqg"
Nov 02 15:07:00.001857 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:07:00.001832    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/210f620a-8ae7-4e4c-acc3-46f75ddc6105-model-cache\") pod \"premium-w5-1cc1b2-kserve-58db46f97c-cgdqg\" (UID: \"210f620a-8ae7-4e4c-acc3-46f75ddc6105\") " pod="e2e-models-e2e-worker-w5-1cc1b2/premium-w5-1cc1b2-kserve-58db46f97c-cgdqg"
Nov 02 15:07:00.002611 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:07:00.002481    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/210f620a-8ae7-4e4c-acc3-46f75ddc6105-tmp-dir\") pod \"premium-w5-1cc1b2-kserve-58db46f97c-cgdqg\" (UID: \"210f620a-8ae7-4e4c-acc3-46f75ddc6105\") " pod="e2e-models-e2e-worker-w5-1cc1b2/premium-w5-1cc1b2-kserve-58db46f97c-cgdqg"
Nov 02 15:07:00.003156 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:07:00.003091    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/210f620a-8ae7-4e4c-acc3-46f75ddc6105-home\") pod \"premium-w5-1cc1b2-kserve-58db46f97c-cgdqg\" (UID: \"210f620a-8ae7-4e4c-acc3-46f75ddc6105\") " pod="e2e-models-e2e-worker-w5-1cc1b2/premium-w5-1cc1b2-kserve-58db46f97c-cgdqg"
Nov 02 15:07:00.005776 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:07:00.005753    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/210f620a-8ae7-4e4c-acc3-46f75ddc6105-tls-certs\") pod \"premium-w5-1cc1b2-kserve-58db46f97c-cgdqg\" (UID: \"210f620a-8ae7-4e4c-acc3-46f75ddc6105\") " pod="e2e-models-e2e-worker-w5-1cc1b2/premium-w5-1cc1b2-kserve-58db46f97c-cgdqg"
Nov 02 15:07:00.006251 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:07:00.006226    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/210f620a-8ae7-4e4c-acc3-46f75ddc6105-dshm\") pod \"premium-w5-1cc1b2-kserve-58db46f97c-cgdqg\" (UID: \"210f620a-8ae7-4e4c-acc3-46f75ddc6105\") " pod="e2e-models-e2e-worker-w5-1cc1b2/premium-w5-1cc1b2-kserve-58db46f97c-cgdqg"
Nov 02 15:07:00.026829 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:07:00.026802    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v2cc5\" (UniqueName: \"kubernetes.io/projected/210f620a-8ae7-4e4c-acc3-46f75ddc6105-kube-api-access-v2cc5\") pod \"premium-w5-1cc1b2-kserve-58db46f97c-cgdqg\" (UID: \"210f620a-8ae7-4e4c-acc3-46f75ddc6105\") " pod="e2e-models-e2e-worker-w5-1cc1b2/premium-w5-1cc1b2-kserve-58db46f97c-cgdqg"
Nov 02 15:07:00.105117 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:07:00.105085    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-1cc1b2/premium-w5-1cc1b2-kserve-58db46f97c-cgdqg"
Nov 02 15:07:00.403704 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:07:00.403613    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-1cc1b2/premium-w5-1cc1b2-kserve-58db46f97c-cgdqg"]
Nov 02 15:07:00.407356 ip-10-0-135-138 kubenswrapper[5304]: W1102 15:07:00.407330    5304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod210f620a_8ae7_4e4c_acc3_46f75ddc6105.slice/crio-7f0c083bc4a986aa1ca5162a54f161f37e52887b19d31f74b292b81288e0a2dd WatchSource:0}: Error finding container 7f0c083bc4a986aa1ca5162a54f161f37e52887b19d31f74b292b81288e0a2dd: Status 404 returned error can't find the container with id 7f0c083bc4a986aa1ca5162a54f161f37e52887b19d31f74b292b81288e0a2dd
Nov 02 15:07:01.187361 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:07:01.187324    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-1cc1b2/premium-w5-1cc1b2-kserve-58db46f97c-cgdqg" event={"ID":"210f620a-8ae7-4e4c-acc3-46f75ddc6105","Type":"ContainerStarted","Data":"ac6c3a0a81e5ed9d73f434bbb1db0c11246882624954e2b5f8c208073408bfd6"}
Nov 02 15:07:01.187361 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:07:01.187368    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-1cc1b2/premium-w5-1cc1b2-kserve-58db46f97c-cgdqg" event={"ID":"210f620a-8ae7-4e4c-acc3-46f75ddc6105","Type":"ContainerStarted","Data":"7f0c083bc4a986aa1ca5162a54f161f37e52887b19d31f74b292b81288e0a2dd"}
Nov 02 15:07:01.228584 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:07:01.228525    5304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-1cc1b2/premium-w5-1cc1b2-kserve-58db46f97c-cgdqg" podStartSLOduration=2.228510902 podStartE2EDuration="2.228510902s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:06:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:07:01.227789656 +0000 UTC m=+621.221614065" watchObservedRunningTime="2026-11-02 15:07:01.228510902 +0000 UTC m=+621.222335311"
Nov 02 15:07:02.105564 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:07:02.105528    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-1cc1b2/premium-w5-1cc1b2-kserve-58db46f97c-cgdqg"
Nov 02 15:07:10.105908 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:07:10.105873    5304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-1cc1b2/premium-w5-1cc1b2-kserve-58db46f97c-cgdqg"
Nov 02 15:07:10.118583 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:07:10.118554    5304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-1cc1b2/premium-w5-1cc1b2-kserve-58db46f97c-cgdqg"
Nov 02 15:07:10.243733 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:07:10.243699    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-1cc1b2/premium-w5-1cc1b2-kserve-58db46f97c-cgdqg"
Nov 02 15:07:12.650301 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:07:12.650267    5304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-5dd8e9/premium-w1-5dd8e9-kserve-67588456fb-j6lph"]
Nov 02 15:07:12.653923 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:07:12.653885    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-5dd8e9/premium-w1-5dd8e9-kserve-67588456fb-j6lph"
Nov 02 15:07:12.682007 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:07:12.681970    5304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-5dd8e9\"/\"premium-w1-5dd8e9-kserve-self-signed-certs\""
Nov 02 15:07:12.683942 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:07:12.683912    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-5dd8e9/premium-w1-5dd8e9-kserve-67588456fb-j6lph"]
Nov 02 15:07:12.754962 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:07:12.754923    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4ccc607a-14e4-43ca-8999-78100d4b799f-model-cache\") pod \"premium-w1-5dd8e9-kserve-67588456fb-j6lph\" (UID: \"4ccc607a-14e4-43ca-8999-78100d4b799f\") " pod="e2e-models-e2e-worker-w1-5dd8e9/premium-w1-5dd8e9-kserve-67588456fb-j6lph"
Nov 02 15:07:12.755173 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:07:12.754983    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4ccc607a-14e4-43ca-8999-78100d4b799f-tmp-dir\") pod \"premium-w1-5dd8e9-kserve-67588456fb-j6lph\" (UID: \"4ccc607a-14e4-43ca-8999-78100d4b799f\") " pod="e2e-models-e2e-worker-w1-5dd8e9/premium-w1-5dd8e9-kserve-67588456fb-j6lph"
Nov 02 15:07:12.755173 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:07:12.755091    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4ccc607a-14e4-43ca-8999-78100d4b799f-home\") pod \"premium-w1-5dd8e9-kserve-67588456fb-j6lph\" (UID: \"4ccc607a-14e4-43ca-8999-78100d4b799f\") " pod="e2e-models-e2e-worker-w1-5dd8e9/premium-w1-5dd8e9-kserve-67588456fb-j6lph"
Nov 02 15:07:12.755173 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:07:12.755137    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4ccc607a-14e4-43ca-8999-78100d4b799f-tls-certs\") pod \"premium-w1-5dd8e9-kserve-67588456fb-j6lph\" (UID: \"4ccc607a-14e4-43ca-8999-78100d4b799f\") " pod="e2e-models-e2e-worker-w1-5dd8e9/premium-w1-5dd8e9-kserve-67588456fb-j6lph"
Nov 02 15:07:12.755321 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:07:12.755284    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4ccc607a-14e4-43ca-8999-78100d4b799f-dshm\") pod \"premium-w1-5dd8e9-kserve-67588456fb-j6lph\" (UID: \"4ccc607a-14e4-43ca-8999-78100d4b799f\") " pod="e2e-models-e2e-worker-w1-5dd8e9/premium-w1-5dd8e9-kserve-67588456fb-j6lph"
Nov 02 15:07:12.755410 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:07:12.755393    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kjhfg\" (UniqueName: \"kubernetes.io/projected/4ccc607a-14e4-43ca-8999-78100d4b799f-kube-api-access-kjhfg\") pod \"premium-w1-5dd8e9-kserve-67588456fb-j6lph\" (UID: \"4ccc607a-14e4-43ca-8999-78100d4b799f\") " pod="e2e-models-e2e-worker-w1-5dd8e9/premium-w1-5dd8e9-kserve-67588456fb-j6lph"
Nov 02 15:07:12.857734 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:07:12.857664    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4ccc607a-14e4-43ca-8999-78100d4b799f-model-cache\") pod \"premium-w1-5dd8e9-kserve-67588456fb-j6lph\" (UID: \"4ccc607a-14e4-43ca-8999-78100d4b799f\") " pod="e2e-models-e2e-worker-w1-5dd8e9/premium-w1-5dd8e9-kserve-67588456fb-j6lph"
Nov 02 15:07:12.857734 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:07:12.857739    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4ccc607a-14e4-43ca-8999-78100d4b799f-tmp-dir\") pod \"premium-w1-5dd8e9-kserve-67588456fb-j6lph\" (UID: \"4ccc607a-14e4-43ca-8999-78100d4b799f\") " pod="e2e-models-e2e-worker-w1-5dd8e9/premium-w1-5dd8e9-kserve-67588456fb-j6lph"
Nov 02 15:07:12.858081 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:07:12.857784    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4ccc607a-14e4-43ca-8999-78100d4b799f-home\") pod \"premium-w1-5dd8e9-kserve-67588456fb-j6lph\" (UID: \"4ccc607a-14e4-43ca-8999-78100d4b799f\") " pod="e2e-models-e2e-worker-w1-5dd8e9/premium-w1-5dd8e9-kserve-67588456fb-j6lph"
Nov 02 15:07:12.858081 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:07:12.857810    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4ccc607a-14e4-43ca-8999-78100d4b799f-tls-certs\") pod \"premium-w1-5dd8e9-kserve-67588456fb-j6lph\" (UID: \"4ccc607a-14e4-43ca-8999-78100d4b799f\") " pod="e2e-models-e2e-worker-w1-5dd8e9/premium-w1-5dd8e9-kserve-67588456fb-j6lph"
Nov 02 15:07:12.858081 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:07:12.857887    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4ccc607a-14e4-43ca-8999-78100d4b799f-dshm\") pod \"premium-w1-5dd8e9-kserve-67588456fb-j6lph\" (UID: \"4ccc607a-14e4-43ca-8999-78100d4b799f\") " pod="e2e-models-e2e-worker-w1-5dd8e9/premium-w1-5dd8e9-kserve-67588456fb-j6lph"
Nov 02 15:07:12.862347 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:07:12.862305    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kjhfg\" (UniqueName: \"kubernetes.io/projected/4ccc607a-14e4-43ca-8999-78100d4b799f-kube-api-access-kjhfg\") pod \"premium-w1-5dd8e9-kserve-67588456fb-j6lph\" (UID: \"4ccc607a-14e4-43ca-8999-78100d4b799f\") " pod="e2e-models-e2e-worker-w1-5dd8e9/premium-w1-5dd8e9-kserve-67588456fb-j6lph"
Nov 02 15:07:12.863058 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:07:12.863036    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4ccc607a-14e4-43ca-8999-78100d4b799f-model-cache\") pod \"premium-w1-5dd8e9-kserve-67588456fb-j6lph\" (UID: \"4ccc607a-14e4-43ca-8999-78100d4b799f\") " pod="e2e-models-e2e-worker-w1-5dd8e9/premium-w1-5dd8e9-kserve-67588456fb-j6lph"
Nov 02 15:07:12.863305 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:07:12.863287    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4ccc607a-14e4-43ca-8999-78100d4b799f-tmp-dir\") pod \"premium-w1-5dd8e9-kserve-67588456fb-j6lph\" (UID: \"4ccc607a-14e4-43ca-8999-78100d4b799f\") " pod="e2e-models-e2e-worker-w1-5dd8e9/premium-w1-5dd8e9-kserve-67588456fb-j6lph"
Nov 02 15:07:12.863502 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:07:12.863472    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4ccc607a-14e4-43ca-8999-78100d4b799f-home\") pod \"premium-w1-5dd8e9-kserve-67588456fb-j6lph\" (UID: \"4ccc607a-14e4-43ca-8999-78100d4b799f\") " pod="e2e-models-e2e-worker-w1-5dd8e9/premium-w1-5dd8e9-kserve-67588456fb-j6lph"
Nov 02 15:07:12.866056 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:07:12.866026    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4ccc607a-14e4-43ca-8999-78100d4b799f-dshm\") pod \"premium-w1-5dd8e9-kserve-67588456fb-j6lph\" (UID: \"4ccc607a-14e4-43ca-8999-78100d4b799f\") " pod="e2e-models-e2e-worker-w1-5dd8e9/premium-w1-5dd8e9-kserve-67588456fb-j6lph"
Nov 02 15:07:12.866179 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:07:12.866097    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4ccc607a-14e4-43ca-8999-78100d4b799f-tls-certs\") pod \"premium-w1-5dd8e9-kserve-67588456fb-j6lph\" (UID: \"4ccc607a-14e4-43ca-8999-78100d4b799f\") " pod="e2e-models-e2e-worker-w1-5dd8e9/premium-w1-5dd8e9-kserve-67588456fb-j6lph"
Nov 02 15:07:12.887992 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:07:12.887971    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kjhfg\" (UniqueName: \"kubernetes.io/projected/4ccc607a-14e4-43ca-8999-78100d4b799f-kube-api-access-kjhfg\") pod \"premium-w1-5dd8e9-kserve-67588456fb-j6lph\" (UID: \"4ccc607a-14e4-43ca-8999-78100d4b799f\") " pod="e2e-models-e2e-worker-w1-5dd8e9/premium-w1-5dd8e9-kserve-67588456fb-j6lph"
Nov 02 15:07:12.969541 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:07:12.969457    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-5dd8e9/premium-w1-5dd8e9-kserve-67588456fb-j6lph"
Nov 02 15:07:13.270949 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:07:13.270865    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-5dd8e9/premium-w1-5dd8e9-kserve-67588456fb-j6lph"]
Nov 02 15:07:13.274435 ip-10-0-135-138 kubenswrapper[5304]: W1102 15:07:13.274408    5304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4ccc607a_14e4_43ca_8999_78100d4b799f.slice/crio-5243f93f225cbb4fdfd9e33c57626288296521ec42cce8e931db744d20252823 WatchSource:0}: Error finding container 5243f93f225cbb4fdfd9e33c57626288296521ec42cce8e931db744d20252823: Status 404 returned error can't find the container with id 5243f93f225cbb4fdfd9e33c57626288296521ec42cce8e931db744d20252823
Nov 02 15:07:14.256063 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:07:14.256025    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-5dd8e9/premium-w1-5dd8e9-kserve-67588456fb-j6lph" event={"ID":"4ccc607a-14e4-43ca-8999-78100d4b799f","Type":"ContainerStarted","Data":"fccc8e8f461aa26eee58ae4e1b570d520d7a89e01d72c90fbcb923c77a067303"}
Nov 02 15:07:14.256063 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:07:14.256070    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-5dd8e9/premium-w1-5dd8e9-kserve-67588456fb-j6lph" event={"ID":"4ccc607a-14e4-43ca-8999-78100d4b799f","Type":"ContainerStarted","Data":"5243f93f225cbb4fdfd9e33c57626288296521ec42cce8e931db744d20252823"}
Nov 02 15:07:14.300421 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:07:14.300359    5304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-5dd8e9/premium-w1-5dd8e9-kserve-67588456fb-j6lph" podStartSLOduration=2.30034474 podStartE2EDuration="2.30034474s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:07:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:07:14.298975031 +0000 UTC m=+634.292799439" watchObservedRunningTime="2026-11-02 15:07:14.30034474 +0000 UTC m=+634.294169148"
Nov 02 15:07:14.969692 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:07:14.969652    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5dd8e9/premium-w1-5dd8e9-kserve-67588456fb-j6lph"
Nov 02 15:07:22.969681 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:07:22.969598    5304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-5dd8e9/premium-w1-5dd8e9-kserve-67588456fb-j6lph"
Nov 02 15:07:22.992015 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:07:22.991980    5304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-5dd8e9/premium-w1-5dd8e9-kserve-67588456fb-j6lph"
Nov 02 15:07:23.311413 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:07:23.311331    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5dd8e9/premium-w1-5dd8e9-kserve-67588456fb-j6lph"
Nov 02 15:07:24.798678 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:07:24.798633    5304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k6fjk" containerName="extractor"
Nov 02 15:07:33.673642 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:07:33.673608    5304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-5113c2/premium-w6-5113c2-kserve-b84945cb5-pc6f8"]
Nov 02 15:07:33.676602 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:07:33.676568    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-5113c2/premium-w6-5113c2-kserve-b84945cb5-pc6f8"
Nov 02 15:07:33.696194 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:07:33.696165    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-5113c2/premium-w6-5113c2-kserve-b84945cb5-pc6f8"]
Nov 02 15:07:33.705993 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:07:33.705962    5304 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-5113c2\"/\"openshift-service-ca.crt\""
Nov 02 15:07:33.706150 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:07:33.706040    5304 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-5113c2\"/\"kube-root-ca.crt\""
Nov 02 15:07:33.706237 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:07:33.706218    5304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-5113c2\"/\"premium-w6-5113c2-kserve-self-signed-certs\""
Nov 02 15:07:33.706296 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:07:33.706282    5304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-5113c2\"/\"default-dockercfg-dn9b4\""
Nov 02 15:07:33.790990 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:07:33.790958    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9fb3e992-7bb0-497a-a82c-6e396ba241c1-tmp-dir\") pod \"premium-w6-5113c2-kserve-b84945cb5-pc6f8\" (UID: \"9fb3e992-7bb0-497a-a82c-6e396ba241c1\") " pod="e2e-models-e2e-worker-w6-5113c2/premium-w6-5113c2-kserve-b84945cb5-pc6f8"
Nov 02 15:07:33.790990 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:07:33.790991    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k8rhb\" (UniqueName: \"kubernetes.io/projected/9fb3e992-7bb0-497a-a82c-6e396ba241c1-kube-api-access-k8rhb\") pod \"premium-w6-5113c2-kserve-b84945cb5-pc6f8\" (UID: \"9fb3e992-7bb0-497a-a82c-6e396ba241c1\") " pod="e2e-models-e2e-worker-w6-5113c2/premium-w6-5113c2-kserve-b84945cb5-pc6f8"
Nov 02 15:07:33.791261 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:07:33.791121    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9fb3e992-7bb0-497a-a82c-6e396ba241c1-model-cache\") pod \"premium-w6-5113c2-kserve-b84945cb5-pc6f8\" (UID: \"9fb3e992-7bb0-497a-a82c-6e396ba241c1\") " pod="e2e-models-e2e-worker-w6-5113c2/premium-w6-5113c2-kserve-b84945cb5-pc6f8"
Nov 02 15:07:33.791261 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:07:33.791218    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9fb3e992-7bb0-497a-a82c-6e396ba241c1-home\") pod \"premium-w6-5113c2-kserve-b84945cb5-pc6f8\" (UID: \"9fb3e992-7bb0-497a-a82c-6e396ba241c1\") " pod="e2e-models-e2e-worker-w6-5113c2/premium-w6-5113c2-kserve-b84945cb5-pc6f8"
Nov 02 15:07:33.791345 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:07:33.791330    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9fb3e992-7bb0-497a-a82c-6e396ba241c1-tls-certs\") pod \"premium-w6-5113c2-kserve-b84945cb5-pc6f8\" (UID: \"9fb3e992-7bb0-497a-a82c-6e396ba241c1\") " pod="e2e-models-e2e-worker-w6-5113c2/premium-w6-5113c2-kserve-b84945cb5-pc6f8"
Nov 02 15:07:33.791384 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:07:33.791360    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9fb3e992-7bb0-497a-a82c-6e396ba241c1-dshm\") pod \"premium-w6-5113c2-kserve-b84945cb5-pc6f8\" (UID: \"9fb3e992-7bb0-497a-a82c-6e396ba241c1\") " pod="e2e-models-e2e-worker-w6-5113c2/premium-w6-5113c2-kserve-b84945cb5-pc6f8"
Nov 02 15:07:33.892392 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:07:33.892359    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9fb3e992-7bb0-497a-a82c-6e396ba241c1-tls-certs\") pod \"premium-w6-5113c2-kserve-b84945cb5-pc6f8\" (UID: \"9fb3e992-7bb0-497a-a82c-6e396ba241c1\") " pod="e2e-models-e2e-worker-w6-5113c2/premium-w6-5113c2-kserve-b84945cb5-pc6f8"
Nov 02 15:07:33.892617 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:07:33.892409    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9fb3e992-7bb0-497a-a82c-6e396ba241c1-dshm\") pod \"premium-w6-5113c2-kserve-b84945cb5-pc6f8\" (UID: \"9fb3e992-7bb0-497a-a82c-6e396ba241c1\") " pod="e2e-models-e2e-worker-w6-5113c2/premium-w6-5113c2-kserve-b84945cb5-pc6f8"
Nov 02 15:07:33.892617 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:07:33.892456    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9fb3e992-7bb0-497a-a82c-6e396ba241c1-tmp-dir\") pod \"premium-w6-5113c2-kserve-b84945cb5-pc6f8\" (UID: \"9fb3e992-7bb0-497a-a82c-6e396ba241c1\") " pod="e2e-models-e2e-worker-w6-5113c2/premium-w6-5113c2-kserve-b84945cb5-pc6f8"
Nov 02 15:07:33.892617 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:07:33.892481    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8rhb\" (UniqueName: \"kubernetes.io/projected/9fb3e992-7bb0-497a-a82c-6e396ba241c1-kube-api-access-k8rhb\") pod \"premium-w6-5113c2-kserve-b84945cb5-pc6f8\" (UID: \"9fb3e992-7bb0-497a-a82c-6e396ba241c1\") " pod="e2e-models-e2e-worker-w6-5113c2/premium-w6-5113c2-kserve-b84945cb5-pc6f8"
Nov 02 15:07:33.892617 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:07:33.892553    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9fb3e992-7bb0-497a-a82c-6e396ba241c1-model-cache\") pod \"premium-w6-5113c2-kserve-b84945cb5-pc6f8\" (UID: \"9fb3e992-7bb0-497a-a82c-6e396ba241c1\") " pod="e2e-models-e2e-worker-w6-5113c2/premium-w6-5113c2-kserve-b84945cb5-pc6f8"
Nov 02 15:07:33.892617 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:07:33.892593    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9fb3e992-7bb0-497a-a82c-6e396ba241c1-home\") pod \"premium-w6-5113c2-kserve-b84945cb5-pc6f8\" (UID: \"9fb3e992-7bb0-497a-a82c-6e396ba241c1\") " pod="e2e-models-e2e-worker-w6-5113c2/premium-w6-5113c2-kserve-b84945cb5-pc6f8"
Nov 02 15:07:33.892988 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:07:33.892953    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9fb3e992-7bb0-497a-a82c-6e396ba241c1-model-cache\") pod \"premium-w6-5113c2-kserve-b84945cb5-pc6f8\" (UID: \"9fb3e992-7bb0-497a-a82c-6e396ba241c1\") " pod="e2e-models-e2e-worker-w6-5113c2/premium-w6-5113c2-kserve-b84945cb5-pc6f8"
Nov 02 15:07:33.893187 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:07:33.892999    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9fb3e992-7bb0-497a-a82c-6e396ba241c1-home\") pod \"premium-w6-5113c2-kserve-b84945cb5-pc6f8\" (UID: \"9fb3e992-7bb0-497a-a82c-6e396ba241c1\") " pod="e2e-models-e2e-worker-w6-5113c2/premium-w6-5113c2-kserve-b84945cb5-pc6f8"
Nov 02 15:07:33.893187 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:07:33.892970    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9fb3e992-7bb0-497a-a82c-6e396ba241c1-tmp-dir\") pod \"premium-w6-5113c2-kserve-b84945cb5-pc6f8\" (UID: \"9fb3e992-7bb0-497a-a82c-6e396ba241c1\") " pod="e2e-models-e2e-worker-w6-5113c2/premium-w6-5113c2-kserve-b84945cb5-pc6f8"
Nov 02 15:07:33.894793 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:07:33.894767    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9fb3e992-7bb0-497a-a82c-6e396ba241c1-dshm\") pod \"premium-w6-5113c2-kserve-b84945cb5-pc6f8\" (UID: \"9fb3e992-7bb0-497a-a82c-6e396ba241c1\") " pod="e2e-models-e2e-worker-w6-5113c2/premium-w6-5113c2-kserve-b84945cb5-pc6f8"
Nov 02 15:07:33.895053 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:07:33.895038    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9fb3e992-7bb0-497a-a82c-6e396ba241c1-tls-certs\") pod \"premium-w6-5113c2-kserve-b84945cb5-pc6f8\" (UID: \"9fb3e992-7bb0-497a-a82c-6e396ba241c1\") " pod="e2e-models-e2e-worker-w6-5113c2/premium-w6-5113c2-kserve-b84945cb5-pc6f8"
Nov 02 15:07:33.926201 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:07:33.926107    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k8rhb\" (UniqueName: \"kubernetes.io/projected/9fb3e992-7bb0-497a-a82c-6e396ba241c1-kube-api-access-k8rhb\") pod \"premium-w6-5113c2-kserve-b84945cb5-pc6f8\" (UID: \"9fb3e992-7bb0-497a-a82c-6e396ba241c1\") " pod="e2e-models-e2e-worker-w6-5113c2/premium-w6-5113c2-kserve-b84945cb5-pc6f8"
Nov 02 15:07:33.988791 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:07:33.988763    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-5113c2/premium-w6-5113c2-kserve-b84945cb5-pc6f8"
Nov 02 15:07:34.287036 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:07:34.286964    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-5113c2/premium-w6-5113c2-kserve-b84945cb5-pc6f8"]
Nov 02 15:07:34.290921 ip-10-0-135-138 kubenswrapper[5304]: W1102 15:07:34.290893    5304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9fb3e992_7bb0_497a_a82c_6e396ba241c1.slice/crio-3c228ada8b883005eebe18993d69e96b95249a253b8a7774aa8ba821ac49c390 WatchSource:0}: Error finding container 3c228ada8b883005eebe18993d69e96b95249a253b8a7774aa8ba821ac49c390: Status 404 returned error can't find the container with id 3c228ada8b883005eebe18993d69e96b95249a253b8a7774aa8ba821ac49c390
Nov 02 15:07:34.358461 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:07:34.358434    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-5113c2/premium-w6-5113c2-kserve-b84945cb5-pc6f8" event={"ID":"9fb3e992-7bb0-497a-a82c-6e396ba241c1","Type":"ContainerStarted","Data":"3c228ada8b883005eebe18993d69e96b95249a253b8a7774aa8ba821ac49c390"}
Nov 02 15:07:35.364735 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:07:35.364703    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-5113c2/premium-w6-5113c2-kserve-b84945cb5-pc6f8" event={"ID":"9fb3e992-7bb0-497a-a82c-6e396ba241c1","Type":"ContainerStarted","Data":"df0454f8110d738a12ee04ac4b3f682c922588634410c0ac973c7be000df702e"}
Nov 02 15:07:35.406603 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:07:35.406548    5304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-5113c2/premium-w6-5113c2-kserve-b84945cb5-pc6f8" podStartSLOduration=2.40653269 podStartE2EDuration="2.40653269s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:07:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:07:35.405669528 +0000 UTC m=+655.399493936" watchObservedRunningTime="2026-11-02 15:07:35.40653269 +0000 UTC m=+655.400357098"
Nov 02 15:07:35.989348 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:07:35.989316    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-5113c2/premium-w6-5113c2-kserve-b84945cb5-pc6f8"
Nov 02 15:07:43.989737 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:07:43.989699    5304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-5113c2/premium-w6-5113c2-kserve-b84945cb5-pc6f8"
Nov 02 15:07:44.002993 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:07:44.002960    5304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-5113c2/premium-w6-5113c2-kserve-b84945cb5-pc6f8"
Nov 02 15:07:44.480153 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:07:44.476453    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-5113c2/premium-w6-5113c2-kserve-b84945cb5-pc6f8"
Nov 02 15:07:54.798063 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:07:54.798027    5304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k6fjk" containerName="extractor"
Nov 02 15:08:24.798456 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:08:24.798420    5304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k6fjk" containerName="extractor"
Nov 02 15:08:41.312265 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:08:41.312229    5304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-7ecfe1/distinct-w4-7ecfe1-kserve-6c847c647d-d2jvv"]
Nov 02 15:08:41.315700 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:08:41.315676    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-7ecfe1/distinct-w4-7ecfe1-kserve-6c847c647d-d2jvv"
Nov 02 15:08:41.334249 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:08:41.334221    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-7ecfe1/distinct-w4-7ecfe1-kserve-6c847c647d-d2jvv"]
Nov 02 15:08:41.359163 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:08:41.357561    5304 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-7ecfe1\"/\"openshift-service-ca.crt\""
Nov 02 15:08:41.359163 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:08:41.358659    5304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-7ecfe1\"/\"default-dockercfg-2cv7c\""
Nov 02 15:08:41.359163 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:08:41.358775    5304 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-7ecfe1\"/\"kube-root-ca.crt\""
Nov 02 15:08:41.364151 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:08:41.360163    5304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-7ecfe1\"/\"distinct-w4-7ecfe1-kserve-self-signed-certs\""
Nov 02 15:08:41.500756 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:08:41.500723    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f1f1ca5e-cca7-4c54-939f-06f095defcdd-dshm\") pod \"distinct-w4-7ecfe1-kserve-6c847c647d-d2jvv\" (UID: \"f1f1ca5e-cca7-4c54-939f-06f095defcdd\") " pod="e2e-models-e2e-worker-w4-7ecfe1/distinct-w4-7ecfe1-kserve-6c847c647d-d2jvv"
Nov 02 15:08:41.500940 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:08:41.500781    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f1f1ca5e-cca7-4c54-939f-06f095defcdd-home\") pod \"distinct-w4-7ecfe1-kserve-6c847c647d-d2jvv\" (UID: \"f1f1ca5e-cca7-4c54-939f-06f095defcdd\") " pod="e2e-models-e2e-worker-w4-7ecfe1/distinct-w4-7ecfe1-kserve-6c847c647d-d2jvv"
Nov 02 15:08:41.500940 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:08:41.500898    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f1f1ca5e-cca7-4c54-939f-06f095defcdd-model-cache\") pod \"distinct-w4-7ecfe1-kserve-6c847c647d-d2jvv\" (UID: \"f1f1ca5e-cca7-4c54-939f-06f095defcdd\") " pod="e2e-models-e2e-worker-w4-7ecfe1/distinct-w4-7ecfe1-kserve-6c847c647d-d2jvv"
Nov 02 15:08:41.500940 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:08:41.500924    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f1f1ca5e-cca7-4c54-939f-06f095defcdd-tls-certs\") pod \"distinct-w4-7ecfe1-kserve-6c847c647d-d2jvv\" (UID: \"f1f1ca5e-cca7-4c54-939f-06f095defcdd\") " pod="e2e-models-e2e-worker-w4-7ecfe1/distinct-w4-7ecfe1-kserve-6c847c647d-d2jvv"
Nov 02 15:08:41.501041 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:08:41.501022    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f1f1ca5e-cca7-4c54-939f-06f095defcdd-tmp-dir\") pod \"distinct-w4-7ecfe1-kserve-6c847c647d-d2jvv\" (UID: \"f1f1ca5e-cca7-4c54-939f-06f095defcdd\") " pod="e2e-models-e2e-worker-w4-7ecfe1/distinct-w4-7ecfe1-kserve-6c847c647d-d2jvv"
Nov 02 15:08:41.501075 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:08:41.501064    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jrdpj\" (UniqueName: \"kubernetes.io/projected/f1f1ca5e-cca7-4c54-939f-06f095defcdd-kube-api-access-jrdpj\") pod \"distinct-w4-7ecfe1-kserve-6c847c647d-d2jvv\" (UID: \"f1f1ca5e-cca7-4c54-939f-06f095defcdd\") " pod="e2e-models-e2e-worker-w4-7ecfe1/distinct-w4-7ecfe1-kserve-6c847c647d-d2jvv"
Nov 02 15:08:41.609262 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:08:41.609214    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f1f1ca5e-cca7-4c54-939f-06f095defcdd-home\") pod \"distinct-w4-7ecfe1-kserve-6c847c647d-d2jvv\" (UID: \"f1f1ca5e-cca7-4c54-939f-06f095defcdd\") " pod="e2e-models-e2e-worker-w4-7ecfe1/distinct-w4-7ecfe1-kserve-6c847c647d-d2jvv"
Nov 02 15:08:41.609459 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:08:41.609420    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f1f1ca5e-cca7-4c54-939f-06f095defcdd-model-cache\") pod \"distinct-w4-7ecfe1-kserve-6c847c647d-d2jvv\" (UID: \"f1f1ca5e-cca7-4c54-939f-06f095defcdd\") " pod="e2e-models-e2e-worker-w4-7ecfe1/distinct-w4-7ecfe1-kserve-6c847c647d-d2jvv"
Nov 02 15:08:41.609537 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:08:41.609469    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f1f1ca5e-cca7-4c54-939f-06f095defcdd-tls-certs\") pod \"distinct-w4-7ecfe1-kserve-6c847c647d-d2jvv\" (UID: \"f1f1ca5e-cca7-4c54-939f-06f095defcdd\") " pod="e2e-models-e2e-worker-w4-7ecfe1/distinct-w4-7ecfe1-kserve-6c847c647d-d2jvv"
Nov 02 15:08:41.609597 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:08:41.609542    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f1f1ca5e-cca7-4c54-939f-06f095defcdd-tmp-dir\") pod \"distinct-w4-7ecfe1-kserve-6c847c647d-d2jvv\" (UID: \"f1f1ca5e-cca7-4c54-939f-06f095defcdd\") " pod="e2e-models-e2e-worker-w4-7ecfe1/distinct-w4-7ecfe1-kserve-6c847c647d-d2jvv"
Nov 02 15:08:41.609664 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:08:41.609595    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jrdpj\" (UniqueName: \"kubernetes.io/projected/f1f1ca5e-cca7-4c54-939f-06f095defcdd-kube-api-access-jrdpj\") pod \"distinct-w4-7ecfe1-kserve-6c847c647d-d2jvv\" (UID: \"f1f1ca5e-cca7-4c54-939f-06f095defcdd\") " pod="e2e-models-e2e-worker-w4-7ecfe1/distinct-w4-7ecfe1-kserve-6c847c647d-d2jvv"
Nov 02 15:08:41.609724 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:08:41.609687    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f1f1ca5e-cca7-4c54-939f-06f095defcdd-dshm\") pod \"distinct-w4-7ecfe1-kserve-6c847c647d-d2jvv\" (UID: \"f1f1ca5e-cca7-4c54-939f-06f095defcdd\") " pod="e2e-models-e2e-worker-w4-7ecfe1/distinct-w4-7ecfe1-kserve-6c847c647d-d2jvv"
Nov 02 15:08:41.611628 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:08:41.611299    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f1f1ca5e-cca7-4c54-939f-06f095defcdd-home\") pod \"distinct-w4-7ecfe1-kserve-6c847c647d-d2jvv\" (UID: \"f1f1ca5e-cca7-4c54-939f-06f095defcdd\") " pod="e2e-models-e2e-worker-w4-7ecfe1/distinct-w4-7ecfe1-kserve-6c847c647d-d2jvv"
Nov 02 15:08:41.611628 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:08:41.611356    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f1f1ca5e-cca7-4c54-939f-06f095defcdd-tmp-dir\") pod \"distinct-w4-7ecfe1-kserve-6c847c647d-d2jvv\" (UID: \"f1f1ca5e-cca7-4c54-939f-06f095defcdd\") " pod="e2e-models-e2e-worker-w4-7ecfe1/distinct-w4-7ecfe1-kserve-6c847c647d-d2jvv"
Nov 02 15:08:41.611628 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:08:41.611586    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f1f1ca5e-cca7-4c54-939f-06f095defcdd-model-cache\") pod \"distinct-w4-7ecfe1-kserve-6c847c647d-d2jvv\" (UID: \"f1f1ca5e-cca7-4c54-939f-06f095defcdd\") " pod="e2e-models-e2e-worker-w4-7ecfe1/distinct-w4-7ecfe1-kserve-6c847c647d-d2jvv"
Nov 02 15:08:41.613785 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:08:41.613762    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f1f1ca5e-cca7-4c54-939f-06f095defcdd-dshm\") pod \"distinct-w4-7ecfe1-kserve-6c847c647d-d2jvv\" (UID: \"f1f1ca5e-cca7-4c54-939f-06f095defcdd\") " pod="e2e-models-e2e-worker-w4-7ecfe1/distinct-w4-7ecfe1-kserve-6c847c647d-d2jvv"
Nov 02 15:08:41.614873 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:08:41.614847    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f1f1ca5e-cca7-4c54-939f-06f095defcdd-tls-certs\") pod \"distinct-w4-7ecfe1-kserve-6c847c647d-d2jvv\" (UID: \"f1f1ca5e-cca7-4c54-939f-06f095defcdd\") " pod="e2e-models-e2e-worker-w4-7ecfe1/distinct-w4-7ecfe1-kserve-6c847c647d-d2jvv"
Nov 02 15:08:41.643150 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:08:41.643072    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jrdpj\" (UniqueName: \"kubernetes.io/projected/f1f1ca5e-cca7-4c54-939f-06f095defcdd-kube-api-access-jrdpj\") pod \"distinct-w4-7ecfe1-kserve-6c847c647d-d2jvv\" (UID: \"f1f1ca5e-cca7-4c54-939f-06f095defcdd\") " pod="e2e-models-e2e-worker-w4-7ecfe1/distinct-w4-7ecfe1-kserve-6c847c647d-d2jvv"
Nov 02 15:08:41.929795 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:08:41.929706    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-7ecfe1/distinct-w4-7ecfe1-kserve-6c847c647d-d2jvv"
Nov 02 15:08:42.240569 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:08:42.240488    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-7ecfe1/distinct-w4-7ecfe1-kserve-6c847c647d-d2jvv"]
Nov 02 15:08:42.244276 ip-10-0-135-138 kubenswrapper[5304]: W1102 15:08:42.244250    5304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf1f1ca5e_cca7_4c54_939f_06f095defcdd.slice/crio-65bcb65d729e2e3cf76abff90ed7d7b84bd240495edaed7f8ddfad0539e0d27f WatchSource:0}: Error finding container 65bcb65d729e2e3cf76abff90ed7d7b84bd240495edaed7f8ddfad0539e0d27f: Status 404 returned error can't find the container with id 65bcb65d729e2e3cf76abff90ed7d7b84bd240495edaed7f8ddfad0539e0d27f
Nov 02 15:08:42.923728 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:08:42.923686    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-7ecfe1/distinct-w4-7ecfe1-kserve-6c847c647d-d2jvv" event={"ID":"f1f1ca5e-cca7-4c54-939f-06f095defcdd","Type":"ContainerStarted","Data":"6c2db364224fb63629fd94c848db97a5b531496d14e8d28e482fe1cc74c4089b"}
Nov 02 15:08:42.923728 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:08:42.923731    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-7ecfe1/distinct-w4-7ecfe1-kserve-6c847c647d-d2jvv" event={"ID":"f1f1ca5e-cca7-4c54-939f-06f095defcdd","Type":"ContainerStarted","Data":"65bcb65d729e2e3cf76abff90ed7d7b84bd240495edaed7f8ddfad0539e0d27f"}
Nov 02 15:08:42.930848 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:08:42.930813    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-7ecfe1/distinct-w4-7ecfe1-kserve-6c847c647d-d2jvv"
Nov 02 15:08:42.975840 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:08:42.975792    5304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-7ecfe1/distinct-w4-7ecfe1-kserve-6c847c647d-d2jvv" podStartSLOduration=1.9757805670000002 podStartE2EDuration="1.975780567s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:08:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:08:42.975187246 +0000 UTC m=+722.969011657" watchObservedRunningTime="2026-11-02 15:08:42.975780567 +0000 UTC m=+722.969604975"
Nov 02 15:08:51.930422 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:08:51.930340    5304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-7ecfe1/distinct-w4-7ecfe1-kserve-6c847c647d-d2jvv"
Nov 02 15:08:51.943291 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:08:51.943266    5304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-7ecfe1/distinct-w4-7ecfe1-kserve-6c847c647d-d2jvv"
Nov 02 15:08:52.040421 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:08:52.040381    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-7ecfe1/distinct-w4-7ecfe1-kserve-6c847c647d-d2jvv"
Nov 02 15:08:54.798947 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:08:54.798904    5304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k6fjk" containerName="extractor"
Nov 02 15:08:59.497433 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:08:59.497401    5304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-6af50f/distinct-w0-6af50f-kserve-c6d994647-hbj9z"]
Nov 02 15:08:59.501187 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:08:59.501164    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-6af50f/distinct-w0-6af50f-kserve-c6d994647-hbj9z"
Nov 02 15:08:59.519182 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:08:59.519077    5304 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-6af50f\"/\"kube-root-ca.crt\""
Nov 02 15:08:59.519182 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:08:59.519082    5304 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-6af50f\"/\"openshift-service-ca.crt\""
Nov 02 15:08:59.531077 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:08:59.531047    5304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-6af50f\"/\"distinct-w0-6af50f-kserve-self-signed-certs\""
Nov 02 15:08:59.531275 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:08:59.531117    5304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-6af50f\"/\"default-dockercfg-thbcx\""
Nov 02 15:08:59.533070 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:08:59.533041    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-6af50f/distinct-w0-6af50f-kserve-c6d994647-hbj9z"]
Nov 02 15:08:59.593480 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:08:59.593450    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3dca5bf3-8e08-47a0-936e-205670d59ede-tmp-dir\") pod \"distinct-w0-6af50f-kserve-c6d994647-hbj9z\" (UID: \"3dca5bf3-8e08-47a0-936e-205670d59ede\") " pod="e2e-models-e2e-worker-w0-6af50f/distinct-w0-6af50f-kserve-c6d994647-hbj9z"
Nov 02 15:08:59.593653 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:08:59.593503    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3dca5bf3-8e08-47a0-936e-205670d59ede-tls-certs\") pod \"distinct-w0-6af50f-kserve-c6d994647-hbj9z\" (UID: \"3dca5bf3-8e08-47a0-936e-205670d59ede\") " pod="e2e-models-e2e-worker-w0-6af50f/distinct-w0-6af50f-kserve-c6d994647-hbj9z"
Nov 02 15:08:59.593653 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:08:59.593600    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3dca5bf3-8e08-47a0-936e-205670d59ede-dshm\") pod \"distinct-w0-6af50f-kserve-c6d994647-hbj9z\" (UID: \"3dca5bf3-8e08-47a0-936e-205670d59ede\") " pod="e2e-models-e2e-worker-w0-6af50f/distinct-w0-6af50f-kserve-c6d994647-hbj9z"
Nov 02 15:08:59.593653 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:08:59.593629    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3dca5bf3-8e08-47a0-936e-205670d59ede-model-cache\") pod \"distinct-w0-6af50f-kserve-c6d994647-hbj9z\" (UID: \"3dca5bf3-8e08-47a0-936e-205670d59ede\") " pod="e2e-models-e2e-worker-w0-6af50f/distinct-w0-6af50f-kserve-c6d994647-hbj9z"
Nov 02 15:08:59.593754 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:08:59.593732    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3dca5bf3-8e08-47a0-936e-205670d59ede-home\") pod \"distinct-w0-6af50f-kserve-c6d994647-hbj9z\" (UID: \"3dca5bf3-8e08-47a0-936e-205670d59ede\") " pod="e2e-models-e2e-worker-w0-6af50f/distinct-w0-6af50f-kserve-c6d994647-hbj9z"
Nov 02 15:08:59.593788 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:08:59.593753    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zxs7c\" (UniqueName: \"kubernetes.io/projected/3dca5bf3-8e08-47a0-936e-205670d59ede-kube-api-access-zxs7c\") pod \"distinct-w0-6af50f-kserve-c6d994647-hbj9z\" (UID: \"3dca5bf3-8e08-47a0-936e-205670d59ede\") " pod="e2e-models-e2e-worker-w0-6af50f/distinct-w0-6af50f-kserve-c6d994647-hbj9z"
Nov 02 15:08:59.697589 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:08:59.697542    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3dca5bf3-8e08-47a0-936e-205670d59ede-tls-certs\") pod \"distinct-w0-6af50f-kserve-c6d994647-hbj9z\" (UID: \"3dca5bf3-8e08-47a0-936e-205670d59ede\") " pod="e2e-models-e2e-worker-w0-6af50f/distinct-w0-6af50f-kserve-c6d994647-hbj9z"
Nov 02 15:08:59.697830 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:08:59.697610    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3dca5bf3-8e08-47a0-936e-205670d59ede-dshm\") pod \"distinct-w0-6af50f-kserve-c6d994647-hbj9z\" (UID: \"3dca5bf3-8e08-47a0-936e-205670d59ede\") " pod="e2e-models-e2e-worker-w0-6af50f/distinct-w0-6af50f-kserve-c6d994647-hbj9z"
Nov 02 15:08:59.697830 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:08:59.697636    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3dca5bf3-8e08-47a0-936e-205670d59ede-model-cache\") pod \"distinct-w0-6af50f-kserve-c6d994647-hbj9z\" (UID: \"3dca5bf3-8e08-47a0-936e-205670d59ede\") " pod="e2e-models-e2e-worker-w0-6af50f/distinct-w0-6af50f-kserve-c6d994647-hbj9z"
Nov 02 15:08:59.697830 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:08:59.697706    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3dca5bf3-8e08-47a0-936e-205670d59ede-home\") pod \"distinct-w0-6af50f-kserve-c6d994647-hbj9z\" (UID: \"3dca5bf3-8e08-47a0-936e-205670d59ede\") " pod="e2e-models-e2e-worker-w0-6af50f/distinct-w0-6af50f-kserve-c6d994647-hbj9z"
Nov 02 15:08:59.697830 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:08:59.697737    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zxs7c\" (UniqueName: \"kubernetes.io/projected/3dca5bf3-8e08-47a0-936e-205670d59ede-kube-api-access-zxs7c\") pod \"distinct-w0-6af50f-kserve-c6d994647-hbj9z\" (UID: \"3dca5bf3-8e08-47a0-936e-205670d59ede\") " pod="e2e-models-e2e-worker-w0-6af50f/distinct-w0-6af50f-kserve-c6d994647-hbj9z"
Nov 02 15:08:59.697830 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:08:59.697824    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3dca5bf3-8e08-47a0-936e-205670d59ede-tmp-dir\") pod \"distinct-w0-6af50f-kserve-c6d994647-hbj9z\" (UID: \"3dca5bf3-8e08-47a0-936e-205670d59ede\") " pod="e2e-models-e2e-worker-w0-6af50f/distinct-w0-6af50f-kserve-c6d994647-hbj9z"
Nov 02 15:08:59.698463 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:08:59.698415    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3dca5bf3-8e08-47a0-936e-205670d59ede-tmp-dir\") pod \"distinct-w0-6af50f-kserve-c6d994647-hbj9z\" (UID: \"3dca5bf3-8e08-47a0-936e-205670d59ede\") " pod="e2e-models-e2e-worker-w0-6af50f/distinct-w0-6af50f-kserve-c6d994647-hbj9z"
Nov 02 15:08:59.698618 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:08:59.698486    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3dca5bf3-8e08-47a0-936e-205670d59ede-home\") pod \"distinct-w0-6af50f-kserve-c6d994647-hbj9z\" (UID: \"3dca5bf3-8e08-47a0-936e-205670d59ede\") " pod="e2e-models-e2e-worker-w0-6af50f/distinct-w0-6af50f-kserve-c6d994647-hbj9z"
Nov 02 15:08:59.698768 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:08:59.698746    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3dca5bf3-8e08-47a0-936e-205670d59ede-model-cache\") pod \"distinct-w0-6af50f-kserve-c6d994647-hbj9z\" (UID: \"3dca5bf3-8e08-47a0-936e-205670d59ede\") " pod="e2e-models-e2e-worker-w0-6af50f/distinct-w0-6af50f-kserve-c6d994647-hbj9z"
Nov 02 15:08:59.701634 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:08:59.701598    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3dca5bf3-8e08-47a0-936e-205670d59ede-dshm\") pod \"distinct-w0-6af50f-kserve-c6d994647-hbj9z\" (UID: \"3dca5bf3-8e08-47a0-936e-205670d59ede\") " pod="e2e-models-e2e-worker-w0-6af50f/distinct-w0-6af50f-kserve-c6d994647-hbj9z"
Nov 02 15:08:59.713229 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:08:59.713193    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3dca5bf3-8e08-47a0-936e-205670d59ede-tls-certs\") pod \"distinct-w0-6af50f-kserve-c6d994647-hbj9z\" (UID: \"3dca5bf3-8e08-47a0-936e-205670d59ede\") " pod="e2e-models-e2e-worker-w0-6af50f/distinct-w0-6af50f-kserve-c6d994647-hbj9z"
Nov 02 15:08:59.721675 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:08:59.721635    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zxs7c\" (UniqueName: \"kubernetes.io/projected/3dca5bf3-8e08-47a0-936e-205670d59ede-kube-api-access-zxs7c\") pod \"distinct-w0-6af50f-kserve-c6d994647-hbj9z\" (UID: \"3dca5bf3-8e08-47a0-936e-205670d59ede\") " pod="e2e-models-e2e-worker-w0-6af50f/distinct-w0-6af50f-kserve-c6d994647-hbj9z"
Nov 02 15:08:59.817351 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:08:59.817266    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-6af50f/distinct-w0-6af50f-kserve-c6d994647-hbj9z"
Nov 02 15:09:00.127645 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:00.127613    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-6af50f/distinct-w0-6af50f-kserve-c6d994647-hbj9z"]
Nov 02 15:09:00.128444 ip-10-0-135-138 kubenswrapper[5304]: W1102 15:09:00.128419    5304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3dca5bf3_8e08_47a0_936e_205670d59ede.slice/crio-b2e1da033c45801a5beba3421a1deeb7613441bd1df7e6a86d456065404bde69 WatchSource:0}: Error finding container b2e1da033c45801a5beba3421a1deeb7613441bd1df7e6a86d456065404bde69: Status 404 returned error can't find the container with id b2e1da033c45801a5beba3421a1deeb7613441bd1df7e6a86d456065404bde69
Nov 02 15:09:01.082091 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:01.082049    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-6af50f/distinct-w0-6af50f-kserve-c6d994647-hbj9z" event={"ID":"3dca5bf3-8e08-47a0-936e-205670d59ede","Type":"ContainerStarted","Data":"5b732982a0835eb5b45687c64c425c6d675c98581f1ec840e03e88464373b88e"}
Nov 02 15:09:01.082091 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:01.082094    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-6af50f/distinct-w0-6af50f-kserve-c6d994647-hbj9z" event={"ID":"3dca5bf3-8e08-47a0-936e-205670d59ede","Type":"ContainerStarted","Data":"b2e1da033c45801a5beba3421a1deeb7613441bd1df7e6a86d456065404bde69"}
Nov 02 15:09:01.127097 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:01.127032    5304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-6af50f/distinct-w0-6af50f-kserve-c6d994647-hbj9z" podStartSLOduration=2.127012906 podStartE2EDuration="2.127012906s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:08:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:09:01.126233697 +0000 UTC m=+741.120058107" watchObservedRunningTime="2026-11-02 15:09:01.127012906 +0000 UTC m=+741.120837316"
Nov 02 15:09:01.818024 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:01.817996    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-6af50f/distinct-w0-6af50f-kserve-c6d994647-hbj9z"
Nov 02 15:09:02.746735 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:02.746690    5304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-1cc1b2/distinct-w5-1cc1b2-kserve-5b7459bc6d-zl8xr"]
Nov 02 15:09:02.750569 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:02.750541    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-1cc1b2/distinct-w5-1cc1b2-kserve-5b7459bc6d-zl8xr"
Nov 02 15:09:02.770845 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:02.770814    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-1cc1b2/distinct-w5-1cc1b2-kserve-5b7459bc6d-zl8xr"]
Nov 02 15:09:02.778684 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:02.778657    5304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-1cc1b2\"/\"distinct-w5-1cc1b2-kserve-self-signed-certs\""
Nov 02 15:09:02.933188 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:02.933109    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/211ad15d-9234-4f57-b1b9-181dd2d74208-home\") pod \"distinct-w5-1cc1b2-kserve-5b7459bc6d-zl8xr\" (UID: \"211ad15d-9234-4f57-b1b9-181dd2d74208\") " pod="e2e-models-e2e-worker-w5-1cc1b2/distinct-w5-1cc1b2-kserve-5b7459bc6d-zl8xr"
Nov 02 15:09:02.933534 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:02.933502    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/211ad15d-9234-4f57-b1b9-181dd2d74208-model-cache\") pod \"distinct-w5-1cc1b2-kserve-5b7459bc6d-zl8xr\" (UID: \"211ad15d-9234-4f57-b1b9-181dd2d74208\") " pod="e2e-models-e2e-worker-w5-1cc1b2/distinct-w5-1cc1b2-kserve-5b7459bc6d-zl8xr"
Nov 02 15:09:02.933731 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:02.933705    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/211ad15d-9234-4f57-b1b9-181dd2d74208-dshm\") pod \"distinct-w5-1cc1b2-kserve-5b7459bc6d-zl8xr\" (UID: \"211ad15d-9234-4f57-b1b9-181dd2d74208\") " pod="e2e-models-e2e-worker-w5-1cc1b2/distinct-w5-1cc1b2-kserve-5b7459bc6d-zl8xr"
Nov 02 15:09:02.933799 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:02.933767    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gdwqm\" (UniqueName: \"kubernetes.io/projected/211ad15d-9234-4f57-b1b9-181dd2d74208-kube-api-access-gdwqm\") pod \"distinct-w5-1cc1b2-kserve-5b7459bc6d-zl8xr\" (UID: \"211ad15d-9234-4f57-b1b9-181dd2d74208\") " pod="e2e-models-e2e-worker-w5-1cc1b2/distinct-w5-1cc1b2-kserve-5b7459bc6d-zl8xr"
Nov 02 15:09:02.933852 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:02.933829    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/211ad15d-9234-4f57-b1b9-181dd2d74208-tls-certs\") pod \"distinct-w5-1cc1b2-kserve-5b7459bc6d-zl8xr\" (UID: \"211ad15d-9234-4f57-b1b9-181dd2d74208\") " pod="e2e-models-e2e-worker-w5-1cc1b2/distinct-w5-1cc1b2-kserve-5b7459bc6d-zl8xr"
Nov 02 15:09:02.934020 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:02.933985    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/211ad15d-9234-4f57-b1b9-181dd2d74208-tmp-dir\") pod \"distinct-w5-1cc1b2-kserve-5b7459bc6d-zl8xr\" (UID: \"211ad15d-9234-4f57-b1b9-181dd2d74208\") " pod="e2e-models-e2e-worker-w5-1cc1b2/distinct-w5-1cc1b2-kserve-5b7459bc6d-zl8xr"
Nov 02 15:09:03.035641 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:03.035552    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/211ad15d-9234-4f57-b1b9-181dd2d74208-dshm\") pod \"distinct-w5-1cc1b2-kserve-5b7459bc6d-zl8xr\" (UID: \"211ad15d-9234-4f57-b1b9-181dd2d74208\") " pod="e2e-models-e2e-worker-w5-1cc1b2/distinct-w5-1cc1b2-kserve-5b7459bc6d-zl8xr"
Nov 02 15:09:03.035641 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:03.035592    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gdwqm\" (UniqueName: \"kubernetes.io/projected/211ad15d-9234-4f57-b1b9-181dd2d74208-kube-api-access-gdwqm\") pod \"distinct-w5-1cc1b2-kserve-5b7459bc6d-zl8xr\" (UID: \"211ad15d-9234-4f57-b1b9-181dd2d74208\") " pod="e2e-models-e2e-worker-w5-1cc1b2/distinct-w5-1cc1b2-kserve-5b7459bc6d-zl8xr"
Nov 02 15:09:03.035641 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:03.035610    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/211ad15d-9234-4f57-b1b9-181dd2d74208-tls-certs\") pod \"distinct-w5-1cc1b2-kserve-5b7459bc6d-zl8xr\" (UID: \"211ad15d-9234-4f57-b1b9-181dd2d74208\") " pod="e2e-models-e2e-worker-w5-1cc1b2/distinct-w5-1cc1b2-kserve-5b7459bc6d-zl8xr"
Nov 02 15:09:03.035920 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:03.035690    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/211ad15d-9234-4f57-b1b9-181dd2d74208-tmp-dir\") pod \"distinct-w5-1cc1b2-kserve-5b7459bc6d-zl8xr\" (UID: \"211ad15d-9234-4f57-b1b9-181dd2d74208\") " pod="e2e-models-e2e-worker-w5-1cc1b2/distinct-w5-1cc1b2-kserve-5b7459bc6d-zl8xr"
Nov 02 15:09:03.035920 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:03.035748    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/211ad15d-9234-4f57-b1b9-181dd2d74208-home\") pod \"distinct-w5-1cc1b2-kserve-5b7459bc6d-zl8xr\" (UID: \"211ad15d-9234-4f57-b1b9-181dd2d74208\") " pod="e2e-models-e2e-worker-w5-1cc1b2/distinct-w5-1cc1b2-kserve-5b7459bc6d-zl8xr"
Nov 02 15:09:03.035920 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:03.035781    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/211ad15d-9234-4f57-b1b9-181dd2d74208-model-cache\") pod \"distinct-w5-1cc1b2-kserve-5b7459bc6d-zl8xr\" (UID: \"211ad15d-9234-4f57-b1b9-181dd2d74208\") " pod="e2e-models-e2e-worker-w5-1cc1b2/distinct-w5-1cc1b2-kserve-5b7459bc6d-zl8xr"
Nov 02 15:09:03.036235 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:03.036205    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/211ad15d-9234-4f57-b1b9-181dd2d74208-model-cache\") pod \"distinct-w5-1cc1b2-kserve-5b7459bc6d-zl8xr\" (UID: \"211ad15d-9234-4f57-b1b9-181dd2d74208\") " pod="e2e-models-e2e-worker-w5-1cc1b2/distinct-w5-1cc1b2-kserve-5b7459bc6d-zl8xr"
Nov 02 15:09:03.036390 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:03.036246    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/211ad15d-9234-4f57-b1b9-181dd2d74208-tmp-dir\") pod \"distinct-w5-1cc1b2-kserve-5b7459bc6d-zl8xr\" (UID: \"211ad15d-9234-4f57-b1b9-181dd2d74208\") " pod="e2e-models-e2e-worker-w5-1cc1b2/distinct-w5-1cc1b2-kserve-5b7459bc6d-zl8xr"
Nov 02 15:09:03.036390 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:03.036335    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/211ad15d-9234-4f57-b1b9-181dd2d74208-home\") pod \"distinct-w5-1cc1b2-kserve-5b7459bc6d-zl8xr\" (UID: \"211ad15d-9234-4f57-b1b9-181dd2d74208\") " pod="e2e-models-e2e-worker-w5-1cc1b2/distinct-w5-1cc1b2-kserve-5b7459bc6d-zl8xr"
Nov 02 15:09:03.038643 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:03.038616    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/211ad15d-9234-4f57-b1b9-181dd2d74208-tls-certs\") pod \"distinct-w5-1cc1b2-kserve-5b7459bc6d-zl8xr\" (UID: \"211ad15d-9234-4f57-b1b9-181dd2d74208\") " pod="e2e-models-e2e-worker-w5-1cc1b2/distinct-w5-1cc1b2-kserve-5b7459bc6d-zl8xr"
Nov 02 15:09:03.039351 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:03.039331    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/211ad15d-9234-4f57-b1b9-181dd2d74208-dshm\") pod \"distinct-w5-1cc1b2-kserve-5b7459bc6d-zl8xr\" (UID: \"211ad15d-9234-4f57-b1b9-181dd2d74208\") " pod="e2e-models-e2e-worker-w5-1cc1b2/distinct-w5-1cc1b2-kserve-5b7459bc6d-zl8xr"
Nov 02 15:09:03.059210 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:03.059181    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gdwqm\" (UniqueName: \"kubernetes.io/projected/211ad15d-9234-4f57-b1b9-181dd2d74208-kube-api-access-gdwqm\") pod \"distinct-w5-1cc1b2-kserve-5b7459bc6d-zl8xr\" (UID: \"211ad15d-9234-4f57-b1b9-181dd2d74208\") " pod="e2e-models-e2e-worker-w5-1cc1b2/distinct-w5-1cc1b2-kserve-5b7459bc6d-zl8xr"
Nov 02 15:09:03.066053 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:03.066027    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-1cc1b2/distinct-w5-1cc1b2-kserve-5b7459bc6d-zl8xr"
Nov 02 15:09:03.365939 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:03.365907    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-1cc1b2/distinct-w5-1cc1b2-kserve-5b7459bc6d-zl8xr"]
Nov 02 15:09:03.369806 ip-10-0-135-138 kubenswrapper[5304]: W1102 15:09:03.369779    5304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod211ad15d_9234_4f57_b1b9_181dd2d74208.slice/crio-e2343f1c408a18a75daf42cea56518aa651372f0a8bbcce11824b7c65b63f3bf WatchSource:0}: Error finding container e2343f1c408a18a75daf42cea56518aa651372f0a8bbcce11824b7c65b63f3bf: Status 404 returned error can't find the container with id e2343f1c408a18a75daf42cea56518aa651372f0a8bbcce11824b7c65b63f3bf
Nov 02 15:09:04.099621 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:04.099577    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-1cc1b2/distinct-w5-1cc1b2-kserve-5b7459bc6d-zl8xr" event={"ID":"211ad15d-9234-4f57-b1b9-181dd2d74208","Type":"ContainerStarted","Data":"fa543fe9e33bbdbadbef9555f23bbc36535b92baa290827f7897eac73e8dd386"}
Nov 02 15:09:04.100073 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:04.099629    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-1cc1b2/distinct-w5-1cc1b2-kserve-5b7459bc6d-zl8xr" event={"ID":"211ad15d-9234-4f57-b1b9-181dd2d74208","Type":"ContainerStarted","Data":"e2343f1c408a18a75daf42cea56518aa651372f0a8bbcce11824b7c65b63f3bf"}
Nov 02 15:09:04.142690 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:04.142639    5304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-1cc1b2/distinct-w5-1cc1b2-kserve-5b7459bc6d-zl8xr" podStartSLOduration=2.142625072 podStartE2EDuration="2.142625072s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:09:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:09:04.140924368 +0000 UTC m=+744.134748776" watchObservedRunningTime="2026-11-02 15:09:04.142625072 +0000 UTC m=+744.136449505"
Nov 02 15:09:05.066341 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:05.066309    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-1cc1b2/distinct-w5-1cc1b2-kserve-5b7459bc6d-zl8xr"
Nov 02 15:09:06.873858 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:06.873828    5304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-4cc4m"]
Nov 02 15:09:06.877723 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:06.877700    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-4cc4m"
Nov 02 15:09:06.893728 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:06.893703    5304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-qm7zm"]
Nov 02 15:09:06.897285 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:06.897268    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-qm7zm"
Nov 02 15:09:06.907108 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:06.907077    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-4cc4m"]
Nov 02 15:09:06.932787 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:06.932745    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-qm7zm"]
Nov 02 15:09:06.980521 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:06.980488    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d0dd04f-b8ff-4ecd-a1fd-ec15ce4dea5a-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-qm7zm\" (UID: \"4d0dd04f-b8ff-4ecd-a1fd-ec15ce4dea5a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-qm7zm"
Nov 02 15:09:06.980691 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:06.980541    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af975b55-20fa-459f-8659-a7a69f2d2248-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-4cc4m\" (UID: \"af975b55-20fa-459f-8659-a7a69f2d2248\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-4cc4m"
Nov 02 15:09:06.980691 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:06.980566    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d0dd04f-b8ff-4ecd-a1fd-ec15ce4dea5a-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-qm7zm\" (UID: \"4d0dd04f-b8ff-4ecd-a1fd-ec15ce4dea5a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-qm7zm"
Nov 02 15:09:06.980691 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:06.980661    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d0dd04f-b8ff-4ecd-a1fd-ec15ce4dea5a-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-qm7zm\" (UID: \"4d0dd04f-b8ff-4ecd-a1fd-ec15ce4dea5a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-qm7zm"
Nov 02 15:09:06.980799 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:06.980695    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af975b55-20fa-459f-8659-a7a69f2d2248-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-4cc4m\" (UID: \"af975b55-20fa-459f-8659-a7a69f2d2248\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-4cc4m"
Nov 02 15:09:06.980799 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:06.980722    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af975b55-20fa-459f-8659-a7a69f2d2248-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-4cc4m\" (UID: \"af975b55-20fa-459f-8659-a7a69f2d2248\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-4cc4m"
Nov 02 15:09:06.980893 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:06.980809    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af975b55-20fa-459f-8659-a7a69f2d2248-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-4cc4m\" (UID: \"af975b55-20fa-459f-8659-a7a69f2d2248\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-4cc4m"
Nov 02 15:09:06.980893 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:06.980827    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af975b55-20fa-459f-8659-a7a69f2d2248-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-4cc4m\" (UID: \"af975b55-20fa-459f-8659-a7a69f2d2248\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-4cc4m"
Nov 02 15:09:06.980893 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:06.980842    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af975b55-20fa-459f-8659-a7a69f2d2248-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-4cc4m\" (UID: \"af975b55-20fa-459f-8659-a7a69f2d2248\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-4cc4m"
Nov 02 15:09:06.980893 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:06.980873    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pzqvg\" (UniqueName: \"kubernetes.io/projected/af975b55-20fa-459f-8659-a7a69f2d2248-kube-api-access-pzqvg\") pod \"maas-default-gateway-openshift-default-fcb75c45f-4cc4m\" (UID: \"af975b55-20fa-459f-8659-a7a69f2d2248\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-4cc4m"
Nov 02 15:09:06.981018 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:06.980905    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d0dd04f-b8ff-4ecd-a1fd-ec15ce4dea5a-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-qm7zm\" (UID: \"4d0dd04f-b8ff-4ecd-a1fd-ec15ce4dea5a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-qm7zm"
Nov 02 15:09:06.981018 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:06.980932    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d0dd04f-b8ff-4ecd-a1fd-ec15ce4dea5a-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-qm7zm\" (UID: \"4d0dd04f-b8ff-4ecd-a1fd-ec15ce4dea5a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-qm7zm"
Nov 02 15:09:06.981018 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:06.980972    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af975b55-20fa-459f-8659-a7a69f2d2248-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-4cc4m\" (UID: \"af975b55-20fa-459f-8659-a7a69f2d2248\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-4cc4m"
Nov 02 15:09:06.981018 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:06.981011    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af975b55-20fa-459f-8659-a7a69f2d2248-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-4cc4m\" (UID: \"af975b55-20fa-459f-8659-a7a69f2d2248\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-4cc4m"
Nov 02 15:09:06.981177 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:06.981033    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d0dd04f-b8ff-4ecd-a1fd-ec15ce4dea5a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-qm7zm\" (UID: \"4d0dd04f-b8ff-4ecd-a1fd-ec15ce4dea5a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-qm7zm"
Nov 02 15:09:06.981177 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:06.981052    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d0dd04f-b8ff-4ecd-a1fd-ec15ce4dea5a-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-qm7zm\" (UID: \"4d0dd04f-b8ff-4ecd-a1fd-ec15ce4dea5a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-qm7zm"
Nov 02 15:09:06.981177 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:06.981074    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d0dd04f-b8ff-4ecd-a1fd-ec15ce4dea5a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-qm7zm\" (UID: \"4d0dd04f-b8ff-4ecd-a1fd-ec15ce4dea5a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-qm7zm"
Nov 02 15:09:06.981274 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:06.981199    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9g4sx\" (UniqueName: \"kubernetes.io/projected/4d0dd04f-b8ff-4ecd-a1fd-ec15ce4dea5a-kube-api-access-9g4sx\") pod \"maas-default-gateway-openshift-default-fcb75c45f-qm7zm\" (UID: \"4d0dd04f-b8ff-4ecd-a1fd-ec15ce4dea5a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-qm7zm"
Nov 02 15:09:07.081774 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:07.081740    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d0dd04f-b8ff-4ecd-a1fd-ec15ce4dea5a-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-qm7zm\" (UID: \"4d0dd04f-b8ff-4ecd-a1fd-ec15ce4dea5a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-qm7zm"
Nov 02 15:09:07.081970 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:07.081787    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af975b55-20fa-459f-8659-a7a69f2d2248-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-4cc4m\" (UID: \"af975b55-20fa-459f-8659-a7a69f2d2248\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-4cc4m"
Nov 02 15:09:07.081970 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:07.081827    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af975b55-20fa-459f-8659-a7a69f2d2248-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-4cc4m\" (UID: \"af975b55-20fa-459f-8659-a7a69f2d2248\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-4cc4m"
Nov 02 15:09:07.081970 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:07.081854    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d0dd04f-b8ff-4ecd-a1fd-ec15ce4dea5a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-qm7zm\" (UID: \"4d0dd04f-b8ff-4ecd-a1fd-ec15ce4dea5a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-qm7zm"
Nov 02 15:09:07.081970 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:07.081880    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d0dd04f-b8ff-4ecd-a1fd-ec15ce4dea5a-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-qm7zm\" (UID: \"4d0dd04f-b8ff-4ecd-a1fd-ec15ce4dea5a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-qm7zm"
Nov 02 15:09:07.081970 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:07.081914    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d0dd04f-b8ff-4ecd-a1fd-ec15ce4dea5a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-qm7zm\" (UID: \"4d0dd04f-b8ff-4ecd-a1fd-ec15ce4dea5a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-qm7zm"
Nov 02 15:09:07.081970 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:07.081957    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9g4sx\" (UniqueName: \"kubernetes.io/projected/4d0dd04f-b8ff-4ecd-a1fd-ec15ce4dea5a-kube-api-access-9g4sx\") pod \"maas-default-gateway-openshift-default-fcb75c45f-qm7zm\" (UID: \"4d0dd04f-b8ff-4ecd-a1fd-ec15ce4dea5a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-qm7zm"
Nov 02 15:09:07.082486 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:07.082025    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d0dd04f-b8ff-4ecd-a1fd-ec15ce4dea5a-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-qm7zm\" (UID: \"4d0dd04f-b8ff-4ecd-a1fd-ec15ce4dea5a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-qm7zm"
Nov 02 15:09:07.082486 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:07.082084    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af975b55-20fa-459f-8659-a7a69f2d2248-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-4cc4m\" (UID: \"af975b55-20fa-459f-8659-a7a69f2d2248\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-4cc4m"
Nov 02 15:09:07.082486 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:07.082115    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d0dd04f-b8ff-4ecd-a1fd-ec15ce4dea5a-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-qm7zm\" (UID: \"4d0dd04f-b8ff-4ecd-a1fd-ec15ce4dea5a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-qm7zm"
Nov 02 15:09:07.082486 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:07.082180    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d0dd04f-b8ff-4ecd-a1fd-ec15ce4dea5a-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-qm7zm\" (UID: \"4d0dd04f-b8ff-4ecd-a1fd-ec15ce4dea5a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-qm7zm"
Nov 02 15:09:07.082486 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:07.082213    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af975b55-20fa-459f-8659-a7a69f2d2248-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-4cc4m\" (UID: \"af975b55-20fa-459f-8659-a7a69f2d2248\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-4cc4m"
Nov 02 15:09:07.082486 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:07.082251    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af975b55-20fa-459f-8659-a7a69f2d2248-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-4cc4m\" (UID: \"af975b55-20fa-459f-8659-a7a69f2d2248\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-4cc4m"
Nov 02 15:09:07.082486 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:07.082320    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af975b55-20fa-459f-8659-a7a69f2d2248-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-4cc4m\" (UID: \"af975b55-20fa-459f-8659-a7a69f2d2248\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-4cc4m"
Nov 02 15:09:07.082486 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:07.082320    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af975b55-20fa-459f-8659-a7a69f2d2248-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-4cc4m\" (UID: \"af975b55-20fa-459f-8659-a7a69f2d2248\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-4cc4m"
Nov 02 15:09:07.082486 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:07.082342    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af975b55-20fa-459f-8659-a7a69f2d2248-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-4cc4m\" (UID: \"af975b55-20fa-459f-8659-a7a69f2d2248\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-4cc4m"
Nov 02 15:09:07.082486 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:07.082387    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af975b55-20fa-459f-8659-a7a69f2d2248-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-4cc4m\" (UID: \"af975b55-20fa-459f-8659-a7a69f2d2248\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-4cc4m"
Nov 02 15:09:07.082486 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:07.082415    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pzqvg\" (UniqueName: \"kubernetes.io/projected/af975b55-20fa-459f-8659-a7a69f2d2248-kube-api-access-pzqvg\") pod \"maas-default-gateway-openshift-default-fcb75c45f-4cc4m\" (UID: \"af975b55-20fa-459f-8659-a7a69f2d2248\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-4cc4m"
Nov 02 15:09:07.082486 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:07.082453    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d0dd04f-b8ff-4ecd-a1fd-ec15ce4dea5a-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-qm7zm\" (UID: \"4d0dd04f-b8ff-4ecd-a1fd-ec15ce4dea5a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-qm7zm"
Nov 02 15:09:07.083148 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:07.082566    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d0dd04f-b8ff-4ecd-a1fd-ec15ce4dea5a-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-qm7zm\" (UID: \"4d0dd04f-b8ff-4ecd-a1fd-ec15ce4dea5a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-qm7zm"
Nov 02 15:09:07.083148 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:07.082582    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d0dd04f-b8ff-4ecd-a1fd-ec15ce4dea5a-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-qm7zm\" (UID: \"4d0dd04f-b8ff-4ecd-a1fd-ec15ce4dea5a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-qm7zm"
Nov 02 15:09:07.083148 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:07.082919    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d0dd04f-b8ff-4ecd-a1fd-ec15ce4dea5a-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-qm7zm\" (UID: \"4d0dd04f-b8ff-4ecd-a1fd-ec15ce4dea5a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-qm7zm"
Nov 02 15:09:07.083148 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:07.082934    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af975b55-20fa-459f-8659-a7a69f2d2248-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-4cc4m\" (UID: \"af975b55-20fa-459f-8659-a7a69f2d2248\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-4cc4m"
Nov 02 15:09:07.083382 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:07.083246    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d0dd04f-b8ff-4ecd-a1fd-ec15ce4dea5a-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-qm7zm\" (UID: \"4d0dd04f-b8ff-4ecd-a1fd-ec15ce4dea5a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-qm7zm"
Nov 02 15:09:07.083382 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:07.083300    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af975b55-20fa-459f-8659-a7a69f2d2248-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-4cc4m\" (UID: \"af975b55-20fa-459f-8659-a7a69f2d2248\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-4cc4m"
Nov 02 15:09:07.083500 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:07.083440    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af975b55-20fa-459f-8659-a7a69f2d2248-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-4cc4m\" (UID: \"af975b55-20fa-459f-8659-a7a69f2d2248\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-4cc4m"
Nov 02 15:09:07.083557 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:07.083500    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af975b55-20fa-459f-8659-a7a69f2d2248-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-4cc4m\" (UID: \"af975b55-20fa-459f-8659-a7a69f2d2248\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-4cc4m"
Nov 02 15:09:07.083876 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:07.083847    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d0dd04f-b8ff-4ecd-a1fd-ec15ce4dea5a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-qm7zm\" (UID: \"4d0dd04f-b8ff-4ecd-a1fd-ec15ce4dea5a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-qm7zm"
Nov 02 15:09:07.084507 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:07.084482    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d0dd04f-b8ff-4ecd-a1fd-ec15ce4dea5a-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-qm7zm\" (UID: \"4d0dd04f-b8ff-4ecd-a1fd-ec15ce4dea5a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-qm7zm"
Nov 02 15:09:07.084609 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:07.084548    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af975b55-20fa-459f-8659-a7a69f2d2248-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-4cc4m\" (UID: \"af975b55-20fa-459f-8659-a7a69f2d2248\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-4cc4m"
Nov 02 15:09:07.084887 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:07.084866    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d0dd04f-b8ff-4ecd-a1fd-ec15ce4dea5a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-qm7zm\" (UID: \"4d0dd04f-b8ff-4ecd-a1fd-ec15ce4dea5a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-qm7zm"
Nov 02 15:09:07.084957 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:07.084904    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af975b55-20fa-459f-8659-a7a69f2d2248-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-4cc4m\" (UID: \"af975b55-20fa-459f-8659-a7a69f2d2248\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-4cc4m"
Nov 02 15:09:07.104980 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:07.104957    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af975b55-20fa-459f-8659-a7a69f2d2248-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-4cc4m\" (UID: \"af975b55-20fa-459f-8659-a7a69f2d2248\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-4cc4m"
Nov 02 15:09:07.105115 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:07.105092    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d0dd04f-b8ff-4ecd-a1fd-ec15ce4dea5a-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-qm7zm\" (UID: \"4d0dd04f-b8ff-4ecd-a1fd-ec15ce4dea5a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-qm7zm"
Nov 02 15:09:07.105289 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:07.105269    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9g4sx\" (UniqueName: \"kubernetes.io/projected/4d0dd04f-b8ff-4ecd-a1fd-ec15ce4dea5a-kube-api-access-9g4sx\") pod \"maas-default-gateway-openshift-default-fcb75c45f-qm7zm\" (UID: \"4d0dd04f-b8ff-4ecd-a1fd-ec15ce4dea5a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-qm7zm"
Nov 02 15:09:07.105483 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:07.105463    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pzqvg\" (UniqueName: \"kubernetes.io/projected/af975b55-20fa-459f-8659-a7a69f2d2248-kube-api-access-pzqvg\") pod \"maas-default-gateway-openshift-default-fcb75c45f-4cc4m\" (UID: \"af975b55-20fa-459f-8659-a7a69f2d2248\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-4cc4m"
Nov 02 15:09:07.191764 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:07.191677    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-4cc4m"
Nov 02 15:09:07.213890 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:07.213861    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-qm7zm"
Nov 02 15:09:07.508359 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:07.508287    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-4cc4m"]
Nov 02 15:09:07.511796 ip-10-0-135-138 kubenswrapper[5304]: W1102 15:09:07.511767    5304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf975b55_20fa_459f_8659_a7a69f2d2248.slice/crio-60eba5d5dfba1e40a3e26ad3337aa40605ee28514ff5a1636e54a748bc1c509a WatchSource:0}: Error finding container 60eba5d5dfba1e40a3e26ad3337aa40605ee28514ff5a1636e54a748bc1c509a: Status 404 returned error can't find the container with id 60eba5d5dfba1e40a3e26ad3337aa40605ee28514ff5a1636e54a748bc1c509a
Nov 02 15:09:07.514106 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:07.514070    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:09:07.514196 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:07.514154    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:09:07.514241 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:07.514204    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:09:07.549228 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:07.549199    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-qm7zm"]
Nov 02 15:09:07.553268 ip-10-0-135-138 kubenswrapper[5304]: W1102 15:09:07.553234    5304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d0dd04f_b8ff_4ecd_a1fd_ec15ce4dea5a.slice/crio-2fc33fa7dd0d06b45af899e3333defb184801230bdda38f9634bcd9702fe5f4a WatchSource:0}: Error finding container 2fc33fa7dd0d06b45af899e3333defb184801230bdda38f9634bcd9702fe5f4a: Status 404 returned error can't find the container with id 2fc33fa7dd0d06b45af899e3333defb184801230bdda38f9634bcd9702fe5f4a
Nov 02 15:09:07.555949 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:07.555918    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:09:07.556026 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:07.555985    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:09:07.556026 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:07.556014    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:09:08.124710 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:08.124671    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-4cc4m" event={"ID":"af975b55-20fa-459f-8659-a7a69f2d2248","Type":"ContainerStarted","Data":"8c317367ce8a8c092812e99fe4869b78b9b68f8abb841513c7e4d33790c8c4e9"}
Nov 02 15:09:08.125206 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:08.124735    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-4cc4m" event={"ID":"af975b55-20fa-459f-8659-a7a69f2d2248","Type":"ContainerStarted","Data":"60eba5d5dfba1e40a3e26ad3337aa40605ee28514ff5a1636e54a748bc1c509a"}
Nov 02 15:09:08.126596 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:08.126570    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-qm7zm" event={"ID":"4d0dd04f-b8ff-4ecd-a1fd-ec15ce4dea5a","Type":"ContainerStarted","Data":"0ee06f62c3aa25c2759c1c9f53bfefcd5eb79054b406994dd091bfc80e08aeb0"}
Nov 02 15:09:08.126732 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:08.126616    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-qm7zm" event={"ID":"4d0dd04f-b8ff-4ecd-a1fd-ec15ce4dea5a","Type":"ContainerStarted","Data":"2fc33fa7dd0d06b45af899e3333defb184801230bdda38f9634bcd9702fe5f4a"}
Nov 02 15:09:08.193530 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:08.192707    5304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-4cc4m"
Nov 02 15:09:08.194113 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:08.194093    5304 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-4cc4m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body=
Nov 02 15:09:08.194195 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:08.194174    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-4cc4m" podUID="af975b55-20fa-459f-8659-a7a69f2d2248" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused"
Nov 02 15:09:08.214752 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:08.214379    5304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-qm7zm"
Nov 02 15:09:08.216362 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:08.216333    5304 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-qm7zm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body=
Nov 02 15:09:08.216493 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:08.216388    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-qm7zm" podUID="4d0dd04f-b8ff-4ecd-a1fd-ec15ce4dea5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused"
Nov 02 15:09:08.273150 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:08.272080    5304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-4cc4m" podStartSLOduration=2.2720608540000002 podStartE2EDuration="2.272060854s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:09:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:09:08.214550413 +0000 UTC m=+748.208374823" watchObservedRunningTime="2026-11-02 15:09:08.272060854 +0000 UTC m=+748.265885260"
Nov 02 15:09:09.031148 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:09.030253    5304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-qm7zm" podStartSLOduration=3.030231256 podStartE2EDuration="3.030231256s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:09:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:09:08.281320458 +0000 UTC m=+748.275144868" watchObservedRunningTime="2026-11-02 15:09:09.030231256 +0000 UTC m=+749.024055666"
Nov 02 15:09:09.115164 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:09.110774    5304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-k9kqx"]
Nov 02 15:09:09.165495 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:09.157162    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-k9kqx"
Nov 02 15:09:09.191621 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:09.191569    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-k9kqx"]
Nov 02 15:09:09.195984 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:09.195951    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f0b208d-d18a-4a38-952c-4a1abb1b30a0-workload-socket\") pod \"e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-k9kqx\" (UID: \"7f0b208d-d18a-4a38-952c-4a1abb1b30a0\") " pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-k9kqx"
Nov 02 15:09:09.197459 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:09.197431    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f0b208d-d18a-4a38-952c-4a1abb1b30a0-istio-podinfo\") pod \"e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-k9kqx\" (UID: \"7f0b208d-d18a-4a38-952c-4a1abb1b30a0\") " pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-k9kqx"
Nov 02 15:09:09.197785 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:09.197635    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s8w4k\" (UniqueName: \"kubernetes.io/projected/7f0b208d-d18a-4a38-952c-4a1abb1b30a0-kube-api-access-s8w4k\") pod \"e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-k9kqx\" (UID: \"7f0b208d-d18a-4a38-952c-4a1abb1b30a0\") " pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-k9kqx"
Nov 02 15:09:09.197929 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:09.197912    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f0b208d-d18a-4a38-952c-4a1abb1b30a0-istiod-ca-cert\") pod \"e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-k9kqx\" (UID: \"7f0b208d-d18a-4a38-952c-4a1abb1b30a0\") " pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-k9kqx"
Nov 02 15:09:09.198076 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:09.198052    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f0b208d-d18a-4a38-952c-4a1abb1b30a0-workload-certs\") pod \"e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-k9kqx\" (UID: \"7f0b208d-d18a-4a38-952c-4a1abb1b30a0\") " pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-k9kqx"
Nov 02 15:09:09.198219 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:09.198201    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f0b208d-d18a-4a38-952c-4a1abb1b30a0-istio-token\") pod \"e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-k9kqx\" (UID: \"7f0b208d-d18a-4a38-952c-4a1abb1b30a0\") " pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-k9kqx"
Nov 02 15:09:09.198522 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:09.198340    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f0b208d-d18a-4a38-952c-4a1abb1b30a0-credential-socket\") pod \"e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-k9kqx\" (UID: \"7f0b208d-d18a-4a38-952c-4a1abb1b30a0\") " pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-k9kqx"
Nov 02 15:09:09.198723 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:09.198705    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f0b208d-d18a-4a38-952c-4a1abb1b30a0-istio-data\") pod \"e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-k9kqx\" (UID: \"7f0b208d-d18a-4a38-952c-4a1abb1b30a0\") " pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-k9kqx"
Nov 02 15:09:09.201223 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:09.201198    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f0b208d-d18a-4a38-952c-4a1abb1b30a0-istio-envoy\") pod \"e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-k9kqx\" (UID: \"7f0b208d-d18a-4a38-952c-4a1abb1b30a0\") " pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-k9kqx"
Nov 02 15:09:09.217332 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:09.216554    5304 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-qm7zm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body=
Nov 02 15:09:09.217619 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:09.217583    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-qm7zm" podUID="4d0dd04f-b8ff-4ecd-a1fd-ec15ce4dea5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused"
Nov 02 15:09:09.290962 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:09.290862    5304 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-4cc4m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body=
Nov 02 15:09:09.291303 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:09.291269    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-4cc4m" podUID="af975b55-20fa-459f-8659-a7a69f2d2248" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused"
Nov 02 15:09:09.306542 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:09.306499    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f0b208d-d18a-4a38-952c-4a1abb1b30a0-workload-socket\") pod \"e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-k9kqx\" (UID: \"7f0b208d-d18a-4a38-952c-4a1abb1b30a0\") " pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-k9kqx"
Nov 02 15:09:09.306808 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:09.306789    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f0b208d-d18a-4a38-952c-4a1abb1b30a0-istio-podinfo\") pod \"e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-k9kqx\" (UID: \"7f0b208d-d18a-4a38-952c-4a1abb1b30a0\") " pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-k9kqx"
Nov 02 15:09:09.306931 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:09.306914    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8w4k\" (UniqueName: \"kubernetes.io/projected/7f0b208d-d18a-4a38-952c-4a1abb1b30a0-kube-api-access-s8w4k\") pod \"e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-k9kqx\" (UID: \"7f0b208d-d18a-4a38-952c-4a1abb1b30a0\") " pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-k9kqx"
Nov 02 15:09:09.307048 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:09.307032    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f0b208d-d18a-4a38-952c-4a1abb1b30a0-istiod-ca-cert\") pod \"e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-k9kqx\" (UID: \"7f0b208d-d18a-4a38-952c-4a1abb1b30a0\") " pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-k9kqx"
Nov 02 15:09:09.307212 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:09.307196    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f0b208d-d18a-4a38-952c-4a1abb1b30a0-workload-certs\") pod \"e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-k9kqx\" (UID: \"7f0b208d-d18a-4a38-952c-4a1abb1b30a0\") " pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-k9kqx"
Nov 02 15:09:09.307363 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:09.307347    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f0b208d-d18a-4a38-952c-4a1abb1b30a0-istio-token\") pod \"e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-k9kqx\" (UID: \"7f0b208d-d18a-4a38-952c-4a1abb1b30a0\") " pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-k9kqx"
Nov 02 15:09:09.307497 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:09.307482    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f0b208d-d18a-4a38-952c-4a1abb1b30a0-credential-socket\") pod \"e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-k9kqx\" (UID: \"7f0b208d-d18a-4a38-952c-4a1abb1b30a0\") " pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-k9kqx"
Nov 02 15:09:09.307659 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:09.307640    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f0b208d-d18a-4a38-952c-4a1abb1b30a0-istio-data\") pod \"e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-k9kqx\" (UID: \"7f0b208d-d18a-4a38-952c-4a1abb1b30a0\") " pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-k9kqx"
Nov 02 15:09:09.307785 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:09.307768    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f0b208d-d18a-4a38-952c-4a1abb1b30a0-istio-envoy\") pod \"e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-k9kqx\" (UID: \"7f0b208d-d18a-4a38-952c-4a1abb1b30a0\") " pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-k9kqx"
Nov 02 15:09:09.309235 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:09.309213    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f0b208d-d18a-4a38-952c-4a1abb1b30a0-istiod-ca-cert\") pod \"e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-k9kqx\" (UID: \"7f0b208d-d18a-4a38-952c-4a1abb1b30a0\") " pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-k9kqx"
Nov 02 15:09:09.313631 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:09.310973    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f0b208d-d18a-4a38-952c-4a1abb1b30a0-workload-certs\") pod \"e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-k9kqx\" (UID: \"7f0b208d-d18a-4a38-952c-4a1abb1b30a0\") " pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-k9kqx"
Nov 02 15:09:09.313788 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:09.311281    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f0b208d-d18a-4a38-952c-4a1abb1b30a0-credential-socket\") pod \"e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-k9kqx\" (UID: \"7f0b208d-d18a-4a38-952c-4a1abb1b30a0\") " pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-k9kqx"
Nov 02 15:09:09.313908 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:09.311524    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f0b208d-d18a-4a38-952c-4a1abb1b30a0-istio-data\") pod \"e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-k9kqx\" (UID: \"7f0b208d-d18a-4a38-952c-4a1abb1b30a0\") " pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-k9kqx"
Nov 02 15:09:09.314003 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:09.312034    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f0b208d-d18a-4a38-952c-4a1abb1b30a0-workload-socket\") pod \"e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-k9kqx\" (UID: \"7f0b208d-d18a-4a38-952c-4a1abb1b30a0\") " pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-k9kqx"
Nov 02 15:09:09.374239 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:09.358690    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f0b208d-d18a-4a38-952c-4a1abb1b30a0-istio-podinfo\") pod \"e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-k9kqx\" (UID: \"7f0b208d-d18a-4a38-952c-4a1abb1b30a0\") " pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-k9kqx"
Nov 02 15:09:09.374239 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:09.359563    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f0b208d-d18a-4a38-952c-4a1abb1b30a0-istio-envoy\") pod \"e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-k9kqx\" (UID: \"7f0b208d-d18a-4a38-952c-4a1abb1b30a0\") " pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-k9kqx"
Nov 02 15:09:09.374239 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:09.372080    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s8w4k\" (UniqueName: \"kubernetes.io/projected/7f0b208d-d18a-4a38-952c-4a1abb1b30a0-kube-api-access-s8w4k\") pod \"e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-k9kqx\" (UID: \"7f0b208d-d18a-4a38-952c-4a1abb1b30a0\") " pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-k9kqx"
Nov 02 15:09:09.396026 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:09.395988    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f0b208d-d18a-4a38-952c-4a1abb1b30a0-istio-token\") pod \"e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-k9kqx\" (UID: \"7f0b208d-d18a-4a38-952c-4a1abb1b30a0\") " pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-k9kqx"
Nov 02 15:09:09.602153 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:09.596379    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-k9kqx"
Nov 02 15:09:09.824150 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:09.823532    5304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-6af50f/distinct-w0-6af50f-kserve-c6d994647-hbj9z"
Nov 02 15:09:10.225343 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:10.207730    5304 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-4cc4m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body=
Nov 02 15:09:10.225343 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:10.207797    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-4cc4m" podUID="af975b55-20fa-459f-8659-a7a69f2d2248" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused"
Nov 02 15:09:10.253817 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:10.226867    5304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-6af50f/distinct-w0-6af50f-kserve-c6d994647-hbj9z"
Nov 02 15:09:10.253817 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:10.227094    5304 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-qm7zm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body=
Nov 02 15:09:10.253817 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:10.227157    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-qm7zm" podUID="4d0dd04f-b8ff-4ecd-a1fd-ec15ce4dea5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused"
Nov 02 15:09:10.601240 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:10.597191    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-k9kqx"]
Nov 02 15:09:10.638305 ip-10-0-135-138 kubenswrapper[5304]: W1102 15:09:10.638274    5304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f0b208d_d18a_4a38_952c_4a1abb1b30a0.slice/crio-a91340c00d2fc77aa513da268cf7cd5a98f0c6c8067da974ee62797409c759db WatchSource:0}: Error finding container a91340c00d2fc77aa513da268cf7cd5a98f0c6c8067da974ee62797409c759db: Status 404 returned error can't find the container with id a91340c00d2fc77aa513da268cf7cd5a98f0c6c8067da974ee62797409c759db
Nov 02 15:09:10.643521 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:10.643474    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:09:10.643777 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:10.643749    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:09:10.643978 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:10.643945    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:09:11.130465 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:11.129775    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-6af50f/distinct-w0-6af50f-kserve-c6d994647-hbj9z"
Nov 02 15:09:11.152867 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:11.131175    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-624254/simulator-w3-624254-kserve-56866b7699-z47sd" podUID="345c710f-b414-4e62-86c7-d51ff4e5be15" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": context deadline exceeded"
Nov 02 15:09:11.152867 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:11.143285    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nkbrk" podUID="1f68d837-fff2-418b-84d7-c2d7eb21a57e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.24:8000/ready\": context deadline exceeded"
Nov 02 15:09:11.153912 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:11.153868    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-739bf1/simulator-w2-739bf1-kserve-7c74f85465-ltk4t" podUID="cf6a1ded-28a3-49eb-878b-b4ad77a0bb5c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:09:11.204262 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:11.204218    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5dd8e9/simulator-w1-5dd8e9-kserve-6f9f755d6f-gnw5d" podUID="66b75e64-cb24-40be-9076-4984c132e208" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:09:11.216238 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:11.215084    5304 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-4cc4m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body=
Nov 02 15:09:11.216238 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:11.215170    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-4cc4m" podUID="af975b55-20fa-459f-8659-a7a69f2d2248" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused"
Nov 02 15:09:11.217353 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:11.217260    5304 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-qm7zm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body=
Nov 02 15:09:11.217353 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:11.217306    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-qm7zm" podUID="4d0dd04f-b8ff-4ecd-a1fd-ec15ce4dea5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused"
Nov 02 15:09:11.256152 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:11.254210    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1cc1b2/premium-w5-1cc1b2-kserve-58db46f97c-cgdqg" podUID="210f620a-8ae7-4e4c-acc3-46f75ddc6105" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:09:11.280171 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:11.276381    5304 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-1cc1b2/premium-w5-1cc1b2-kserve-58db46f97c-cgdqg" podUID="210f620a-8ae7-4e4c-acc3-46f75ddc6105" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:09:11.318204 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:11.313262    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5dd8e9/premium-w1-5dd8e9-kserve-67588456fb-j6lph" podUID="4ccc607a-14e4-43ca-8999-78100d4b799f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:09:11.430069 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:11.430031    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-k9kqx" event={"ID":"7f0b208d-d18a-4a38-952c-4a1abb1b30a0","Type":"ContainerStarted","Data":"df5313aa08c19e33bc82d9ee5d39df53edec151bc8e4706fed0c3eca101ff271"}
Nov 02 15:09:11.430266 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:11.430096    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-k9kqx" event={"ID":"7f0b208d-d18a-4a38-952c-4a1abb1b30a0","Type":"ContainerStarted","Data":"a91340c00d2fc77aa513da268cf7cd5a98f0c6c8067da974ee62797409c759db"}
Nov 02 15:09:11.498193 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:11.492265    5304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-k9kqx" podStartSLOduration=3.492243683 podStartE2EDuration="3.492243683s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:09:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:09:11.491336362 +0000 UTC m=+751.485160773" watchObservedRunningTime="2026-11-02 15:09:11.492243683 +0000 UTC m=+751.486068093"
Nov 02 15:09:11.596917 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:11.596886    5304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-k9kqx"
Nov 02 15:09:12.193043 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:12.193000    5304 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-4cc4m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body=
Nov 02 15:09:12.193240 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:12.193055    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-4cc4m" podUID="af975b55-20fa-459f-8659-a7a69f2d2248" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused"
Nov 02 15:09:12.214246 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:12.214208    5304 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-qm7zm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body=
Nov 02 15:09:12.214409 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:12.214266    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-qm7zm" podUID="4d0dd04f-b8ff-4ecd-a1fd-ec15ce4dea5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused"
Nov 02 15:09:12.600163 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:12.598693    5304 patch_prober.go:29] interesting pod/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-k9kqx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body=
Nov 02 15:09:12.600163 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:12.598757    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-k9kqx" podUID="7f0b208d-d18a-4a38-952c-4a1abb1b30a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused"
Nov 02 15:09:13.071210 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:13.070192    5304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-1cc1b2/distinct-w5-1cc1b2-kserve-5b7459bc6d-zl8xr"
Nov 02 15:09:13.126756 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:13.124619    5304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-1cc1b2/distinct-w5-1cc1b2-kserve-5b7459bc6d-zl8xr"
Nov 02 15:09:13.204151 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:13.203269    5304 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-4cc4m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body=
Nov 02 15:09:13.204151 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:13.203336    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-4cc4m" podUID="af975b55-20fa-459f-8659-a7a69f2d2248" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused"
Nov 02 15:09:13.214635 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:13.214599    5304 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-qm7zm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body=
Nov 02 15:09:13.214791 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:13.214673    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-qm7zm" podUID="4d0dd04f-b8ff-4ecd-a1fd-ec15ce4dea5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused"
Nov 02 15:09:13.598149 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:13.597526    5304 patch_prober.go:29] interesting pod/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-k9kqx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body=
Nov 02 15:09:13.598149 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:13.597588    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-k9kqx" podUID="7f0b208d-d18a-4a38-952c-4a1abb1b30a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused"
Nov 02 15:09:13.623924 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:13.623646    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-1cc1b2/distinct-w5-1cc1b2-kserve-5b7459bc6d-zl8xr"
Nov 02 15:09:14.192738 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:14.192697    5304 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-4cc4m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body=
Nov 02 15:09:14.192913 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:14.192763    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-4cc4m" podUID="af975b55-20fa-459f-8659-a7a69f2d2248" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused"
Nov 02 15:09:14.214973 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:14.214934    5304 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-qm7zm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body=
Nov 02 15:09:14.215154 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:14.214991    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-qm7zm" podUID="4d0dd04f-b8ff-4ecd-a1fd-ec15ce4dea5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused"
Nov 02 15:09:14.597768 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:14.597731    5304 patch_prober.go:29] interesting pod/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-k9kqx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body=
Nov 02 15:09:14.597969 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:14.597793    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-k9kqx" podUID="7f0b208d-d18a-4a38-952c-4a1abb1b30a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused"
Nov 02 15:09:15.192344 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:15.192308    5304 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-4cc4m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body=
Nov 02 15:09:15.192749 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:15.192365    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-4cc4m" podUID="af975b55-20fa-459f-8659-a7a69f2d2248" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused"
Nov 02 15:09:15.215064 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:15.215030    5304 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-qm7zm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body=
Nov 02 15:09:15.215232 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:15.215091    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-qm7zm" podUID="4d0dd04f-b8ff-4ecd-a1fd-ec15ce4dea5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused"
Nov 02 15:09:15.598327 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:15.597392    5304 patch_prober.go:29] interesting pod/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-k9kqx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body=
Nov 02 15:09:15.598327 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:15.597460    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-k9kqx" podUID="7f0b208d-d18a-4a38-952c-4a1abb1b30a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused"
Nov 02 15:09:16.597783 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:16.597746    5304 patch_prober.go:29] interesting pod/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-k9kqx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body=
Nov 02 15:09:16.598236 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:16.597811    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-k9kqx" podUID="7f0b208d-d18a-4a38-952c-4a1abb1b30a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused"
Nov 02 15:09:17.192696 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:17.192660    5304 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-4cc4m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 15:09:17.192875 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:17.192849    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-4cc4m" podUID="af975b55-20fa-459f-8659-a7a69f2d2248" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:09:17.205201 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:17.205173    5304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-qm7zm"
Nov 02 15:09:17.206184 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:17.206116    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-qm7zm"
Nov 02 15:09:17.207183 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:17.207164    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-qm7zm"
Nov 02 15:09:17.216351 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:17.216318    5304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-4cc4m"
Nov 02 15:09:17.216801 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:17.216782    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-4cc4m"
Nov 02 15:09:17.217831 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:17.217810    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-4cc4m"
Nov 02 15:09:17.597648 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:17.597611    5304 patch_prober.go:29] interesting pod/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-k9kqx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body=
Nov 02 15:09:17.597813 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:17.597676    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-k9kqx" podUID="7f0b208d-d18a-4a38-952c-4a1abb1b30a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused"
Nov 02 15:09:18.291188 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:18.287239    5304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-7nj6n"]
Nov 02 15:09:18.299168 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:18.292192    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-7nj6n"
Nov 02 15:09:18.314179 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:18.311735    5304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-5113c2-openshift-default-dockercfg-kwr29\""
Nov 02 15:09:18.323015 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:18.322979    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ebb8197-273d-4e96-bec7-00c7a85a12a9-istio-data\") pod \"e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-7nj6n\" (UID: \"3ebb8197-273d-4e96-bec7-00c7a85a12a9\") " pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-7nj6n"
Nov 02 15:09:18.323226 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:18.323074    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ebb8197-273d-4e96-bec7-00c7a85a12a9-workload-certs\") pod \"e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-7nj6n\" (UID: \"3ebb8197-273d-4e96-bec7-00c7a85a12a9\") " pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-7nj6n"
Nov 02 15:09:18.323226 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:18.323167    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ebb8197-273d-4e96-bec7-00c7a85a12a9-istiod-ca-cert\") pod \"e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-7nj6n\" (UID: \"3ebb8197-273d-4e96-bec7-00c7a85a12a9\") " pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-7nj6n"
Nov 02 15:09:18.323226 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:18.323202    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7lggw\" (UniqueName: \"kubernetes.io/projected/3ebb8197-273d-4e96-bec7-00c7a85a12a9-kube-api-access-7lggw\") pod \"e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-7nj6n\" (UID: \"3ebb8197-273d-4e96-bec7-00c7a85a12a9\") " pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-7nj6n"
Nov 02 15:09:18.323411 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:18.323274    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ebb8197-273d-4e96-bec7-00c7a85a12a9-workload-socket\") pod \"e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-7nj6n\" (UID: \"3ebb8197-273d-4e96-bec7-00c7a85a12a9\") " pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-7nj6n"
Nov 02 15:09:18.323463 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:18.323404    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ebb8197-273d-4e96-bec7-00c7a85a12a9-istio-podinfo\") pod \"e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-7nj6n\" (UID: \"3ebb8197-273d-4e96-bec7-00c7a85a12a9\") " pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-7nj6n"
Nov 02 15:09:18.323519 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:18.323453    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ebb8197-273d-4e96-bec7-00c7a85a12a9-credential-socket\") pod \"e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-7nj6n\" (UID: \"3ebb8197-273d-4e96-bec7-00c7a85a12a9\") " pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-7nj6n"
Nov 02 15:09:18.323709 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:18.323686    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ebb8197-273d-4e96-bec7-00c7a85a12a9-istio-envoy\") pod \"e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-7nj6n\" (UID: \"3ebb8197-273d-4e96-bec7-00c7a85a12a9\") " pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-7nj6n"
Nov 02 15:09:18.323780 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:18.323743    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ebb8197-273d-4e96-bec7-00c7a85a12a9-istio-token\") pod \"e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-7nj6n\" (UID: \"3ebb8197-273d-4e96-bec7-00c7a85a12a9\") " pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-7nj6n"
Nov 02 15:09:18.326630 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:18.326601    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-7nj6n"]
Nov 02 15:09:18.425115 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:18.425079    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ebb8197-273d-4e96-bec7-00c7a85a12a9-istiod-ca-cert\") pod \"e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-7nj6n\" (UID: \"3ebb8197-273d-4e96-bec7-00c7a85a12a9\") " pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-7nj6n"
Nov 02 15:09:18.425336 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:18.425140    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7lggw\" (UniqueName: \"kubernetes.io/projected/3ebb8197-273d-4e96-bec7-00c7a85a12a9-kube-api-access-7lggw\") pod \"e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-7nj6n\" (UID: \"3ebb8197-273d-4e96-bec7-00c7a85a12a9\") " pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-7nj6n"
Nov 02 15:09:18.425336 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:18.425167    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ebb8197-273d-4e96-bec7-00c7a85a12a9-workload-socket\") pod \"e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-7nj6n\" (UID: \"3ebb8197-273d-4e96-bec7-00c7a85a12a9\") " pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-7nj6n"
Nov 02 15:09:18.425336 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:18.425215    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ebb8197-273d-4e96-bec7-00c7a85a12a9-istio-podinfo\") pod \"e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-7nj6n\" (UID: \"3ebb8197-273d-4e96-bec7-00c7a85a12a9\") " pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-7nj6n"
Nov 02 15:09:18.425336 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:18.425239    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ebb8197-273d-4e96-bec7-00c7a85a12a9-credential-socket\") pod \"e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-7nj6n\" (UID: \"3ebb8197-273d-4e96-bec7-00c7a85a12a9\") " pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-7nj6n"
Nov 02 15:09:18.425336 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:18.425319    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ebb8197-273d-4e96-bec7-00c7a85a12a9-istio-envoy\") pod \"e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-7nj6n\" (UID: \"3ebb8197-273d-4e96-bec7-00c7a85a12a9\") " pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-7nj6n"
Nov 02 15:09:18.425605 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:18.425358    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ebb8197-273d-4e96-bec7-00c7a85a12a9-istio-token\") pod \"e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-7nj6n\" (UID: \"3ebb8197-273d-4e96-bec7-00c7a85a12a9\") " pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-7nj6n"
Nov 02 15:09:18.425605 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:18.425401    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ebb8197-273d-4e96-bec7-00c7a85a12a9-istio-data\") pod \"e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-7nj6n\" (UID: \"3ebb8197-273d-4e96-bec7-00c7a85a12a9\") " pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-7nj6n"
Nov 02 15:09:18.425605 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:18.425469    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ebb8197-273d-4e96-bec7-00c7a85a12a9-workload-certs\") pod \"e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-7nj6n\" (UID: \"3ebb8197-273d-4e96-bec7-00c7a85a12a9\") " pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-7nj6n"
Nov 02 15:09:18.425605 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:18.425595    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ebb8197-273d-4e96-bec7-00c7a85a12a9-workload-socket\") pod \"e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-7nj6n\" (UID: \"3ebb8197-273d-4e96-bec7-00c7a85a12a9\") " pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-7nj6n"
Nov 02 15:09:18.425925 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:18.425904    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ebb8197-273d-4e96-bec7-00c7a85a12a9-workload-certs\") pod \"e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-7nj6n\" (UID: \"3ebb8197-273d-4e96-bec7-00c7a85a12a9\") " pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-7nj6n"
Nov 02 15:09:18.426007 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:18.425899    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ebb8197-273d-4e96-bec7-00c7a85a12a9-istiod-ca-cert\") pod \"e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-7nj6n\" (UID: \"3ebb8197-273d-4e96-bec7-00c7a85a12a9\") " pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-7nj6n"
Nov 02 15:09:18.426007 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:18.425993    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ebb8197-273d-4e96-bec7-00c7a85a12a9-credential-socket\") pod \"e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-7nj6n\" (UID: \"3ebb8197-273d-4e96-bec7-00c7a85a12a9\") " pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-7nj6n"
Nov 02 15:09:18.426262 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:18.426242    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ebb8197-273d-4e96-bec7-00c7a85a12a9-istio-data\") pod \"e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-7nj6n\" (UID: \"3ebb8197-273d-4e96-bec7-00c7a85a12a9\") " pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-7nj6n"
Nov 02 15:09:18.427889 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:18.427866    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ebb8197-273d-4e96-bec7-00c7a85a12a9-istio-envoy\") pod \"e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-7nj6n\" (UID: \"3ebb8197-273d-4e96-bec7-00c7a85a12a9\") " pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-7nj6n"
Nov 02 15:09:18.428146 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:18.428095    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ebb8197-273d-4e96-bec7-00c7a85a12a9-istio-podinfo\") pod \"e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-7nj6n\" (UID: \"3ebb8197-273d-4e96-bec7-00c7a85a12a9\") " pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-7nj6n"
Nov 02 15:09:18.449115 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:18.449081    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ebb8197-273d-4e96-bec7-00c7a85a12a9-istio-token\") pod \"e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-7nj6n\" (UID: \"3ebb8197-273d-4e96-bec7-00c7a85a12a9\") " pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-7nj6n"
Nov 02 15:09:18.449307 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:18.449116    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7lggw\" (UniqueName: \"kubernetes.io/projected/3ebb8197-273d-4e96-bec7-00c7a85a12a9-kube-api-access-7lggw\") pod \"e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-7nj6n\" (UID: \"3ebb8197-273d-4e96-bec7-00c7a85a12a9\") " pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-7nj6n"
Nov 02 15:09:18.633101 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:18.633072    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-7nj6n"
Nov 02 15:09:18.748861 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:18.748770    5304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-k9kqx"
Nov 02 15:09:18.750462 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:18.750240    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-k9kqx"
Nov 02 15:09:18.751633 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:18.751606    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-k9kqx"
Nov 02 15:09:18.956741 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:18.956659    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-7nj6n"]
Nov 02 15:09:18.960157 ip-10-0-135-138 kubenswrapper[5304]: W1102 15:09:18.960113    5304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ebb8197_273d_4e96_bec7_00c7a85a12a9.slice/crio-196ab17e3dbd9dafa9eb688e338d605d2c808cb03ba2747bc309835883d6238b WatchSource:0}: Error finding container 196ab17e3dbd9dafa9eb688e338d605d2c808cb03ba2747bc309835883d6238b: Status 404 returned error can't find the container with id 196ab17e3dbd9dafa9eb688e338d605d2c808cb03ba2747bc309835883d6238b
Nov 02 15:09:18.962494 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:18.962462    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:09:18.962597 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:18.962535    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:09:18.962704 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:18.962687    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:09:19.577934 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:19.577891    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-7nj6n" event={"ID":"3ebb8197-273d-4e96-bec7-00c7a85a12a9","Type":"ContainerStarted","Data":"db9ed33df2feb8a3dd2cdb179234648679154d230db955b2cc925161eb56b931"}
Nov 02 15:09:19.578115 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:19.577992    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-7nj6n" event={"ID":"3ebb8197-273d-4e96-bec7-00c7a85a12a9","Type":"ContainerStarted","Data":"196ab17e3dbd9dafa9eb688e338d605d2c808cb03ba2747bc309835883d6238b"}
Nov 02 15:09:19.624182 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:19.624108    5304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-7nj6n" podStartSLOduration=1.624093218 podStartE2EDuration="1.624093218s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:09:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:09:19.621346383 +0000 UTC m=+759.615170793" watchObservedRunningTime="2026-11-02 15:09:19.624093218 +0000 UTC m=+759.617917626"
Nov 02 15:09:19.635149 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:19.633920    5304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-7nj6n"
Nov 02 15:09:20.656465 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:20.656431    5304 patch_prober.go:29] interesting pod/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-7nj6n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body=
Nov 02 15:09:20.656888 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:20.656481    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-7nj6n" podUID="3ebb8197-273d-4e96-bec7-00c7a85a12a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Nov 02 15:09:21.651153 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:21.646584    5304 patch_prober.go:29] interesting pod/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-7nj6n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body=
Nov 02 15:09:21.651153 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:21.646652    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-7nj6n" podUID="3ebb8197-273d-4e96-bec7-00c7a85a12a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Nov 02 15:09:22.651152 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:22.640490    5304 patch_prober.go:29] interesting pod/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-7nj6n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body=
Nov 02 15:09:22.651152 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:22.640554    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-7nj6n" podUID="3ebb8197-273d-4e96-bec7-00c7a85a12a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Nov 02 15:09:23.617149 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:23.617092    5304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-b88h7"]
Nov 02 15:09:23.621082 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:23.621061    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-b88h7"
Nov 02 15:09:23.660151 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:23.659335    5304 patch_prober.go:29] interesting pod/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-7nj6n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body=
Nov 02 15:09:23.660151 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:23.659401    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-7nj6n" podUID="3ebb8197-273d-4e96-bec7-00c7a85a12a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Nov 02 15:09:23.661838 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:23.661795    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-b88h7"]
Nov 02 15:09:23.699585 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:23.699552    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d4f7624-882c-4cb3-8811-9ca682e0c1d5-istiod-ca-cert\") pod \"e2e-worker-w3-624254-openshift-default-7988c9d8b5-b88h7\" (UID: \"3d4f7624-882c-4cb3-8811-9ca682e0c1d5\") " pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-b88h7"
Nov 02 15:09:23.699751 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:23.699597    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d4f7624-882c-4cb3-8811-9ca682e0c1d5-istio-data\") pod \"e2e-worker-w3-624254-openshift-default-7988c9d8b5-b88h7\" (UID: \"3d4f7624-882c-4cb3-8811-9ca682e0c1d5\") " pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-b88h7"
Nov 02 15:09:23.699751 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:23.699736    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d4f7624-882c-4cb3-8811-9ca682e0c1d5-workload-socket\") pod \"e2e-worker-w3-624254-openshift-default-7988c9d8b5-b88h7\" (UID: \"3d4f7624-882c-4cb3-8811-9ca682e0c1d5\") " pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-b88h7"
Nov 02 15:09:23.699848 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:23.699768    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d4f7624-882c-4cb3-8811-9ca682e0c1d5-istio-envoy\") pod \"e2e-worker-w3-624254-openshift-default-7988c9d8b5-b88h7\" (UID: \"3d4f7624-882c-4cb3-8811-9ca682e0c1d5\") " pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-b88h7"
Nov 02 15:09:23.699924 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:23.699897    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d4f7624-882c-4cb3-8811-9ca682e0c1d5-istio-podinfo\") pod \"e2e-worker-w3-624254-openshift-default-7988c9d8b5-b88h7\" (UID: \"3d4f7624-882c-4cb3-8811-9ca682e0c1d5\") " pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-b88h7"
Nov 02 15:09:23.699981 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:23.699943    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d4f7624-882c-4cb3-8811-9ca682e0c1d5-credential-socket\") pod \"e2e-worker-w3-624254-openshift-default-7988c9d8b5-b88h7\" (UID: \"3d4f7624-882c-4cb3-8811-9ca682e0c1d5\") " pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-b88h7"
Nov 02 15:09:23.700065 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:23.700051    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d4f7624-882c-4cb3-8811-9ca682e0c1d5-istio-token\") pod \"e2e-worker-w3-624254-openshift-default-7988c9d8b5-b88h7\" (UID: \"3d4f7624-882c-4cb3-8811-9ca682e0c1d5\") " pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-b88h7"
Nov 02 15:09:23.700139 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:23.700091    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-smsjj\" (UniqueName: \"kubernetes.io/projected/3d4f7624-882c-4cb3-8811-9ca682e0c1d5-kube-api-access-smsjj\") pod \"e2e-worker-w3-624254-openshift-default-7988c9d8b5-b88h7\" (UID: \"3d4f7624-882c-4cb3-8811-9ca682e0c1d5\") " pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-b88h7"
Nov 02 15:09:23.700270 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:23.700253    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d4f7624-882c-4cb3-8811-9ca682e0c1d5-workload-certs\") pod \"e2e-worker-w3-624254-openshift-default-7988c9d8b5-b88h7\" (UID: \"3d4f7624-882c-4cb3-8811-9ca682e0c1d5\") " pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-b88h7"
Nov 02 15:09:23.801278 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:23.801226    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d4f7624-882c-4cb3-8811-9ca682e0c1d5-istiod-ca-cert\") pod \"e2e-worker-w3-624254-openshift-default-7988c9d8b5-b88h7\" (UID: \"3d4f7624-882c-4cb3-8811-9ca682e0c1d5\") " pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-b88h7"
Nov 02 15:09:23.801488 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:23.801288    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d4f7624-882c-4cb3-8811-9ca682e0c1d5-istio-data\") pod \"e2e-worker-w3-624254-openshift-default-7988c9d8b5-b88h7\" (UID: \"3d4f7624-882c-4cb3-8811-9ca682e0c1d5\") " pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-b88h7"
Nov 02 15:09:23.801488 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:23.801391    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d4f7624-882c-4cb3-8811-9ca682e0c1d5-workload-socket\") pod \"e2e-worker-w3-624254-openshift-default-7988c9d8b5-b88h7\" (UID: \"3d4f7624-882c-4cb3-8811-9ca682e0c1d5\") " pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-b88h7"
Nov 02 15:09:23.801488 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:23.801418    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d4f7624-882c-4cb3-8811-9ca682e0c1d5-istio-envoy\") pod \"e2e-worker-w3-624254-openshift-default-7988c9d8b5-b88h7\" (UID: \"3d4f7624-882c-4cb3-8811-9ca682e0c1d5\") " pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-b88h7"
Nov 02 15:09:23.801488 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:23.801473    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d4f7624-882c-4cb3-8811-9ca682e0c1d5-istio-podinfo\") pod \"e2e-worker-w3-624254-openshift-default-7988c9d8b5-b88h7\" (UID: \"3d4f7624-882c-4cb3-8811-9ca682e0c1d5\") " pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-b88h7"
Nov 02 15:09:23.801678 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:23.801564    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d4f7624-882c-4cb3-8811-9ca682e0c1d5-credential-socket\") pod \"e2e-worker-w3-624254-openshift-default-7988c9d8b5-b88h7\" (UID: \"3d4f7624-882c-4cb3-8811-9ca682e0c1d5\") " pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-b88h7"
Nov 02 15:09:23.801678 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:23.801641    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d4f7624-882c-4cb3-8811-9ca682e0c1d5-istio-token\") pod \"e2e-worker-w3-624254-openshift-default-7988c9d8b5-b88h7\" (UID: \"3d4f7624-882c-4cb3-8811-9ca682e0c1d5\") " pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-b88h7"
Nov 02 15:09:23.802009 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:23.801677    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-smsjj\" (UniqueName: \"kubernetes.io/projected/3d4f7624-882c-4cb3-8811-9ca682e0c1d5-kube-api-access-smsjj\") pod \"e2e-worker-w3-624254-openshift-default-7988c9d8b5-b88h7\" (UID: \"3d4f7624-882c-4cb3-8811-9ca682e0c1d5\") " pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-b88h7"
Nov 02 15:09:23.802009 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:23.801725    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d4f7624-882c-4cb3-8811-9ca682e0c1d5-istio-data\") pod \"e2e-worker-w3-624254-openshift-default-7988c9d8b5-b88h7\" (UID: \"3d4f7624-882c-4cb3-8811-9ca682e0c1d5\") " pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-b88h7"
Nov 02 15:09:23.802009 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:23.801907    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d4f7624-882c-4cb3-8811-9ca682e0c1d5-workload-socket\") pod \"e2e-worker-w3-624254-openshift-default-7988c9d8b5-b88h7\" (UID: \"3d4f7624-882c-4cb3-8811-9ca682e0c1d5\") " pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-b88h7"
Nov 02 15:09:23.802009 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:23.801965    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d4f7624-882c-4cb3-8811-9ca682e0c1d5-credential-socket\") pod \"e2e-worker-w3-624254-openshift-default-7988c9d8b5-b88h7\" (UID: \"3d4f7624-882c-4cb3-8811-9ca682e0c1d5\") " pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-b88h7"
Nov 02 15:09:23.802250 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:23.802055    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d4f7624-882c-4cb3-8811-9ca682e0c1d5-workload-certs\") pod \"e2e-worker-w3-624254-openshift-default-7988c9d8b5-b88h7\" (UID: \"3d4f7624-882c-4cb3-8811-9ca682e0c1d5\") " pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-b88h7"
Nov 02 15:09:23.802250 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:23.802120    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d4f7624-882c-4cb3-8811-9ca682e0c1d5-istiod-ca-cert\") pod \"e2e-worker-w3-624254-openshift-default-7988c9d8b5-b88h7\" (UID: \"3d4f7624-882c-4cb3-8811-9ca682e0c1d5\") " pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-b88h7"
Nov 02 15:09:23.802353 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:23.802335    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d4f7624-882c-4cb3-8811-9ca682e0c1d5-workload-certs\") pod \"e2e-worker-w3-624254-openshift-default-7988c9d8b5-b88h7\" (UID: \"3d4f7624-882c-4cb3-8811-9ca682e0c1d5\") " pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-b88h7"
Nov 02 15:09:23.804315 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:23.804280    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d4f7624-882c-4cb3-8811-9ca682e0c1d5-istio-envoy\") pod \"e2e-worker-w3-624254-openshift-default-7988c9d8b5-b88h7\" (UID: \"3d4f7624-882c-4cb3-8811-9ca682e0c1d5\") " pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-b88h7"
Nov 02 15:09:23.804478 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:23.804463    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d4f7624-882c-4cb3-8811-9ca682e0c1d5-istio-podinfo\") pod \"e2e-worker-w3-624254-openshift-default-7988c9d8b5-b88h7\" (UID: \"3d4f7624-882c-4cb3-8811-9ca682e0c1d5\") " pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-b88h7"
Nov 02 15:09:23.824747 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:23.824718    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d4f7624-882c-4cb3-8811-9ca682e0c1d5-istio-token\") pod \"e2e-worker-w3-624254-openshift-default-7988c9d8b5-b88h7\" (UID: \"3d4f7624-882c-4cb3-8811-9ca682e0c1d5\") " pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-b88h7"
Nov 02 15:09:23.824889 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:23.824871    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-smsjj\" (UniqueName: \"kubernetes.io/projected/3d4f7624-882c-4cb3-8811-9ca682e0c1d5-kube-api-access-smsjj\") pod \"e2e-worker-w3-624254-openshift-default-7988c9d8b5-b88h7\" (UID: \"3d4f7624-882c-4cb3-8811-9ca682e0c1d5\") " pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-b88h7"
Nov 02 15:09:23.966571 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:23.966479    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-b88h7"
Nov 02 15:09:24.259246 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:24.259158    5304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-mp9v6"]
Nov 02 15:09:24.263972 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:24.263952    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-mp9v6"
Nov 02 15:09:24.287242 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:24.287212    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-mp9v6"]
Nov 02 15:09:24.298158 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:24.298114    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-b88h7"]
Nov 02 15:09:24.304901 ip-10-0-135-138 kubenswrapper[5304]: W1102 15:09:24.304870    5304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d4f7624_882c_4cb3_8811_9ca682e0c1d5.slice/crio-19ed571e5ff8bcda7823a7e117f1d5b1b014be7c2df13bbd46a1a7072ec89928 WatchSource:0}: Error finding container 19ed571e5ff8bcda7823a7e117f1d5b1b014be7c2df13bbd46a1a7072ec89928: Status 404 returned error can't find the container with id 19ed571e5ff8bcda7823a7e117f1d5b1b014be7c2df13bbd46a1a7072ec89928
Nov 02 15:09:24.307757 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:24.307712    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:09:24.307863 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:24.307815    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:09:24.307927 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:24.307866    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:09:24.308989 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:24.308961    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1786cd64-e217-4f69-bcb6-6bda8aace53d-istiod-ca-cert\") pod \"e2e-worker-w0-6af50f-openshift-default-8545fb68f-mp9v6\" (UID: \"1786cd64-e217-4f69-bcb6-6bda8aace53d\") " pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-mp9v6"
Nov 02 15:09:24.309091 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:24.309033    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1786cd64-e217-4f69-bcb6-6bda8aace53d-istio-data\") pod \"e2e-worker-w0-6af50f-openshift-default-8545fb68f-mp9v6\" (UID: \"1786cd64-e217-4f69-bcb6-6bda8aace53d\") " pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-mp9v6"
Nov 02 15:09:24.309199 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:24.309178    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1786cd64-e217-4f69-bcb6-6bda8aace53d-istio-podinfo\") pod \"e2e-worker-w0-6af50f-openshift-default-8545fb68f-mp9v6\" (UID: \"1786cd64-e217-4f69-bcb6-6bda8aace53d\") " pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-mp9v6"
Nov 02 15:09:24.309257 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:24.309223    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1786cd64-e217-4f69-bcb6-6bda8aace53d-workload-certs\") pod \"e2e-worker-w0-6af50f-openshift-default-8545fb68f-mp9v6\" (UID: \"1786cd64-e217-4f69-bcb6-6bda8aace53d\") " pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-mp9v6"
Nov 02 15:09:24.309313 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:24.309253    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1786cd64-e217-4f69-bcb6-6bda8aace53d-workload-socket\") pod \"e2e-worker-w0-6af50f-openshift-default-8545fb68f-mp9v6\" (UID: \"1786cd64-e217-4f69-bcb6-6bda8aace53d\") " pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-mp9v6"
Nov 02 15:09:24.309371 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:24.309321    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1786cd64-e217-4f69-bcb6-6bda8aace53d-istio-token\") pod \"e2e-worker-w0-6af50f-openshift-default-8545fb68f-mp9v6\" (UID: \"1786cd64-e217-4f69-bcb6-6bda8aace53d\") " pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-mp9v6"
Nov 02 15:09:24.309371 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:24.309346    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1786cd64-e217-4f69-bcb6-6bda8aace53d-istio-envoy\") pod \"e2e-worker-w0-6af50f-openshift-default-8545fb68f-mp9v6\" (UID: \"1786cd64-e217-4f69-bcb6-6bda8aace53d\") " pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-mp9v6"
Nov 02 15:09:24.309472 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:24.309373    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gtsq5\" (UniqueName: \"kubernetes.io/projected/1786cd64-e217-4f69-bcb6-6bda8aace53d-kube-api-access-gtsq5\") pod \"e2e-worker-w0-6af50f-openshift-default-8545fb68f-mp9v6\" (UID: \"1786cd64-e217-4f69-bcb6-6bda8aace53d\") " pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-mp9v6"
Nov 02 15:09:24.309472 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:24.309395    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1786cd64-e217-4f69-bcb6-6bda8aace53d-credential-socket\") pod \"e2e-worker-w0-6af50f-openshift-default-8545fb68f-mp9v6\" (UID: \"1786cd64-e217-4f69-bcb6-6bda8aace53d\") " pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-mp9v6"
Nov 02 15:09:24.410430 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:24.410398    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1786cd64-e217-4f69-bcb6-6bda8aace53d-workload-certs\") pod \"e2e-worker-w0-6af50f-openshift-default-8545fb68f-mp9v6\" (UID: \"1786cd64-e217-4f69-bcb6-6bda8aace53d\") " pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-mp9v6"
Nov 02 15:09:24.410430 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:24.410440    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1786cd64-e217-4f69-bcb6-6bda8aace53d-workload-socket\") pod \"e2e-worker-w0-6af50f-openshift-default-8545fb68f-mp9v6\" (UID: \"1786cd64-e217-4f69-bcb6-6bda8aace53d\") " pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-mp9v6"
Nov 02 15:09:24.410657 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:24.410507    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1786cd64-e217-4f69-bcb6-6bda8aace53d-istio-token\") pod \"e2e-worker-w0-6af50f-openshift-default-8545fb68f-mp9v6\" (UID: \"1786cd64-e217-4f69-bcb6-6bda8aace53d\") " pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-mp9v6"
Nov 02 15:09:24.410657 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:24.410535    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1786cd64-e217-4f69-bcb6-6bda8aace53d-istio-envoy\") pod \"e2e-worker-w0-6af50f-openshift-default-8545fb68f-mp9v6\" (UID: \"1786cd64-e217-4f69-bcb6-6bda8aace53d\") " pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-mp9v6"
Nov 02 15:09:24.410657 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:24.410563    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gtsq5\" (UniqueName: \"kubernetes.io/projected/1786cd64-e217-4f69-bcb6-6bda8aace53d-kube-api-access-gtsq5\") pod \"e2e-worker-w0-6af50f-openshift-default-8545fb68f-mp9v6\" (UID: \"1786cd64-e217-4f69-bcb6-6bda8aace53d\") " pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-mp9v6"
Nov 02 15:09:24.410657 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:24.410588    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1786cd64-e217-4f69-bcb6-6bda8aace53d-credential-socket\") pod \"e2e-worker-w0-6af50f-openshift-default-8545fb68f-mp9v6\" (UID: \"1786cd64-e217-4f69-bcb6-6bda8aace53d\") " pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-mp9v6"
Nov 02 15:09:24.410846 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:24.410682    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1786cd64-e217-4f69-bcb6-6bda8aace53d-istiod-ca-cert\") pod \"e2e-worker-w0-6af50f-openshift-default-8545fb68f-mp9v6\" (UID: \"1786cd64-e217-4f69-bcb6-6bda8aace53d\") " pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-mp9v6"
Nov 02 15:09:24.410846 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:24.410735    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1786cd64-e217-4f69-bcb6-6bda8aace53d-istio-data\") pod \"e2e-worker-w0-6af50f-openshift-default-8545fb68f-mp9v6\" (UID: \"1786cd64-e217-4f69-bcb6-6bda8aace53d\") " pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-mp9v6"
Nov 02 15:09:24.410846 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:24.410830    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1786cd64-e217-4f69-bcb6-6bda8aace53d-workload-certs\") pod \"e2e-worker-w0-6af50f-openshift-default-8545fb68f-mp9v6\" (UID: \"1786cd64-e217-4f69-bcb6-6bda8aace53d\") " pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-mp9v6"
Nov 02 15:09:24.410993 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:24.410884    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1786cd64-e217-4f69-bcb6-6bda8aace53d-istio-podinfo\") pod \"e2e-worker-w0-6af50f-openshift-default-8545fb68f-mp9v6\" (UID: \"1786cd64-e217-4f69-bcb6-6bda8aace53d\") " pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-mp9v6"
Nov 02 15:09:24.411094 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:24.411054    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1786cd64-e217-4f69-bcb6-6bda8aace53d-credential-socket\") pod \"e2e-worker-w0-6af50f-openshift-default-8545fb68f-mp9v6\" (UID: \"1786cd64-e217-4f69-bcb6-6bda8aace53d\") " pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-mp9v6"
Nov 02 15:09:24.411212 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:24.411155    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1786cd64-e217-4f69-bcb6-6bda8aace53d-istio-data\") pod \"e2e-worker-w0-6af50f-openshift-default-8545fb68f-mp9v6\" (UID: \"1786cd64-e217-4f69-bcb6-6bda8aace53d\") " pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-mp9v6"
Nov 02 15:09:24.411298 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:24.411274    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1786cd64-e217-4f69-bcb6-6bda8aace53d-workload-socket\") pod \"e2e-worker-w0-6af50f-openshift-default-8545fb68f-mp9v6\" (UID: \"1786cd64-e217-4f69-bcb6-6bda8aace53d\") " pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-mp9v6"
Nov 02 15:09:24.411645 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:24.411619    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1786cd64-e217-4f69-bcb6-6bda8aace53d-istiod-ca-cert\") pod \"e2e-worker-w0-6af50f-openshift-default-8545fb68f-mp9v6\" (UID: \"1786cd64-e217-4f69-bcb6-6bda8aace53d\") " pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-mp9v6"
Nov 02 15:09:24.412870 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:24.412850    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1786cd64-e217-4f69-bcb6-6bda8aace53d-istio-envoy\") pod \"e2e-worker-w0-6af50f-openshift-default-8545fb68f-mp9v6\" (UID: \"1786cd64-e217-4f69-bcb6-6bda8aace53d\") " pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-mp9v6"
Nov 02 15:09:24.413159 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:24.413142    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1786cd64-e217-4f69-bcb6-6bda8aace53d-istio-podinfo\") pod \"e2e-worker-w0-6af50f-openshift-default-8545fb68f-mp9v6\" (UID: \"1786cd64-e217-4f69-bcb6-6bda8aace53d\") " pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-mp9v6"
Nov 02 15:09:24.463860 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:24.463826    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1786cd64-e217-4f69-bcb6-6bda8aace53d-istio-token\") pod \"e2e-worker-w0-6af50f-openshift-default-8545fb68f-mp9v6\" (UID: \"1786cd64-e217-4f69-bcb6-6bda8aace53d\") " pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-mp9v6"
Nov 02 15:09:24.464838 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:24.464460    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gtsq5\" (UniqueName: \"kubernetes.io/projected/1786cd64-e217-4f69-bcb6-6bda8aace53d-kube-api-access-gtsq5\") pod \"e2e-worker-w0-6af50f-openshift-default-8545fb68f-mp9v6\" (UID: \"1786cd64-e217-4f69-bcb6-6bda8aace53d\") " pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-mp9v6"
Nov 02 15:09:24.578744 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:24.577472    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-mp9v6"
Nov 02 15:09:24.605870 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:24.605835    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-b88h7" event={"ID":"3d4f7624-882c-4cb3-8811-9ca682e0c1d5","Type":"ContainerStarted","Data":"f308b7b6046e7500e8e6d2d6f77c217a3ab3773533f03a01f6b15aa9ca7b1d96"}
Nov 02 15:09:24.606008 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:24.605899    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-b88h7" event={"ID":"3d4f7624-882c-4cb3-8811-9ca682e0c1d5","Type":"ContainerStarted","Data":"19ed571e5ff8bcda7823a7e117f1d5b1b014be7c2df13bbd46a1a7072ec89928"}
Nov 02 15:09:24.638980 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:24.638560    5304 patch_prober.go:29] interesting pod/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-7nj6n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body=
Nov 02 15:09:24.638980 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:24.638616    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-7nj6n" podUID="3ebb8197-273d-4e96-bec7-00c7a85a12a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Nov 02 15:09:24.667268 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:24.665790    5304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-b88h7" podStartSLOduration=1.665770164 podStartE2EDuration="1.665770164s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:09:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:09:24.664164647 +0000 UTC m=+764.657989058" watchObservedRunningTime="2026-11-02 15:09:24.665770164 +0000 UTC m=+764.659594572"
Nov 02 15:09:24.817083 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:09:24.817051    5304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k6fjk" containerName="extractor"
Nov 02 15:09:24.937925 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:24.937891    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-mp9v6"]
Nov 02 15:09:24.941906 ip-10-0-135-138 kubenswrapper[5304]: W1102 15:09:24.941242    5304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1786cd64_e217_4f69_bcb6_6bda8aace53d.slice/crio-760e0529e852ab648e707ddfe2337490b33951940c378960d86b986f778b7825 WatchSource:0}: Error finding container 760e0529e852ab648e707ddfe2337490b33951940c378960d86b986f778b7825: Status 404 returned error can't find the container with id 760e0529e852ab648e707ddfe2337490b33951940c378960d86b986f778b7825
Nov 02 15:09:24.944018 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:24.943979    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:09:24.944115 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:24.944093    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:09:24.944181 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:24.944166    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:09:24.966790 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:24.966769    5304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-b88h7"
Nov 02 15:09:25.615152 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:25.614183    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-mp9v6" event={"ID":"1786cd64-e217-4f69-bcb6-6bda8aace53d","Type":"ContainerStarted","Data":"ffc66fd9acc334a377664485bc76bcce9222da324488bcf59f177a31a186afd9"}
Nov 02 15:09:25.615152 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:25.614262    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-mp9v6" event={"ID":"1786cd64-e217-4f69-bcb6-6bda8aace53d","Type":"ContainerStarted","Data":"760e0529e852ab648e707ddfe2337490b33951940c378960d86b986f778b7825"}
Nov 02 15:09:25.898176 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:25.897504    5304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-7nj6n"
Nov 02 15:09:25.898670 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:25.898619    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-7nj6n"
Nov 02 15:09:25.899745 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:25.899719    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-7nj6n"
Nov 02 15:09:25.984185 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:25.966472    5304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-mp9v6" podStartSLOduration=1.9664516779999999 podStartE2EDuration="1.966451678s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:09:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:09:25.693438493 +0000 UTC m=+765.687262903" watchObservedRunningTime="2026-11-02 15:09:25.966451678 +0000 UTC m=+765.960276089"
Nov 02 15:09:25.985359 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:25.984864    5304 patch_prober.go:29] interesting pod/e2e-worker-w3-624254-openshift-default-7988c9d8b5-b88h7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body=
Nov 02 15:09:25.985359 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:25.984928    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-b88h7" podUID="3d4f7624-882c-4cb3-8811-9ca682e0c1d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused"
Nov 02 15:09:26.582593 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:26.580506    5304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-mp9v6"
Nov 02 15:09:26.584385 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:26.584028    5304 patch_prober.go:29] interesting pod/e2e-worker-w0-6af50f-openshift-default-8545fb68f-mp9v6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Nov 02 15:09:26.584385 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:26.584101    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-mp9v6" podUID="1786cd64-e217-4f69-bcb6-6bda8aace53d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Nov 02 15:09:26.971209 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:26.970588    5304 patch_prober.go:29] interesting pod/e2e-worker-w3-624254-openshift-default-7988c9d8b5-b88h7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body=
Nov 02 15:09:26.971209 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:26.970651    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-b88h7" podUID="3d4f7624-882c-4cb3-8811-9ca682e0c1d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused"
Nov 02 15:09:27.580154 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:27.579563    5304 patch_prober.go:29] interesting pod/e2e-worker-w0-6af50f-openshift-default-8545fb68f-mp9v6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Nov 02 15:09:27.580154 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:27.579627    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-mp9v6" podUID="1786cd64-e217-4f69-bcb6-6bda8aace53d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Nov 02 15:09:27.976204 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:27.969622    5304 patch_prober.go:29] interesting pod/e2e-worker-w3-624254-openshift-default-7988c9d8b5-b88h7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body=
Nov 02 15:09:27.976204 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:27.969700    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-b88h7" podUID="3d4f7624-882c-4cb3-8811-9ca682e0c1d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused"
Nov 02 15:09:28.586149 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:28.584530    5304 patch_prober.go:29] interesting pod/e2e-worker-w0-6af50f-openshift-default-8545fb68f-mp9v6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Nov 02 15:09:28.586149 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:28.584595    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-mp9v6" podUID="1786cd64-e217-4f69-bcb6-6bda8aace53d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Nov 02 15:09:28.969209 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:28.968667    5304 patch_prober.go:29] interesting pod/e2e-worker-w3-624254-openshift-default-7988c9d8b5-b88h7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body=
Nov 02 15:09:28.969209 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:28.968763    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-b88h7" podUID="3d4f7624-882c-4cb3-8811-9ca682e0c1d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused"
Nov 02 15:09:29.588150 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:29.580482    5304 patch_prober.go:29] interesting pod/e2e-worker-w0-6af50f-openshift-default-8545fb68f-mp9v6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Nov 02 15:09:29.588150 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:29.580557    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-mp9v6" podUID="1786cd64-e217-4f69-bcb6-6bda8aace53d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Nov 02 15:09:29.972208 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:29.971582    5304 patch_prober.go:29] interesting pod/e2e-worker-w3-624254-openshift-default-7988c9d8b5-b88h7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body=
Nov 02 15:09:29.972208 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:29.971647    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-b88h7" podUID="3d4f7624-882c-4cb3-8811-9ca682e0c1d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused"
Nov 02 15:09:30.578602 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:30.578561    5304 patch_prober.go:29] interesting pod/e2e-worker-w0-6af50f-openshift-default-8545fb68f-mp9v6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Nov 02 15:09:30.578916 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:30.578883    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-mp9v6" podUID="1786cd64-e217-4f69-bcb6-6bda8aace53d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Nov 02 15:09:30.966992 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:30.966959    5304 patch_prober.go:29] interesting pod/e2e-worker-w3-624254-openshift-default-7988c9d8b5-b88h7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body=
Nov 02 15:09:30.967440 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:30.967012    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-b88h7" podUID="3d4f7624-882c-4cb3-8811-9ca682e0c1d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused"
Nov 02 15:09:31.578910 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:31.578870    5304 patch_prober.go:29] interesting pod/e2e-worker-w0-6af50f-openshift-default-8545fb68f-mp9v6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Nov 02 15:09:31.579248 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:31.578937    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-mp9v6" podUID="1786cd64-e217-4f69-bcb6-6bda8aace53d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Nov 02 15:09:32.541816 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:32.541784    5304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-b88h7"
Nov 02 15:09:32.542470 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:32.542451    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-b88h7"
Nov 02 15:09:32.544862 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:32.544842    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-b88h7"
Nov 02 15:09:32.582334 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:32.582302    5304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-mp9v6"
Nov 02 15:09:32.582600 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:32.582572    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-mp9v6"
Nov 02 15:09:32.583320 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:32.583303    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-mp9v6"
Nov 02 15:09:54.029801 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:54.029762    5304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-6q5dn"]
Nov 02 15:09:54.044000 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:54.043958    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-6q5dn"
Nov 02 15:09:54.159043 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:54.159009    5304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-cwnwk"]
Nov 02 15:09:54.166521 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:54.166484    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aee20b2e-e86a-44d0-a69c-069a9ca35892-workload-certs\") pod \"e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-6q5dn\" (UID: \"aee20b2e-e86a-44d0-a69c-069a9ca35892\") " pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-6q5dn"
Nov 02 15:09:54.167056 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:54.167031    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aee20b2e-e86a-44d0-a69c-069a9ca35892-istio-podinfo\") pod \"e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-6q5dn\" (UID: \"aee20b2e-e86a-44d0-a69c-069a9ca35892\") " pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-6q5dn"
Nov 02 15:09:54.167307 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:54.167288    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aee20b2e-e86a-44d0-a69c-069a9ca35892-istio-envoy\") pod \"e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-6q5dn\" (UID: \"aee20b2e-e86a-44d0-a69c-069a9ca35892\") " pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-6q5dn"
Nov 02 15:09:54.167485 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:54.167468    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aee20b2e-e86a-44d0-a69c-069a9ca35892-workload-socket\") pod \"e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-6q5dn\" (UID: \"aee20b2e-e86a-44d0-a69c-069a9ca35892\") " pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-6q5dn"
Nov 02 15:09:54.167599 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:54.167585    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aee20b2e-e86a-44d0-a69c-069a9ca35892-istio-data\") pod \"e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-6q5dn\" (UID: \"aee20b2e-e86a-44d0-a69c-069a9ca35892\") " pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-6q5dn"
Nov 02 15:09:54.167703 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:54.167690    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aee20b2e-e86a-44d0-a69c-069a9ca35892-istio-token\") pod \"e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-6q5dn\" (UID: \"aee20b2e-e86a-44d0-a69c-069a9ca35892\") " pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-6q5dn"
Nov 02 15:09:54.167948 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:54.167927    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aee20b2e-e86a-44d0-a69c-069a9ca35892-credential-socket\") pod \"e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-6q5dn\" (UID: \"aee20b2e-e86a-44d0-a69c-069a9ca35892\") " pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-6q5dn"
Nov 02 15:09:54.168144 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:54.168109    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8jfzm\" (UniqueName: \"kubernetes.io/projected/aee20b2e-e86a-44d0-a69c-069a9ca35892-kube-api-access-8jfzm\") pod \"e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-6q5dn\" (UID: \"aee20b2e-e86a-44d0-a69c-069a9ca35892\") " pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-6q5dn"
Nov 02 15:09:54.168332 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:54.168315    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aee20b2e-e86a-44d0-a69c-069a9ca35892-istiod-ca-cert\") pod \"e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-6q5dn\" (UID: \"aee20b2e-e86a-44d0-a69c-069a9ca35892\") " pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-6q5dn"
Nov 02 15:09:54.202474 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:54.202440    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-6q5dn"]
Nov 02 15:09:54.203702 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:54.203113    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-cwnwk"
Nov 02 15:09:54.276190 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:54.275096    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aee20b2e-e86a-44d0-a69c-069a9ca35892-istio-token\") pod \"e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-6q5dn\" (UID: \"aee20b2e-e86a-44d0-a69c-069a9ca35892\") " pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-6q5dn"
Nov 02 15:09:54.276515 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:54.276488    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79e8599a-22fa-4df3-a35a-4d16c271cfe2-istio-podinfo\") pod \"e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-cwnwk\" (UID: \"79e8599a-22fa-4df3-a35a-4d16c271cfe2\") " pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-cwnwk"
Nov 02 15:09:54.276775 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:54.276758    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79e8599a-22fa-4df3-a35a-4d16c271cfe2-istiod-ca-cert\") pod \"e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-cwnwk\" (UID: \"79e8599a-22fa-4df3-a35a-4d16c271cfe2\") " pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-cwnwk"
Nov 02 15:09:54.276910 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:54.276893    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vqmt9\" (UniqueName: \"kubernetes.io/projected/79e8599a-22fa-4df3-a35a-4d16c271cfe2-kube-api-access-vqmt9\") pod \"e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-cwnwk\" (UID: \"79e8599a-22fa-4df3-a35a-4d16c271cfe2\") " pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-cwnwk"
Nov 02 15:09:54.277045 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:54.277029    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aee20b2e-e86a-44d0-a69c-069a9ca35892-credential-socket\") pod \"e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-6q5dn\" (UID: \"aee20b2e-e86a-44d0-a69c-069a9ca35892\") " pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-6q5dn"
Nov 02 15:09:54.277247 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:54.277216    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8jfzm\" (UniqueName: \"kubernetes.io/projected/aee20b2e-e86a-44d0-a69c-069a9ca35892-kube-api-access-8jfzm\") pod \"e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-6q5dn\" (UID: \"aee20b2e-e86a-44d0-a69c-069a9ca35892\") " pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-6q5dn"
Nov 02 15:09:54.277400 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:54.277384    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aee20b2e-e86a-44d0-a69c-069a9ca35892-istiod-ca-cert\") pod \"e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-6q5dn\" (UID: \"aee20b2e-e86a-44d0-a69c-069a9ca35892\") " pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-6q5dn"
Nov 02 15:09:54.277576 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:54.277551    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79e8599a-22fa-4df3-a35a-4d16c271cfe2-istio-envoy\") pod \"e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-cwnwk\" (UID: \"79e8599a-22fa-4df3-a35a-4d16c271cfe2\") " pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-cwnwk"
Nov 02 15:09:54.278007 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:54.202759    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-cwnwk"]
Nov 02 15:09:54.278317 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:54.277943    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aee20b2e-e86a-44d0-a69c-069a9ca35892-workload-certs\") pod \"e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-6q5dn\" (UID: \"aee20b2e-e86a-44d0-a69c-069a9ca35892\") " pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-6q5dn"
Nov 02 15:09:54.278726 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:54.278708    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aee20b2e-e86a-44d0-a69c-069a9ca35892-istio-podinfo\") pod \"e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-6q5dn\" (UID: \"aee20b2e-e86a-44d0-a69c-069a9ca35892\") " pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-6q5dn"
Nov 02 15:09:54.278889 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:54.278868    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79e8599a-22fa-4df3-a35a-4d16c271cfe2-workload-socket\") pod \"e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-cwnwk\" (UID: \"79e8599a-22fa-4df3-a35a-4d16c271cfe2\") " pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-cwnwk"
Nov 02 15:09:54.279560 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:54.279538    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aee20b2e-e86a-44d0-a69c-069a9ca35892-istio-envoy\") pod \"e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-6q5dn\" (UID: \"aee20b2e-e86a-44d0-a69c-069a9ca35892\") " pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-6q5dn"
Nov 02 15:09:54.280398 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:54.280336    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79e8599a-22fa-4df3-a35a-4d16c271cfe2-istio-data\") pod \"e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-cwnwk\" (UID: \"79e8599a-22fa-4df3-a35a-4d16c271cfe2\") " pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-cwnwk"
Nov 02 15:09:54.280578 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:54.280550    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79e8599a-22fa-4df3-a35a-4d16c271cfe2-workload-certs\") pod \"e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-cwnwk\" (UID: \"79e8599a-22fa-4df3-a35a-4d16c271cfe2\") " pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-cwnwk"
Nov 02 15:09:54.280765 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:54.280706    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79e8599a-22fa-4df3-a35a-4d16c271cfe2-istio-token\") pod \"e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-cwnwk\" (UID: \"79e8599a-22fa-4df3-a35a-4d16c271cfe2\") " pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-cwnwk"
Nov 02 15:09:54.281052 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:54.279775    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aee20b2e-e86a-44d0-a69c-069a9ca35892-credential-socket\") pod \"e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-6q5dn\" (UID: \"aee20b2e-e86a-44d0-a69c-069a9ca35892\") " pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-6q5dn"
Nov 02 15:09:54.281982 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:54.280254    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aee20b2e-e86a-44d0-a69c-069a9ca35892-workload-certs\") pod \"e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-6q5dn\" (UID: \"aee20b2e-e86a-44d0-a69c-069a9ca35892\") " pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-6q5dn"
Nov 02 15:09:54.282157 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:54.282100    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79e8599a-22fa-4df3-a35a-4d16c271cfe2-credential-socket\") pod \"e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-cwnwk\" (UID: \"79e8599a-22fa-4df3-a35a-4d16c271cfe2\") " pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-cwnwk"
Nov 02 15:09:54.282418 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:54.282311    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aee20b2e-e86a-44d0-a69c-069a9ca35892-workload-socket\") pod \"e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-6q5dn\" (UID: \"aee20b2e-e86a-44d0-a69c-069a9ca35892\") " pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-6q5dn"
Nov 02 15:09:54.282822 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:54.282795    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aee20b2e-e86a-44d0-a69c-069a9ca35892-istio-data\") pod \"e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-6q5dn\" (UID: \"aee20b2e-e86a-44d0-a69c-069a9ca35892\") " pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-6q5dn"
Nov 02 15:09:54.283404 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:54.283386    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aee20b2e-e86a-44d0-a69c-069a9ca35892-workload-socket\") pod \"e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-6q5dn\" (UID: \"aee20b2e-e86a-44d0-a69c-069a9ca35892\") " pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-6q5dn"
Nov 02 15:09:54.283771 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:54.283752    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aee20b2e-e86a-44d0-a69c-069a9ca35892-istio-data\") pod \"e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-6q5dn\" (UID: \"aee20b2e-e86a-44d0-a69c-069a9ca35892\") " pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-6q5dn"
Nov 02 15:09:54.287180 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:54.287117    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aee20b2e-e86a-44d0-a69c-069a9ca35892-istio-envoy\") pod \"e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-6q5dn\" (UID: \"aee20b2e-e86a-44d0-a69c-069a9ca35892\") " pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-6q5dn"
Nov 02 15:09:54.288732 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:54.288712    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aee20b2e-e86a-44d0-a69c-069a9ca35892-istiod-ca-cert\") pod \"e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-6q5dn\" (UID: \"aee20b2e-e86a-44d0-a69c-069a9ca35892\") " pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-6q5dn"
Nov 02 15:09:54.291365 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:54.291320    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aee20b2e-e86a-44d0-a69c-069a9ca35892-istio-podinfo\") pod \"e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-6q5dn\" (UID: \"aee20b2e-e86a-44d0-a69c-069a9ca35892\") " pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-6q5dn"
Nov 02 15:09:54.395160 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:54.371681    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aee20b2e-e86a-44d0-a69c-069a9ca35892-istio-token\") pod \"e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-6q5dn\" (UID: \"aee20b2e-e86a-44d0-a69c-069a9ca35892\") " pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-6q5dn"
Nov 02 15:09:54.408791 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:54.408753    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79e8599a-22fa-4df3-a35a-4d16c271cfe2-istio-podinfo\") pod \"e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-cwnwk\" (UID: \"79e8599a-22fa-4df3-a35a-4d16c271cfe2\") " pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-cwnwk"
Nov 02 15:09:54.409158 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:54.409113    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79e8599a-22fa-4df3-a35a-4d16c271cfe2-istiod-ca-cert\") pod \"e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-cwnwk\" (UID: \"79e8599a-22fa-4df3-a35a-4d16c271cfe2\") " pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-cwnwk"
Nov 02 15:09:54.409318 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:54.409302    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vqmt9\" (UniqueName: \"kubernetes.io/projected/79e8599a-22fa-4df3-a35a-4d16c271cfe2-kube-api-access-vqmt9\") pod \"e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-cwnwk\" (UID: \"79e8599a-22fa-4df3-a35a-4d16c271cfe2\") " pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-cwnwk"
Nov 02 15:09:54.434494 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:54.415906    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8jfzm\" (UniqueName: \"kubernetes.io/projected/aee20b2e-e86a-44d0-a69c-069a9ca35892-kube-api-access-8jfzm\") pod \"e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-6q5dn\" (UID: \"aee20b2e-e86a-44d0-a69c-069a9ca35892\") " pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-6q5dn"
Nov 02 15:09:54.434494 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:54.419174    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79e8599a-22fa-4df3-a35a-4d16c271cfe2-istiod-ca-cert\") pod \"e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-cwnwk\" (UID: \"79e8599a-22fa-4df3-a35a-4d16c271cfe2\") " pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-cwnwk"
Nov 02 15:09:54.434494 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:54.423270    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79e8599a-22fa-4df3-a35a-4d16c271cfe2-istio-envoy\") pod \"e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-cwnwk\" (UID: \"79e8599a-22fa-4df3-a35a-4d16c271cfe2\") " pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-cwnwk"
Nov 02 15:09:54.434494 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:54.423426    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79e8599a-22fa-4df3-a35a-4d16c271cfe2-workload-socket\") pod \"e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-cwnwk\" (UID: \"79e8599a-22fa-4df3-a35a-4d16c271cfe2\") " pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-cwnwk"
Nov 02 15:09:54.434494 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:54.423461    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79e8599a-22fa-4df3-a35a-4d16c271cfe2-istio-data\") pod \"e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-cwnwk\" (UID: \"79e8599a-22fa-4df3-a35a-4d16c271cfe2\") " pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-cwnwk"
Nov 02 15:09:54.434494 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:54.423500    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79e8599a-22fa-4df3-a35a-4d16c271cfe2-workload-certs\") pod \"e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-cwnwk\" (UID: \"79e8599a-22fa-4df3-a35a-4d16c271cfe2\") " pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-cwnwk"
Nov 02 15:09:54.434494 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:54.423535    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79e8599a-22fa-4df3-a35a-4d16c271cfe2-istio-token\") pod \"e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-cwnwk\" (UID: \"79e8599a-22fa-4df3-a35a-4d16c271cfe2\") " pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-cwnwk"
Nov 02 15:09:54.434494 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:54.423997    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79e8599a-22fa-4df3-a35a-4d16c271cfe2-workload-certs\") pod \"e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-cwnwk\" (UID: \"79e8599a-22fa-4df3-a35a-4d16c271cfe2\") " pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-cwnwk"
Nov 02 15:09:54.434494 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:54.424744    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79e8599a-22fa-4df3-a35a-4d16c271cfe2-workload-socket\") pod \"e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-cwnwk\" (UID: \"79e8599a-22fa-4df3-a35a-4d16c271cfe2\") " pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-cwnwk"
Nov 02 15:09:54.434494 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:54.425033    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79e8599a-22fa-4df3-a35a-4d16c271cfe2-credential-socket\") pod \"e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-cwnwk\" (UID: \"79e8599a-22fa-4df3-a35a-4d16c271cfe2\") " pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-cwnwk"
Nov 02 15:09:54.434494 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:54.425340    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79e8599a-22fa-4df3-a35a-4d16c271cfe2-istio-data\") pod \"e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-cwnwk\" (UID: \"79e8599a-22fa-4df3-a35a-4d16c271cfe2\") " pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-cwnwk"
Nov 02 15:09:54.435929 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:54.435394    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79e8599a-22fa-4df3-a35a-4d16c271cfe2-istio-envoy\") pod \"e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-cwnwk\" (UID: \"79e8599a-22fa-4df3-a35a-4d16c271cfe2\") " pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-cwnwk"
Nov 02 15:09:54.435929 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:54.435877    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79e8599a-22fa-4df3-a35a-4d16c271cfe2-istio-podinfo\") pod \"e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-cwnwk\" (UID: \"79e8599a-22fa-4df3-a35a-4d16c271cfe2\") " pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-cwnwk"
Nov 02 15:09:54.436349 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:54.435937    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-6q5dn"
Nov 02 15:09:54.449953 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:54.442177    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79e8599a-22fa-4df3-a35a-4d16c271cfe2-credential-socket\") pod \"e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-cwnwk\" (UID: \"79e8599a-22fa-4df3-a35a-4d16c271cfe2\") " pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-cwnwk"
Nov 02 15:09:54.460143 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:54.456786    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vqmt9\" (UniqueName: \"kubernetes.io/projected/79e8599a-22fa-4df3-a35a-4d16c271cfe2-kube-api-access-vqmt9\") pod \"e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-cwnwk\" (UID: \"79e8599a-22fa-4df3-a35a-4d16c271cfe2\") " pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-cwnwk"
Nov 02 15:09:54.507683 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:54.507654    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79e8599a-22fa-4df3-a35a-4d16c271cfe2-istio-token\") pod \"e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-cwnwk\" (UID: \"79e8599a-22fa-4df3-a35a-4d16c271cfe2\") " pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-cwnwk"
Nov 02 15:09:54.587646 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:54.587379    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-cwnwk"
Nov 02 15:09:54.993772 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:54.993734    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-6q5dn"]
Nov 02 15:09:55.108147 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:55.098099    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:09:55.108147 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:55.098215    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:09:55.108147 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:55.098266    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:09:55.392286 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:55.392254    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-cwnwk"]
Nov 02 15:09:55.448873 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:55.436650    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:09:55.448873 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:55.436762    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:09:55.448873 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:55.436807    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:09:55.588150 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:09:55.587444    5304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k6fjk" containerName="extractor"
Nov 02 15:09:55.925265 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:55.925216    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-cwnwk" event={"ID":"79e8599a-22fa-4df3-a35a-4d16c271cfe2","Type":"ContainerStarted","Data":"165e1e14e0dc19782087cbf0db0d14128fb6fd4ec462abe1c9b3ae9d0ea73fe3"}
Nov 02 15:09:55.925460 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:55.925282    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-cwnwk" event={"ID":"79e8599a-22fa-4df3-a35a-4d16c271cfe2","Type":"ContainerStarted","Data":"6076748778bb853bb1a421f610a8aa9a94ae135b15c3c6200f836acf6aba3a66"}
Nov 02 15:09:55.927227 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:55.927198    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-6q5dn" event={"ID":"aee20b2e-e86a-44d0-a69c-069a9ca35892","Type":"ContainerStarted","Data":"f631f140ca6152560e503207d387e6965b8c0c336f4e093e33b9da1ed3f9c7c8"}
Nov 02 15:09:55.927374 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:55.927247    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-6q5dn" event={"ID":"aee20b2e-e86a-44d0-a69c-069a9ca35892","Type":"ContainerStarted","Data":"89f96020de1fe121d1e64096c344b79654da577fab9453c06be8ab0a11eb9955"}
Nov 02 15:09:55.976202 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:55.975986    5304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-cwnwk" podStartSLOduration=1.975964584 podStartE2EDuration="1.975964584s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:09:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:09:55.973500707 +0000 UTC m=+795.967325117" watchObservedRunningTime="2026-11-02 15:09:55.975964584 +0000 UTC m=+795.969788994"
Nov 02 15:09:56.025004 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:56.024944    5304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-6q5dn" podStartSLOduration=3.024928702 podStartE2EDuration="3.024928702s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:09:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:09:56.018717075 +0000 UTC m=+796.012541485" watchObservedRunningTime="2026-11-02 15:09:56.024928702 +0000 UTC m=+796.018753111"
Nov 02 15:09:56.441150 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:56.439478    5304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-6q5dn"
Nov 02 15:09:56.447151 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:56.445271    5304 patch_prober.go:29] interesting pod/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-6q5dn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Nov 02 15:09:56.447151 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:56.445338    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-6q5dn" podUID="aee20b2e-e86a-44d0-a69c-069a9ca35892" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Nov 02 15:09:56.588480 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:56.588439    5304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-cwnwk"
Nov 02 15:09:56.589978 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:56.589950    5304 patch_prober.go:29] interesting pod/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-cwnwk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body=
Nov 02 15:09:56.590185 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:56.590163    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-cwnwk" podUID="79e8599a-22fa-4df3-a35a-4d16c271cfe2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused"
Nov 02 15:09:57.439616 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:57.439574    5304 patch_prober.go:29] interesting pod/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-6q5dn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Nov 02 15:09:57.439905 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:57.439876    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-6q5dn" podUID="aee20b2e-e86a-44d0-a69c-069a9ca35892" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Nov 02 15:09:57.588737 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:57.588704    5304 patch_prober.go:29] interesting pod/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-cwnwk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body=
Nov 02 15:09:57.589189 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:57.588762    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-cwnwk" podUID="79e8599a-22fa-4df3-a35a-4d16c271cfe2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused"
Nov 02 15:09:58.459154 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:58.438580    5304 patch_prober.go:29] interesting pod/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-6q5dn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Nov 02 15:09:58.459494 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:58.459451    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-6q5dn" podUID="aee20b2e-e86a-44d0-a69c-069a9ca35892" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Nov 02 15:09:58.588365 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:58.588324    5304 patch_prober.go:29] interesting pod/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-cwnwk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body=
Nov 02 15:09:58.588549 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:58.588387    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-cwnwk" podUID="79e8599a-22fa-4df3-a35a-4d16c271cfe2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused"
Nov 02 15:09:59.438613 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:59.438578    5304 patch_prober.go:29] interesting pod/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-6q5dn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Nov 02 15:09:59.439753 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:59.439717    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-6q5dn" podUID="aee20b2e-e86a-44d0-a69c-069a9ca35892" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Nov 02 15:09:59.589007 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:59.588969    5304 patch_prober.go:29] interesting pod/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-cwnwk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body=
Nov 02 15:09:59.589199 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:09:59.589029    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-cwnwk" podUID="79e8599a-22fa-4df3-a35a-4d16c271cfe2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused"
Nov 02 15:10:00.444563 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:00.444524    5304 patch_prober.go:29] interesting pod/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-6q5dn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Nov 02 15:10:00.449289 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:00.449235    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-6q5dn" podUID="aee20b2e-e86a-44d0-a69c-069a9ca35892" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Nov 02 15:10:00.588965 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:00.588924    5304 patch_prober.go:29] interesting pod/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-cwnwk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body=
Nov 02 15:10:00.589167 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:00.588985    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-cwnwk" podUID="79e8599a-22fa-4df3-a35a-4d16c271cfe2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused"
Nov 02 15:10:01.438558 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:01.438518    5304 patch_prober.go:29] interesting pod/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-6q5dn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Nov 02 15:10:01.438855 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:01.438825    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-6q5dn" podUID="aee20b2e-e86a-44d0-a69c-069a9ca35892" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Nov 02 15:10:01.596520 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:01.596475    5304 patch_prober.go:29] interesting pod/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-cwnwk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body=
Nov 02 15:10:01.597051 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:01.596532    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-cwnwk" podUID="79e8599a-22fa-4df3-a35a-4d16c271cfe2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused"
Nov 02 15:10:02.437772 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:02.437726    5304 patch_prober.go:29] interesting pod/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-6q5dn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Nov 02 15:10:02.437956 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:02.437801    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-6q5dn" podUID="aee20b2e-e86a-44d0-a69c-069a9ca35892" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Nov 02 15:10:02.588426 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:02.588371    5304 patch_prober.go:29] interesting pod/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-cwnwk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body=
Nov 02 15:10:02.588625 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:02.588444    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-cwnwk" podUID="79e8599a-22fa-4df3-a35a-4d16c271cfe2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused"
Nov 02 15:10:03.441792 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:03.441762    5304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-6q5dn"
Nov 02 15:10:03.442249 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:03.442034    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-6q5dn"
Nov 02 15:10:03.442673 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:03.442653    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-6q5dn"
Nov 02 15:10:03.592840 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:03.592812    5304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-cwnwk"
Nov 02 15:10:03.593265 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:03.593241    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-cwnwk"
Nov 02 15:10:03.594039 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:03.594022    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-cwnwk"
Nov 02 15:10:08.397609 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:08.397568    5304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-624254/distinct-w3-624254-kserve-5465dc4765-dmq67"]
Nov 02 15:10:08.402725 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:08.402703    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-624254/distinct-w3-624254-kserve-5465dc4765-dmq67"
Nov 02 15:10:08.432699 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:08.432669    5304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-624254\"/\"distinct-w3-624254-kserve-self-signed-certs\""
Nov 02 15:10:08.435321 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:08.435296    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-624254/distinct-w3-624254-kserve-5465dc4765-dmq67"]
Nov 02 15:10:08.484752 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:08.484721    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1a5aa910-a6de-417b-8638-81e4ec5c1825-home\") pod \"distinct-w3-624254-kserve-5465dc4765-dmq67\" (UID: \"1a5aa910-a6de-417b-8638-81e4ec5c1825\") " pod="e2e-models-e2e-worker-w3-624254/distinct-w3-624254-kserve-5465dc4765-dmq67"
Nov 02 15:10:08.484960 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:08.484755    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a5aa910-a6de-417b-8638-81e4ec5c1825-tls-certs\") pod \"distinct-w3-624254-kserve-5465dc4765-dmq67\" (UID: \"1a5aa910-a6de-417b-8638-81e4ec5c1825\") " pod="e2e-models-e2e-worker-w3-624254/distinct-w3-624254-kserve-5465dc4765-dmq67"
Nov 02 15:10:08.485040 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:08.484951    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c9r42\" (UniqueName: \"kubernetes.io/projected/1a5aa910-a6de-417b-8638-81e4ec5c1825-kube-api-access-c9r42\") pod \"distinct-w3-624254-kserve-5465dc4765-dmq67\" (UID: \"1a5aa910-a6de-417b-8638-81e4ec5c1825\") " pod="e2e-models-e2e-worker-w3-624254/distinct-w3-624254-kserve-5465dc4765-dmq67"
Nov 02 15:10:08.485040 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:08.485019    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1a5aa910-a6de-417b-8638-81e4ec5c1825-dshm\") pod \"distinct-w3-624254-kserve-5465dc4765-dmq67\" (UID: \"1a5aa910-a6de-417b-8638-81e4ec5c1825\") " pod="e2e-models-e2e-worker-w3-624254/distinct-w3-624254-kserve-5465dc4765-dmq67"
Nov 02 15:10:08.485196 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:08.485049    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a5aa910-a6de-417b-8638-81e4ec5c1825-tmp-dir\") pod \"distinct-w3-624254-kserve-5465dc4765-dmq67\" (UID: \"1a5aa910-a6de-417b-8638-81e4ec5c1825\") " pod="e2e-models-e2e-worker-w3-624254/distinct-w3-624254-kserve-5465dc4765-dmq67"
Nov 02 15:10:08.485196 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:08.485172    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1a5aa910-a6de-417b-8638-81e4ec5c1825-model-cache\") pod \"distinct-w3-624254-kserve-5465dc4765-dmq67\" (UID: \"1a5aa910-a6de-417b-8638-81e4ec5c1825\") " pod="e2e-models-e2e-worker-w3-624254/distinct-w3-624254-kserve-5465dc4765-dmq67"
Nov 02 15:10:08.586528 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:08.586488    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1a5aa910-a6de-417b-8638-81e4ec5c1825-home\") pod \"distinct-w3-624254-kserve-5465dc4765-dmq67\" (UID: \"1a5aa910-a6de-417b-8638-81e4ec5c1825\") " pod="e2e-models-e2e-worker-w3-624254/distinct-w3-624254-kserve-5465dc4765-dmq67"
Nov 02 15:10:08.586702 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:08.586536    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a5aa910-a6de-417b-8638-81e4ec5c1825-tls-certs\") pod \"distinct-w3-624254-kserve-5465dc4765-dmq67\" (UID: \"1a5aa910-a6de-417b-8638-81e4ec5c1825\") " pod="e2e-models-e2e-worker-w3-624254/distinct-w3-624254-kserve-5465dc4765-dmq67"
Nov 02 15:10:08.586745 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:08.586700    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c9r42\" (UniqueName: \"kubernetes.io/projected/1a5aa910-a6de-417b-8638-81e4ec5c1825-kube-api-access-c9r42\") pod \"distinct-w3-624254-kserve-5465dc4765-dmq67\" (UID: \"1a5aa910-a6de-417b-8638-81e4ec5c1825\") " pod="e2e-models-e2e-worker-w3-624254/distinct-w3-624254-kserve-5465dc4765-dmq67"
Nov 02 15:10:08.586783 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:08.586759    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1a5aa910-a6de-417b-8638-81e4ec5c1825-dshm\") pod \"distinct-w3-624254-kserve-5465dc4765-dmq67\" (UID: \"1a5aa910-a6de-417b-8638-81e4ec5c1825\") " pod="e2e-models-e2e-worker-w3-624254/distinct-w3-624254-kserve-5465dc4765-dmq67"
Nov 02 15:10:08.586814 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:08.586786    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a5aa910-a6de-417b-8638-81e4ec5c1825-tmp-dir\") pod \"distinct-w3-624254-kserve-5465dc4765-dmq67\" (UID: \"1a5aa910-a6de-417b-8638-81e4ec5c1825\") " pod="e2e-models-e2e-worker-w3-624254/distinct-w3-624254-kserve-5465dc4765-dmq67"
Nov 02 15:10:08.586855 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:08.586847    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1a5aa910-a6de-417b-8638-81e4ec5c1825-model-cache\") pod \"distinct-w3-624254-kserve-5465dc4765-dmq67\" (UID: \"1a5aa910-a6de-417b-8638-81e4ec5c1825\") " pod="e2e-models-e2e-worker-w3-624254/distinct-w3-624254-kserve-5465dc4765-dmq67"
Nov 02 15:10:08.586956 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:08.586930    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1a5aa910-a6de-417b-8638-81e4ec5c1825-home\") pod \"distinct-w3-624254-kserve-5465dc4765-dmq67\" (UID: \"1a5aa910-a6de-417b-8638-81e4ec5c1825\") " pod="e2e-models-e2e-worker-w3-624254/distinct-w3-624254-kserve-5465dc4765-dmq67"
Nov 02 15:10:08.587253 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:08.587221    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a5aa910-a6de-417b-8638-81e4ec5c1825-tmp-dir\") pod \"distinct-w3-624254-kserve-5465dc4765-dmq67\" (UID: \"1a5aa910-a6de-417b-8638-81e4ec5c1825\") " pod="e2e-models-e2e-worker-w3-624254/distinct-w3-624254-kserve-5465dc4765-dmq67"
Nov 02 15:10:08.587428 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:08.587270    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1a5aa910-a6de-417b-8638-81e4ec5c1825-model-cache\") pod \"distinct-w3-624254-kserve-5465dc4765-dmq67\" (UID: \"1a5aa910-a6de-417b-8638-81e4ec5c1825\") " pod="e2e-models-e2e-worker-w3-624254/distinct-w3-624254-kserve-5465dc4765-dmq67"
Nov 02 15:10:08.589870 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:08.589839    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1a5aa910-a6de-417b-8638-81e4ec5c1825-dshm\") pod \"distinct-w3-624254-kserve-5465dc4765-dmq67\" (UID: \"1a5aa910-a6de-417b-8638-81e4ec5c1825\") " pod="e2e-models-e2e-worker-w3-624254/distinct-w3-624254-kserve-5465dc4765-dmq67"
Nov 02 15:10:08.627210 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:08.617447    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a5aa910-a6de-417b-8638-81e4ec5c1825-tls-certs\") pod \"distinct-w3-624254-kserve-5465dc4765-dmq67\" (UID: \"1a5aa910-a6de-417b-8638-81e4ec5c1825\") " pod="e2e-models-e2e-worker-w3-624254/distinct-w3-624254-kserve-5465dc4765-dmq67"
Nov 02 15:10:08.627485 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:08.623237    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c9r42\" (UniqueName: \"kubernetes.io/projected/1a5aa910-a6de-417b-8638-81e4ec5c1825-kube-api-access-c9r42\") pod \"distinct-w3-624254-kserve-5465dc4765-dmq67\" (UID: \"1a5aa910-a6de-417b-8638-81e4ec5c1825\") " pod="e2e-models-e2e-worker-w3-624254/distinct-w3-624254-kserve-5465dc4765-dmq67"
Nov 02 15:10:08.669584 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:08.669483    5304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-25flx"]
Nov 02 15:10:08.674208 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:08.674183    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-25flx"
Nov 02 15:10:08.707758 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:08.707724    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-25flx"]
Nov 02 15:10:08.717668 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:08.717637    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-624254/distinct-w3-624254-kserve-5465dc4765-dmq67"
Nov 02 15:10:08.790189 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:08.789767    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f80b62a-49ba-4922-8f3e-804d6bdcc46f-workload-socket\") pod \"e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-25flx\" (UID: \"6f80b62a-49ba-4922-8f3e-804d6bdcc46f\") " pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-25flx"
Nov 02 15:10:08.790189 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:08.789820    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f80b62a-49ba-4922-8f3e-804d6bdcc46f-istiod-ca-cert\") pod \"e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-25flx\" (UID: \"6f80b62a-49ba-4922-8f3e-804d6bdcc46f\") " pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-25flx"
Nov 02 15:10:08.790189 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:08.789865    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f80b62a-49ba-4922-8f3e-804d6bdcc46f-istio-token\") pod \"e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-25flx\" (UID: \"6f80b62a-49ba-4922-8f3e-804d6bdcc46f\") " pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-25flx"
Nov 02 15:10:08.790189 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:08.790006    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f80b62a-49ba-4922-8f3e-804d6bdcc46f-istio-podinfo\") pod \"e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-25flx\" (UID: \"6f80b62a-49ba-4922-8f3e-804d6bdcc46f\") " pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-25flx"
Nov 02 15:10:08.790189 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:08.790075    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h4p8n\" (UniqueName: \"kubernetes.io/projected/6f80b62a-49ba-4922-8f3e-804d6bdcc46f-kube-api-access-h4p8n\") pod \"e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-25flx\" (UID: \"6f80b62a-49ba-4922-8f3e-804d6bdcc46f\") " pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-25flx"
Nov 02 15:10:08.790607 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:08.790306    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f80b62a-49ba-4922-8f3e-804d6bdcc46f-credential-socket\") pod \"e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-25flx\" (UID: \"6f80b62a-49ba-4922-8f3e-804d6bdcc46f\") " pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-25flx"
Nov 02 15:10:08.790607 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:08.790346    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f80b62a-49ba-4922-8f3e-804d6bdcc46f-istio-data\") pod \"e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-25flx\" (UID: \"6f80b62a-49ba-4922-8f3e-804d6bdcc46f\") " pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-25flx"
Nov 02 15:10:08.790607 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:08.790380    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f80b62a-49ba-4922-8f3e-804d6bdcc46f-istio-envoy\") pod \"e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-25flx\" (UID: \"6f80b62a-49ba-4922-8f3e-804d6bdcc46f\") " pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-25flx"
Nov 02 15:10:08.790607 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:08.790407    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f80b62a-49ba-4922-8f3e-804d6bdcc46f-workload-certs\") pod \"e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-25flx\" (UID: \"6f80b62a-49ba-4922-8f3e-804d6bdcc46f\") " pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-25flx"
Nov 02 15:10:08.891999 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:08.891964    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f80b62a-49ba-4922-8f3e-804d6bdcc46f-credential-socket\") pod \"e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-25flx\" (UID: \"6f80b62a-49ba-4922-8f3e-804d6bdcc46f\") " pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-25flx"
Nov 02 15:10:08.892235 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:08.892008    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f80b62a-49ba-4922-8f3e-804d6bdcc46f-istio-data\") pod \"e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-25flx\" (UID: \"6f80b62a-49ba-4922-8f3e-804d6bdcc46f\") " pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-25flx"
Nov 02 15:10:08.892235 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:08.892025    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f80b62a-49ba-4922-8f3e-804d6bdcc46f-istio-envoy\") pod \"e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-25flx\" (UID: \"6f80b62a-49ba-4922-8f3e-804d6bdcc46f\") " pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-25flx"
Nov 02 15:10:08.892235 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:08.892048    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f80b62a-49ba-4922-8f3e-804d6bdcc46f-workload-certs\") pod \"e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-25flx\" (UID: \"6f80b62a-49ba-4922-8f3e-804d6bdcc46f\") " pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-25flx"
Nov 02 15:10:08.892235 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:08.892225    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f80b62a-49ba-4922-8f3e-804d6bdcc46f-workload-socket\") pod \"e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-25flx\" (UID: \"6f80b62a-49ba-4922-8f3e-804d6bdcc46f\") " pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-25flx"
Nov 02 15:10:08.892468 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:08.892272    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f80b62a-49ba-4922-8f3e-804d6bdcc46f-istiod-ca-cert\") pod \"e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-25flx\" (UID: \"6f80b62a-49ba-4922-8f3e-804d6bdcc46f\") " pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-25flx"
Nov 02 15:10:08.892468 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:08.892303    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f80b62a-49ba-4922-8f3e-804d6bdcc46f-istio-token\") pod \"e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-25flx\" (UID: \"6f80b62a-49ba-4922-8f3e-804d6bdcc46f\") " pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-25flx"
Nov 02 15:10:08.892468 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:08.892354    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f80b62a-49ba-4922-8f3e-804d6bdcc46f-istio-podinfo\") pod \"e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-25flx\" (UID: \"6f80b62a-49ba-4922-8f3e-804d6bdcc46f\") " pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-25flx"
Nov 02 15:10:08.892468 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:08.892394    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h4p8n\" (UniqueName: \"kubernetes.io/projected/6f80b62a-49ba-4922-8f3e-804d6bdcc46f-kube-api-access-h4p8n\") pod \"e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-25flx\" (UID: \"6f80b62a-49ba-4922-8f3e-804d6bdcc46f\") " pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-25flx"
Nov 02 15:10:08.892468 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:08.892442    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f80b62a-49ba-4922-8f3e-804d6bdcc46f-credential-socket\") pod \"e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-25flx\" (UID: \"6f80b62a-49ba-4922-8f3e-804d6bdcc46f\") " pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-25flx"
Nov 02 15:10:08.892724 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:08.892542    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f80b62a-49ba-4922-8f3e-804d6bdcc46f-istio-data\") pod \"e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-25flx\" (UID: \"6f80b62a-49ba-4922-8f3e-804d6bdcc46f\") " pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-25flx"
Nov 02 15:10:08.892724 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:08.892601    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f80b62a-49ba-4922-8f3e-804d6bdcc46f-workload-socket\") pod \"e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-25flx\" (UID: \"6f80b62a-49ba-4922-8f3e-804d6bdcc46f\") " pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-25flx"
Nov 02 15:10:08.892724 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:08.892600    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f80b62a-49ba-4922-8f3e-804d6bdcc46f-workload-certs\") pod \"e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-25flx\" (UID: \"6f80b62a-49ba-4922-8f3e-804d6bdcc46f\") " pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-25flx"
Nov 02 15:10:08.893000 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:08.892979    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f80b62a-49ba-4922-8f3e-804d6bdcc46f-istiod-ca-cert\") pod \"e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-25flx\" (UID: \"6f80b62a-49ba-4922-8f3e-804d6bdcc46f\") " pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-25flx"
Nov 02 15:10:08.894427 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:08.894406    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f80b62a-49ba-4922-8f3e-804d6bdcc46f-istio-envoy\") pod \"e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-25flx\" (UID: \"6f80b62a-49ba-4922-8f3e-804d6bdcc46f\") " pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-25flx"
Nov 02 15:10:08.894788 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:08.894770    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f80b62a-49ba-4922-8f3e-804d6bdcc46f-istio-podinfo\") pod \"e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-25flx\" (UID: \"6f80b62a-49ba-4922-8f3e-804d6bdcc46f\") " pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-25flx"
Nov 02 15:10:08.915712 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:08.915688    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f80b62a-49ba-4922-8f3e-804d6bdcc46f-istio-token\") pod \"e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-25flx\" (UID: \"6f80b62a-49ba-4922-8f3e-804d6bdcc46f\") " pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-25flx"
Nov 02 15:10:08.915824 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:08.915806    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h4p8n\" (UniqueName: \"kubernetes.io/projected/6f80b62a-49ba-4922-8f3e-804d6bdcc46f-kube-api-access-h4p8n\") pod \"e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-25flx\" (UID: \"6f80b62a-49ba-4922-8f3e-804d6bdcc46f\") " pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-25flx"
Nov 02 15:10:08.995269 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:08.995187    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-25flx"
Nov 02 15:10:09.042038 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:09.042006    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-624254/distinct-w3-624254-kserve-5465dc4765-dmq67"]
Nov 02 15:10:09.045462 ip-10-0-135-138 kubenswrapper[5304]: W1102 15:10:09.045425    5304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1a5aa910_a6de_417b_8638_81e4ec5c1825.slice/crio-d0e79cbb2555ba5258b965ab5be4703d6a0fe741d9d5d364187904dee3d76cbc WatchSource:0}: Error finding container d0e79cbb2555ba5258b965ab5be4703d6a0fe741d9d5d364187904dee3d76cbc: Status 404 returned error can't find the container with id d0e79cbb2555ba5258b965ab5be4703d6a0fe741d9d5d364187904dee3d76cbc
Nov 02 15:10:09.074390 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:09.074362    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-624254/distinct-w3-624254-kserve-5465dc4765-dmq67" event={"ID":"1a5aa910-a6de-417b-8638-81e4ec5c1825","Type":"ContainerStarted","Data":"d0e79cbb2555ba5258b965ab5be4703d6a0fe741d9d5d364187904dee3d76cbc"}
Nov 02 15:10:09.304669 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:09.304505    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-25flx"]
Nov 02 15:10:09.307520 ip-10-0-135-138 kubenswrapper[5304]: W1102 15:10:09.307490    5304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f80b62a_49ba_4922_8f3e_804d6bdcc46f.slice/crio-962ed1c22a2df2872c45e5df6e5223119ad3f40b6a85e995a11141f9cebc6b97 WatchSource:0}: Error finding container 962ed1c22a2df2872c45e5df6e5223119ad3f40b6a85e995a11141f9cebc6b97: Status 404 returned error can't find the container with id 962ed1c22a2df2872c45e5df6e5223119ad3f40b6a85e995a11141f9cebc6b97
Nov 02 15:10:09.310066 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:09.310033    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:10:09.310172 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:09.310105    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:10:09.310172 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:09.310158    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:10:10.081620 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:10.081579    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-25flx" event={"ID":"6f80b62a-49ba-4922-8f3e-804d6bdcc46f","Type":"ContainerStarted","Data":"bb9feef4520f31371456ece67144a6dd30fc5fb819a8f39c85e7bb9b83206635"}
Nov 02 15:10:10.082095 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:10.081649    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-25flx" event={"ID":"6f80b62a-49ba-4922-8f3e-804d6bdcc46f","Type":"ContainerStarted","Data":"962ed1c22a2df2872c45e5df6e5223119ad3f40b6a85e995a11141f9cebc6b97"}
Nov 02 15:10:10.083175 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:10.083147    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-624254/distinct-w3-624254-kserve-5465dc4765-dmq67" event={"ID":"1a5aa910-a6de-417b-8638-81e4ec5c1825","Type":"ContainerStarted","Data":"010a8321841e466f631db5a26ccc650099bcbda947dd780929cd5f1cb1a3c922"}
Nov 02 15:10:10.128497 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:10.128428    5304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-25flx" podStartSLOduration=2.128401708 podStartE2EDuration="2.128401708s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:10:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:10:10.126717997 +0000 UTC m=+810.120542407" watchObservedRunningTime="2026-11-02 15:10:10.128401708 +0000 UTC m=+810.122226119"
Nov 02 15:10:10.183933 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:10.183878    5304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-624254/distinct-w3-624254-kserve-5465dc4765-dmq67" podStartSLOduration=2.183862036 podStartE2EDuration="2.183862036s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:10:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:10:10.181960377 +0000 UTC m=+810.175784786" watchObservedRunningTime="2026-11-02 15:10:10.183862036 +0000 UTC m=+810.177686445"
Nov 02 15:10:10.718712 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:10.718675    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-624254/distinct-w3-624254-kserve-5465dc4765-dmq67"
Nov 02 15:10:10.995769 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:10.995680    5304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-25flx"
Nov 02 15:10:10.997053 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:10.997021    5304 patch_prober.go:29] interesting pod/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-25flx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Nov 02 15:10:10.997202 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:10.997081    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-25flx" podUID="6f80b62a-49ba-4922-8f3e-804d6bdcc46f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Nov 02 15:10:11.995834 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:11.995797    5304 patch_prober.go:29] interesting pod/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-25flx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Nov 02 15:10:11.996347 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:11.995869    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-25flx" podUID="6f80b62a-49ba-4922-8f3e-804d6bdcc46f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Nov 02 15:10:12.995814 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:12.995780    5304 patch_prober.go:29] interesting pod/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-25flx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Nov 02 15:10:12.996017 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:12.995851    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-25flx" podUID="6f80b62a-49ba-4922-8f3e-804d6bdcc46f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Nov 02 15:10:13.995717 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:13.995678    5304 patch_prober.go:29] interesting pod/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-25flx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Nov 02 15:10:13.995897 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:13.995742    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-25flx" podUID="6f80b62a-49ba-4922-8f3e-804d6bdcc46f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Nov 02 15:10:14.996137 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:14.996099    5304 patch_prober.go:29] interesting pod/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-25flx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Nov 02 15:10:14.996537 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:14.996193    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-25flx" podUID="6f80b62a-49ba-4922-8f3e-804d6bdcc46f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Nov 02 15:10:15.995979 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:15.995943    5304 patch_prober.go:29] interesting pod/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-25flx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Nov 02 15:10:15.996172 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:15.996011    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-25flx" podUID="6f80b62a-49ba-4922-8f3e-804d6bdcc46f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Nov 02 15:10:17.000801 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:17.000770    5304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-25flx"
Nov 02 15:10:17.001205 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:17.001159    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-25flx"
Nov 02 15:10:17.001989 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:17.001968    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-25flx"
Nov 02 15:10:18.718196 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:18.718156    5304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-624254/distinct-w3-624254-kserve-5465dc4765-dmq67"
Nov 02 15:10:18.731775 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:18.731745    5304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-624254/distinct-w3-624254-kserve-5465dc4765-dmq67"
Nov 02 15:10:19.152355 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:19.152318    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-624254/distinct-w3-624254-kserve-5465dc4765-dmq67"
Nov 02 15:10:23.064223 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:23.064112    5304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-739bf1/distinct-2-w2-739bf1-kserve-5c58479c48-qgfkh"]
Nov 02 15:10:23.069072 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:23.069048    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-739bf1/distinct-2-w2-739bf1-kserve-5c58479c48-qgfkh"
Nov 02 15:10:23.090189 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:23.090153    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-739bf1/distinct-2-w2-739bf1-kserve-5c58479c48-qgfkh"]
Nov 02 15:10:23.093947 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:23.093372    5304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-739bf1\"/\"distinct-2-w2-739bf1-kserve-self-signed-certs\""
Nov 02 15:10:23.136036 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:23.135997    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9746bf8a-91bd-4715-a02e-ae2e651ca54d-dshm\") pod \"distinct-2-w2-739bf1-kserve-5c58479c48-qgfkh\" (UID: \"9746bf8a-91bd-4715-a02e-ae2e651ca54d\") " pod="e2e-models-e2e-worker-w2-739bf1/distinct-2-w2-739bf1-kserve-5c58479c48-qgfkh"
Nov 02 15:10:23.136217 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:23.136062    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9746bf8a-91bd-4715-a02e-ae2e651ca54d-home\") pod \"distinct-2-w2-739bf1-kserve-5c58479c48-qgfkh\" (UID: \"9746bf8a-91bd-4715-a02e-ae2e651ca54d\") " pod="e2e-models-e2e-worker-w2-739bf1/distinct-2-w2-739bf1-kserve-5c58479c48-qgfkh"
Nov 02 15:10:23.136269 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:23.136238    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9746bf8a-91bd-4715-a02e-ae2e651ca54d-model-cache\") pod \"distinct-2-w2-739bf1-kserve-5c58479c48-qgfkh\" (UID: \"9746bf8a-91bd-4715-a02e-ae2e651ca54d\") " pod="e2e-models-e2e-worker-w2-739bf1/distinct-2-w2-739bf1-kserve-5c58479c48-qgfkh"
Nov 02 15:10:23.136311 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:23.136291    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cwwcn\" (UniqueName: \"kubernetes.io/projected/9746bf8a-91bd-4715-a02e-ae2e651ca54d-kube-api-access-cwwcn\") pod \"distinct-2-w2-739bf1-kserve-5c58479c48-qgfkh\" (UID: \"9746bf8a-91bd-4715-a02e-ae2e651ca54d\") " pod="e2e-models-e2e-worker-w2-739bf1/distinct-2-w2-739bf1-kserve-5c58479c48-qgfkh"
Nov 02 15:10:23.136400 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:23.136384    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9746bf8a-91bd-4715-a02e-ae2e651ca54d-tls-certs\") pod \"distinct-2-w2-739bf1-kserve-5c58479c48-qgfkh\" (UID: \"9746bf8a-91bd-4715-a02e-ae2e651ca54d\") " pod="e2e-models-e2e-worker-w2-739bf1/distinct-2-w2-739bf1-kserve-5c58479c48-qgfkh"
Nov 02 15:10:23.136516 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:23.136499    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9746bf8a-91bd-4715-a02e-ae2e651ca54d-tmp-dir\") pod \"distinct-2-w2-739bf1-kserve-5c58479c48-qgfkh\" (UID: \"9746bf8a-91bd-4715-a02e-ae2e651ca54d\") " pod="e2e-models-e2e-worker-w2-739bf1/distinct-2-w2-739bf1-kserve-5c58479c48-qgfkh"
Nov 02 15:10:23.237828 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:23.237790    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9746bf8a-91bd-4715-a02e-ae2e651ca54d-home\") pod \"distinct-2-w2-739bf1-kserve-5c58479c48-qgfkh\" (UID: \"9746bf8a-91bd-4715-a02e-ae2e651ca54d\") " pod="e2e-models-e2e-worker-w2-739bf1/distinct-2-w2-739bf1-kserve-5c58479c48-qgfkh"
Nov 02 15:10:23.238158 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:23.238092    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9746bf8a-91bd-4715-a02e-ae2e651ca54d-model-cache\") pod \"distinct-2-w2-739bf1-kserve-5c58479c48-qgfkh\" (UID: \"9746bf8a-91bd-4715-a02e-ae2e651ca54d\") " pod="e2e-models-e2e-worker-w2-739bf1/distinct-2-w2-739bf1-kserve-5c58479c48-qgfkh"
Nov 02 15:10:23.238292 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:23.238159    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cwwcn\" (UniqueName: \"kubernetes.io/projected/9746bf8a-91bd-4715-a02e-ae2e651ca54d-kube-api-access-cwwcn\") pod \"distinct-2-w2-739bf1-kserve-5c58479c48-qgfkh\" (UID: \"9746bf8a-91bd-4715-a02e-ae2e651ca54d\") " pod="e2e-models-e2e-worker-w2-739bf1/distinct-2-w2-739bf1-kserve-5c58479c48-qgfkh"
Nov 02 15:10:23.238292 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:23.238211    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9746bf8a-91bd-4715-a02e-ae2e651ca54d-tls-certs\") pod \"distinct-2-w2-739bf1-kserve-5c58479c48-qgfkh\" (UID: \"9746bf8a-91bd-4715-a02e-ae2e651ca54d\") " pod="e2e-models-e2e-worker-w2-739bf1/distinct-2-w2-739bf1-kserve-5c58479c48-qgfkh"
Nov 02 15:10:23.238412 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:23.238295    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9746bf8a-91bd-4715-a02e-ae2e651ca54d-tmp-dir\") pod \"distinct-2-w2-739bf1-kserve-5c58479c48-qgfkh\" (UID: \"9746bf8a-91bd-4715-a02e-ae2e651ca54d\") " pod="e2e-models-e2e-worker-w2-739bf1/distinct-2-w2-739bf1-kserve-5c58479c48-qgfkh"
Nov 02 15:10:23.238486 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:23.238468    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9746bf8a-91bd-4715-a02e-ae2e651ca54d-dshm\") pod \"distinct-2-w2-739bf1-kserve-5c58479c48-qgfkh\" (UID: \"9746bf8a-91bd-4715-a02e-ae2e651ca54d\") " pod="e2e-models-e2e-worker-w2-739bf1/distinct-2-w2-739bf1-kserve-5c58479c48-qgfkh"
Nov 02 15:10:23.239104 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:23.238699    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9746bf8a-91bd-4715-a02e-ae2e651ca54d-home\") pod \"distinct-2-w2-739bf1-kserve-5c58479c48-qgfkh\" (UID: \"9746bf8a-91bd-4715-a02e-ae2e651ca54d\") " pod="e2e-models-e2e-worker-w2-739bf1/distinct-2-w2-739bf1-kserve-5c58479c48-qgfkh"
Nov 02 15:10:23.242227 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:23.239243    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9746bf8a-91bd-4715-a02e-ae2e651ca54d-model-cache\") pod \"distinct-2-w2-739bf1-kserve-5c58479c48-qgfkh\" (UID: \"9746bf8a-91bd-4715-a02e-ae2e651ca54d\") " pod="e2e-models-e2e-worker-w2-739bf1/distinct-2-w2-739bf1-kserve-5c58479c48-qgfkh"
Nov 02 15:10:23.243461 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:23.243405    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9746bf8a-91bd-4715-a02e-ae2e651ca54d-tmp-dir\") pod \"distinct-2-w2-739bf1-kserve-5c58479c48-qgfkh\" (UID: \"9746bf8a-91bd-4715-a02e-ae2e651ca54d\") " pod="e2e-models-e2e-worker-w2-739bf1/distinct-2-w2-739bf1-kserve-5c58479c48-qgfkh"
Nov 02 15:10:23.243582 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:23.243461    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9746bf8a-91bd-4715-a02e-ae2e651ca54d-dshm\") pod \"distinct-2-w2-739bf1-kserve-5c58479c48-qgfkh\" (UID: \"9746bf8a-91bd-4715-a02e-ae2e651ca54d\") " pod="e2e-models-e2e-worker-w2-739bf1/distinct-2-w2-739bf1-kserve-5c58479c48-qgfkh"
Nov 02 15:10:23.247375 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:23.247348    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9746bf8a-91bd-4715-a02e-ae2e651ca54d-tls-certs\") pod \"distinct-2-w2-739bf1-kserve-5c58479c48-qgfkh\" (UID: \"9746bf8a-91bd-4715-a02e-ae2e651ca54d\") " pod="e2e-models-e2e-worker-w2-739bf1/distinct-2-w2-739bf1-kserve-5c58479c48-qgfkh"
Nov 02 15:10:23.275638 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:23.275521    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cwwcn\" (UniqueName: \"kubernetes.io/projected/9746bf8a-91bd-4715-a02e-ae2e651ca54d-kube-api-access-cwwcn\") pod \"distinct-2-w2-739bf1-kserve-5c58479c48-qgfkh\" (UID: \"9746bf8a-91bd-4715-a02e-ae2e651ca54d\") " pod="e2e-models-e2e-worker-w2-739bf1/distinct-2-w2-739bf1-kserve-5c58479c48-qgfkh"
Nov 02 15:10:23.405160 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:23.405091    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-739bf1/distinct-2-w2-739bf1-kserve-5c58479c48-qgfkh"
Nov 02 15:10:23.706995 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:23.706880    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-739bf1/distinct-2-w2-739bf1-kserve-5c58479c48-qgfkh"]
Nov 02 15:10:23.710010 ip-10-0-135-138 kubenswrapper[5304]: W1102 15:10:23.709979    5304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9746bf8a_91bd_4715_a02e_ae2e651ca54d.slice/crio-3453060ccb0010a2764e10c301fb0adf36b53e281c3dbd7aded2faff5c826399 WatchSource:0}: Error finding container 3453060ccb0010a2764e10c301fb0adf36b53e281c3dbd7aded2faff5c826399: Status 404 returned error can't find the container with id 3453060ccb0010a2764e10c301fb0adf36b53e281c3dbd7aded2faff5c826399
Nov 02 15:10:24.213462 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:24.213426    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-739bf1/distinct-2-w2-739bf1-kserve-5c58479c48-qgfkh" event={"ID":"9746bf8a-91bd-4715-a02e-ae2e651ca54d","Type":"ContainerStarted","Data":"318eb6e36e1c1b0c12dfd74bf206603c50ab68c02996f556680b6f712cde3d5d"}
Nov 02 15:10:24.213925 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:24.213472    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-739bf1/distinct-2-w2-739bf1-kserve-5c58479c48-qgfkh" event={"ID":"9746bf8a-91bd-4715-a02e-ae2e651ca54d","Type":"ContainerStarted","Data":"3453060ccb0010a2764e10c301fb0adf36b53e281c3dbd7aded2faff5c826399"}
Nov 02 15:10:24.259137 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:24.259076    5304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-739bf1/distinct-2-w2-739bf1-kserve-5c58479c48-qgfkh" podStartSLOduration=1.259060946 podStartE2EDuration="1.259060946s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:10:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:10:24.255737737 +0000 UTC m=+824.249562137" watchObservedRunningTime="2026-11-02 15:10:24.259060946 +0000 UTC m=+824.252885354"
Nov 02 15:10:24.405442 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:24.405405    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-739bf1/distinct-2-w2-739bf1-kserve-5c58479c48-qgfkh"
Nov 02 15:10:24.798670 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:10:24.798633    5304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k6fjk" containerName="extractor"
Nov 02 15:10:33.406187 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:33.406151    5304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-739bf1/distinct-2-w2-739bf1-kserve-5c58479c48-qgfkh"
Nov 02 15:10:33.418808 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:33.418782    5304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-739bf1/distinct-2-w2-739bf1-kserve-5c58479c48-qgfkh"
Nov 02 15:10:34.289628 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:34.289596    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-739bf1/distinct-2-w2-739bf1-kserve-5c58479c48-qgfkh"
Nov 02 15:10:38.850477 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:38.850444    5304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-dq45c"]
Nov 02 15:10:38.854761 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:38.854737    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-dq45c"
Nov 02 15:10:38.888201 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:38.888156    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-dq45c"]
Nov 02 15:10:38.892168 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:38.892120    5304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-576tm"]
Nov 02 15:10:38.900339 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:38.900319    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-576tm"
Nov 02 15:10:38.912770 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:38.912746    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-576tm"]
Nov 02 15:10:38.940306 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:38.940270    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fdbb90e5-42c8-4409-95e1-a44ed4abead2-istio-data\") pod \"e2e-worker-w3-624254-openshift-default-7988c9d8b5-576tm\" (UID: \"fdbb90e5-42c8-4409-95e1-a44ed4abead2\") " pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-576tm"
Nov 02 15:10:38.940306 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:38.940310    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fdbb90e5-42c8-4409-95e1-a44ed4abead2-istio-token\") pod \"e2e-worker-w3-624254-openshift-default-7988c9d8b5-576tm\" (UID: \"fdbb90e5-42c8-4409-95e1-a44ed4abead2\") " pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-576tm"
Nov 02 15:10:38.940538 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:38.940334    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbef5be7-9697-4eb4-b8c7-cd985e58c6b6-credential-socket\") pod \"e2e-worker-w3-624254-openshift-default-7988c9d8b5-dq45c\" (UID: \"dbef5be7-9697-4eb4-b8c7-cd985e58c6b6\") " pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-dq45c"
Nov 02 15:10:38.940538 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:38.940445    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dbef5be7-9697-4eb4-b8c7-cd985e58c6b6-istio-envoy\") pod \"e2e-worker-w3-624254-openshift-default-7988c9d8b5-dq45c\" (UID: \"dbef5be7-9697-4eb4-b8c7-cd985e58c6b6\") " pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-dq45c"
Nov 02 15:10:38.940538 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:38.940486    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dbef5be7-9697-4eb4-b8c7-cd985e58c6b6-istio-podinfo\") pod \"e2e-worker-w3-624254-openshift-default-7988c9d8b5-dq45c\" (UID: \"dbef5be7-9697-4eb4-b8c7-cd985e58c6b6\") " pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-dq45c"
Nov 02 15:10:38.940656 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:38.940538    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pmp5b\" (UniqueName: \"kubernetes.io/projected/fdbb90e5-42c8-4409-95e1-a44ed4abead2-kube-api-access-pmp5b\") pod \"e2e-worker-w3-624254-openshift-default-7988c9d8b5-576tm\" (UID: \"fdbb90e5-42c8-4409-95e1-a44ed4abead2\") " pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-576tm"
Nov 02 15:10:38.940656 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:38.940561    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fdbb90e5-42c8-4409-95e1-a44ed4abead2-istio-podinfo\") pod \"e2e-worker-w3-624254-openshift-default-7988c9d8b5-576tm\" (UID: \"fdbb90e5-42c8-4409-95e1-a44ed4abead2\") " pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-576tm"
Nov 02 15:10:38.940656 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:38.940578    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fdbb90e5-42c8-4409-95e1-a44ed4abead2-istio-envoy\") pod \"e2e-worker-w3-624254-openshift-default-7988c9d8b5-576tm\" (UID: \"fdbb90e5-42c8-4409-95e1-a44ed4abead2\") " pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-576tm"
Nov 02 15:10:38.940656 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:38.940632    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dbef5be7-9697-4eb4-b8c7-cd985e58c6b6-istio-data\") pod \"e2e-worker-w3-624254-openshift-default-7988c9d8b5-dq45c\" (UID: \"dbef5be7-9697-4eb4-b8c7-cd985e58c6b6\") " pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-dq45c"
Nov 02 15:10:38.940891 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:38.940676    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fdbb90e5-42c8-4409-95e1-a44ed4abead2-istiod-ca-cert\") pod \"e2e-worker-w3-624254-openshift-default-7988c9d8b5-576tm\" (UID: \"fdbb90e5-42c8-4409-95e1-a44ed4abead2\") " pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-576tm"
Nov 02 15:10:38.940891 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:38.940730    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdbb90e5-42c8-4409-95e1-a44ed4abead2-workload-socket\") pod \"e2e-worker-w3-624254-openshift-default-7988c9d8b5-576tm\" (UID: \"fdbb90e5-42c8-4409-95e1-a44ed4abead2\") " pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-576tm"
Nov 02 15:10:38.940891 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:38.940751    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dbef5be7-9697-4eb4-b8c7-cd985e58c6b6-istiod-ca-cert\") pod \"e2e-worker-w3-624254-openshift-default-7988c9d8b5-dq45c\" (UID: \"dbef5be7-9697-4eb4-b8c7-cd985e58c6b6\") " pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-dq45c"
Nov 02 15:10:38.940891 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:38.940879    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdbb90e5-42c8-4409-95e1-a44ed4abead2-credential-socket\") pod \"e2e-worker-w3-624254-openshift-default-7988c9d8b5-576tm\" (UID: \"fdbb90e5-42c8-4409-95e1-a44ed4abead2\") " pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-576tm"
Nov 02 15:10:38.941097 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:38.940990    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dbef5be7-9697-4eb4-b8c7-cd985e58c6b6-workload-certs\") pod \"e2e-worker-w3-624254-openshift-default-7988c9d8b5-dq45c\" (UID: \"dbef5be7-9697-4eb4-b8c7-cd985e58c6b6\") " pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-dq45c"
Nov 02 15:10:38.941097 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:38.941021    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ls2fk\" (UniqueName: \"kubernetes.io/projected/dbef5be7-9697-4eb4-b8c7-cd985e58c6b6-kube-api-access-ls2fk\") pod \"e2e-worker-w3-624254-openshift-default-7988c9d8b5-dq45c\" (UID: \"dbef5be7-9697-4eb4-b8c7-cd985e58c6b6\") " pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-dq45c"
Nov 02 15:10:38.941097 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:38.941056    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fdbb90e5-42c8-4409-95e1-a44ed4abead2-workload-certs\") pod \"e2e-worker-w3-624254-openshift-default-7988c9d8b5-576tm\" (UID: \"fdbb90e5-42c8-4409-95e1-a44ed4abead2\") " pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-576tm"
Nov 02 15:10:38.941280 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:38.941103    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dbef5be7-9697-4eb4-b8c7-cd985e58c6b6-istio-token\") pod \"e2e-worker-w3-624254-openshift-default-7988c9d8b5-dq45c\" (UID: \"dbef5be7-9697-4eb4-b8c7-cd985e58c6b6\") " pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-dq45c"
Nov 02 15:10:38.941280 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:38.941177    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbef5be7-9697-4eb4-b8c7-cd985e58c6b6-workload-socket\") pod \"e2e-worker-w3-624254-openshift-default-7988c9d8b5-dq45c\" (UID: \"dbef5be7-9697-4eb4-b8c7-cd985e58c6b6\") " pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-dq45c"
Nov 02 15:10:38.966981 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:38.966950    5304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-cmbdw"]
Nov 02 15:10:38.970929 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:38.970911    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-cmbdw"
Nov 02 15:10:38.996855 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:38.996821    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-cmbdw"]
Nov 02 15:10:39.042500 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:39.042462    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fdbb90e5-42c8-4409-95e1-a44ed4abead2-istio-data\") pod \"e2e-worker-w3-624254-openshift-default-7988c9d8b5-576tm\" (UID: \"fdbb90e5-42c8-4409-95e1-a44ed4abead2\") " pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-576tm"
Nov 02 15:10:39.042708 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:39.042517    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fdbb90e5-42c8-4409-95e1-a44ed4abead2-istio-token\") pod \"e2e-worker-w3-624254-openshift-default-7988c9d8b5-576tm\" (UID: \"fdbb90e5-42c8-4409-95e1-a44ed4abead2\") " pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-576tm"
Nov 02 15:10:39.042708 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:39.042557    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbef5be7-9697-4eb4-b8c7-cd985e58c6b6-credential-socket\") pod \"e2e-worker-w3-624254-openshift-default-7988c9d8b5-dq45c\" (UID: \"dbef5be7-9697-4eb4-b8c7-cd985e58c6b6\") " pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-dq45c"
Nov 02 15:10:39.042708 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:39.042589    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dbef5be7-9697-4eb4-b8c7-cd985e58c6b6-istio-envoy\") pod \"e2e-worker-w3-624254-openshift-default-7988c9d8b5-dq45c\" (UID: \"dbef5be7-9697-4eb4-b8c7-cd985e58c6b6\") " pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-dq45c"
Nov 02 15:10:39.042708 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:39.042612    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dbef5be7-9697-4eb4-b8c7-cd985e58c6b6-istio-podinfo\") pod \"e2e-worker-w3-624254-openshift-default-7988c9d8b5-dq45c\" (UID: \"dbef5be7-9697-4eb4-b8c7-cd985e58c6b6\") " pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-dq45c"
Nov 02 15:10:39.042708 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:39.042637    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmp5b\" (UniqueName: \"kubernetes.io/projected/fdbb90e5-42c8-4409-95e1-a44ed4abead2-kube-api-access-pmp5b\") pod \"e2e-worker-w3-624254-openshift-default-7988c9d8b5-576tm\" (UID: \"fdbb90e5-42c8-4409-95e1-a44ed4abead2\") " pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-576tm"
Nov 02 15:10:39.042708 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:39.042666    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fdbb90e5-42c8-4409-95e1-a44ed4abead2-istio-podinfo\") pod \"e2e-worker-w3-624254-openshift-default-7988c9d8b5-576tm\" (UID: \"fdbb90e5-42c8-4409-95e1-a44ed4abead2\") " pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-576tm"
Nov 02 15:10:39.042708 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:39.042691    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fdbb90e5-42c8-4409-95e1-a44ed4abead2-istio-envoy\") pod \"e2e-worker-w3-624254-openshift-default-7988c9d8b5-576tm\" (UID: \"fdbb90e5-42c8-4409-95e1-a44ed4abead2\") " pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-576tm"
Nov 02 15:10:39.042947 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:39.042716    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dbef5be7-9697-4eb4-b8c7-cd985e58c6b6-istio-data\") pod \"e2e-worker-w3-624254-openshift-default-7988c9d8b5-dq45c\" (UID: \"dbef5be7-9697-4eb4-b8c7-cd985e58c6b6\") " pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-dq45c"
Nov 02 15:10:39.042947 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:39.042739    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fdbb90e5-42c8-4409-95e1-a44ed4abead2-istiod-ca-cert\") pod \"e2e-worker-w3-624254-openshift-default-7988c9d8b5-576tm\" (UID: \"fdbb90e5-42c8-4409-95e1-a44ed4abead2\") " pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-576tm"
Nov 02 15:10:39.042947 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:39.042776    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdbb90e5-42c8-4409-95e1-a44ed4abead2-workload-socket\") pod \"e2e-worker-w3-624254-openshift-default-7988c9d8b5-576tm\" (UID: \"fdbb90e5-42c8-4409-95e1-a44ed4abead2\") " pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-576tm"
Nov 02 15:10:39.042947 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:39.042803    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dbef5be7-9697-4eb4-b8c7-cd985e58c6b6-istiod-ca-cert\") pod \"e2e-worker-w3-624254-openshift-default-7988c9d8b5-dq45c\" (UID: \"dbef5be7-9697-4eb4-b8c7-cd985e58c6b6\") " pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-dq45c"
Nov 02 15:10:39.042947 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:39.042842    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d335628-6e81-44d0-809a-e6ccb181679c-istio-token\") pod \"e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-cmbdw\" (UID: \"8d335628-6e81-44d0-809a-e6ccb181679c\") " pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-cmbdw"
Nov 02 15:10:39.042947 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:39.042855    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fdbb90e5-42c8-4409-95e1-a44ed4abead2-istio-data\") pod \"e2e-worker-w3-624254-openshift-default-7988c9d8b5-576tm\" (UID: \"fdbb90e5-42c8-4409-95e1-a44ed4abead2\") " pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-576tm"
Nov 02 15:10:39.042947 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:39.042945    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdbb90e5-42c8-4409-95e1-a44ed4abead2-credential-socket\") pod \"e2e-worker-w3-624254-openshift-default-7988c9d8b5-576tm\" (UID: \"fdbb90e5-42c8-4409-95e1-a44ed4abead2\") " pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-576tm"
Nov 02 15:10:39.043189 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:39.042995    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d335628-6e81-44d0-809a-e6ccb181679c-credential-socket\") pod \"e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-cmbdw\" (UID: \"8d335628-6e81-44d0-809a-e6ccb181679c\") " pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-cmbdw"
Nov 02 15:10:39.043189 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:39.043015    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d335628-6e81-44d0-809a-e6ccb181679c-istio-envoy\") pod \"e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-cmbdw\" (UID: \"8d335628-6e81-44d0-809a-e6ccb181679c\") " pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-cmbdw"
Nov 02 15:10:39.043189 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:39.043042    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d335628-6e81-44d0-809a-e6ccb181679c-istio-podinfo\") pod \"e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-cmbdw\" (UID: \"8d335628-6e81-44d0-809a-e6ccb181679c\") " pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-cmbdw"
Nov 02 15:10:39.043189 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:39.043063    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dbef5be7-9697-4eb4-b8c7-cd985e58c6b6-workload-certs\") pod \"e2e-worker-w3-624254-openshift-default-7988c9d8b5-dq45c\" (UID: \"dbef5be7-9697-4eb4-b8c7-cd985e58c6b6\") " pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-dq45c"
Nov 02 15:10:39.043189 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:39.043078    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ls2fk\" (UniqueName: \"kubernetes.io/projected/dbef5be7-9697-4eb4-b8c7-cd985e58c6b6-kube-api-access-ls2fk\") pod \"e2e-worker-w3-624254-openshift-default-7988c9d8b5-dq45c\" (UID: \"dbef5be7-9697-4eb4-b8c7-cd985e58c6b6\") " pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-dq45c"
Nov 02 15:10:39.043189 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:39.043102    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fdbb90e5-42c8-4409-95e1-a44ed4abead2-workload-certs\") pod \"e2e-worker-w3-624254-openshift-default-7988c9d8b5-576tm\" (UID: \"fdbb90e5-42c8-4409-95e1-a44ed4abead2\") " pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-576tm"
Nov 02 15:10:39.043189 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:39.043116    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d335628-6e81-44d0-809a-e6ccb181679c-istio-data\") pod \"e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-cmbdw\" (UID: \"8d335628-6e81-44d0-809a-e6ccb181679c\") " pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-cmbdw"
Nov 02 15:10:39.043189 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:39.043151    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dbef5be7-9697-4eb4-b8c7-cd985e58c6b6-istio-token\") pod \"e2e-worker-w3-624254-openshift-default-7988c9d8b5-dq45c\" (UID: \"dbef5be7-9697-4eb4-b8c7-cd985e58c6b6\") " pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-dq45c"
Nov 02 15:10:39.043189 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:39.043166    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2fmsn\" (UniqueName: \"kubernetes.io/projected/8d335628-6e81-44d0-809a-e6ccb181679c-kube-api-access-2fmsn\") pod \"e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-cmbdw\" (UID: \"8d335628-6e81-44d0-809a-e6ccb181679c\") " pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-cmbdw"
Nov 02 15:10:39.043189 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:39.043185    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbef5be7-9697-4eb4-b8c7-cd985e58c6b6-workload-socket\") pod \"e2e-worker-w3-624254-openshift-default-7988c9d8b5-dq45c\" (UID: \"dbef5be7-9697-4eb4-b8c7-cd985e58c6b6\") " pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-dq45c"
Nov 02 15:10:39.043572 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:39.043239    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d335628-6e81-44d0-809a-e6ccb181679c-istiod-ca-cert\") pod \"e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-cmbdw\" (UID: \"8d335628-6e81-44d0-809a-e6ccb181679c\") " pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-cmbdw"
Nov 02 15:10:39.043572 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:39.043267    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d335628-6e81-44d0-809a-e6ccb181679c-workload-certs\") pod \"e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-cmbdw\" (UID: \"8d335628-6e81-44d0-809a-e6ccb181679c\") " pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-cmbdw"
Nov 02 15:10:39.043572 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:39.043298    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d335628-6e81-44d0-809a-e6ccb181679c-workload-socket\") pod \"e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-cmbdw\" (UID: \"8d335628-6e81-44d0-809a-e6ccb181679c\") " pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-cmbdw"
Nov 02 15:10:39.043572 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:39.043506    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbef5be7-9697-4eb4-b8c7-cd985e58c6b6-credential-socket\") pod \"e2e-worker-w3-624254-openshift-default-7988c9d8b5-dq45c\" (UID: \"dbef5be7-9697-4eb4-b8c7-cd985e58c6b6\") " pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-dq45c"
Nov 02 15:10:39.043703 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:39.043600    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fdbb90e5-42c8-4409-95e1-a44ed4abead2-workload-certs\") pod \"e2e-worker-w3-624254-openshift-default-7988c9d8b5-576tm\" (UID: \"fdbb90e5-42c8-4409-95e1-a44ed4abead2\") " pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-576tm"
Nov 02 15:10:39.044066 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:39.044003    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbef5be7-9697-4eb4-b8c7-cd985e58c6b6-workload-socket\") pod \"e2e-worker-w3-624254-openshift-default-7988c9d8b5-dq45c\" (UID: \"dbef5be7-9697-4eb4-b8c7-cd985e58c6b6\") " pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-dq45c"
Nov 02 15:10:39.044236 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:39.044219    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dbef5be7-9697-4eb4-b8c7-cd985e58c6b6-istio-data\") pod \"e2e-worker-w3-624254-openshift-default-7988c9d8b5-dq45c\" (UID: \"dbef5be7-9697-4eb4-b8c7-cd985e58c6b6\") " pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-dq45c"
Nov 02 15:10:39.044316 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:39.044278    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdbb90e5-42c8-4409-95e1-a44ed4abead2-credential-socket\") pod \"e2e-worker-w3-624254-openshift-default-7988c9d8b5-576tm\" (UID: \"fdbb90e5-42c8-4409-95e1-a44ed4abead2\") " pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-576tm"
Nov 02 15:10:39.044404 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:39.044380    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dbef5be7-9697-4eb4-b8c7-cd985e58c6b6-istiod-ca-cert\") pod \"e2e-worker-w3-624254-openshift-default-7988c9d8b5-dq45c\" (UID: \"dbef5be7-9697-4eb4-b8c7-cd985e58c6b6\") " pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-dq45c"
Nov 02 15:10:39.044474 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:39.044385    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fdbb90e5-42c8-4409-95e1-a44ed4abead2-istiod-ca-cert\") pod \"e2e-worker-w3-624254-openshift-default-7988c9d8b5-576tm\" (UID: \"fdbb90e5-42c8-4409-95e1-a44ed4abead2\") " pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-576tm"
Nov 02 15:10:39.044612 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:39.044587    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dbef5be7-9697-4eb4-b8c7-cd985e58c6b6-workload-certs\") pod \"e2e-worker-w3-624254-openshift-default-7988c9d8b5-dq45c\" (UID: \"dbef5be7-9697-4eb4-b8c7-cd985e58c6b6\") " pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-dq45c"
Nov 02 15:10:39.044837 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:39.044812    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdbb90e5-42c8-4409-95e1-a44ed4abead2-workload-socket\") pod \"e2e-worker-w3-624254-openshift-default-7988c9d8b5-576tm\" (UID: \"fdbb90e5-42c8-4409-95e1-a44ed4abead2\") " pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-576tm"
Nov 02 15:10:39.046433 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:39.046399    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fdbb90e5-42c8-4409-95e1-a44ed4abead2-istio-envoy\") pod \"e2e-worker-w3-624254-openshift-default-7988c9d8b5-576tm\" (UID: \"fdbb90e5-42c8-4409-95e1-a44ed4abead2\") " pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-576tm"
Nov 02 15:10:39.046513 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:39.046407    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dbef5be7-9697-4eb4-b8c7-cd985e58c6b6-istio-envoy\") pod \"e2e-worker-w3-624254-openshift-default-7988c9d8b5-dq45c\" (UID: \"dbef5be7-9697-4eb4-b8c7-cd985e58c6b6\") " pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-dq45c"
Nov 02 15:10:39.046513 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:39.046453    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fdbb90e5-42c8-4409-95e1-a44ed4abead2-istio-podinfo\") pod \"e2e-worker-w3-624254-openshift-default-7988c9d8b5-576tm\" (UID: \"fdbb90e5-42c8-4409-95e1-a44ed4abead2\") " pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-576tm"
Nov 02 15:10:39.046783 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:39.046765    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dbef5be7-9697-4eb4-b8c7-cd985e58c6b6-istio-podinfo\") pod \"e2e-worker-w3-624254-openshift-default-7988c9d8b5-dq45c\" (UID: \"dbef5be7-9697-4eb4-b8c7-cd985e58c6b6\") " pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-dq45c"
Nov 02 15:10:39.067295 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:39.067267    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dbef5be7-9697-4eb4-b8c7-cd985e58c6b6-istio-token\") pod \"e2e-worker-w3-624254-openshift-default-7988c9d8b5-dq45c\" (UID: \"dbef5be7-9697-4eb4-b8c7-cd985e58c6b6\") " pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-dq45c"
Nov 02 15:10:39.067542 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:39.067523    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fdbb90e5-42c8-4409-95e1-a44ed4abead2-istio-token\") pod \"e2e-worker-w3-624254-openshift-default-7988c9d8b5-576tm\" (UID: \"fdbb90e5-42c8-4409-95e1-a44ed4abead2\") " pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-576tm"
Nov 02 15:10:39.067651 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:39.067629    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pmp5b\" (UniqueName: \"kubernetes.io/projected/fdbb90e5-42c8-4409-95e1-a44ed4abead2-kube-api-access-pmp5b\") pod \"e2e-worker-w3-624254-openshift-default-7988c9d8b5-576tm\" (UID: \"fdbb90e5-42c8-4409-95e1-a44ed4abead2\") " pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-576tm"
Nov 02 15:10:39.067950 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:39.067930    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ls2fk\" (UniqueName: \"kubernetes.io/projected/dbef5be7-9697-4eb4-b8c7-cd985e58c6b6-kube-api-access-ls2fk\") pod \"e2e-worker-w3-624254-openshift-default-7988c9d8b5-dq45c\" (UID: \"dbef5be7-9697-4eb4-b8c7-cd985e58c6b6\") " pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-dq45c"
Nov 02 15:10:39.145657 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:39.144924    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d335628-6e81-44d0-809a-e6ccb181679c-credential-socket\") pod \"e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-cmbdw\" (UID: \"8d335628-6e81-44d0-809a-e6ccb181679c\") " pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-cmbdw"
Nov 02 15:10:39.145657 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:39.144971    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d335628-6e81-44d0-809a-e6ccb181679c-istio-envoy\") pod \"e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-cmbdw\" (UID: \"8d335628-6e81-44d0-809a-e6ccb181679c\") " pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-cmbdw"
Nov 02 15:10:39.145657 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:39.145023    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d335628-6e81-44d0-809a-e6ccb181679c-istio-podinfo\") pod \"e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-cmbdw\" (UID: \"8d335628-6e81-44d0-809a-e6ccb181679c\") " pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-cmbdw"
Nov 02 15:10:39.145657 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:39.145059    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d335628-6e81-44d0-809a-e6ccb181679c-istio-data\") pod \"e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-cmbdw\" (UID: \"8d335628-6e81-44d0-809a-e6ccb181679c\") " pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-cmbdw"
Nov 02 15:10:39.145657 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:39.145092    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2fmsn\" (UniqueName: \"kubernetes.io/projected/8d335628-6e81-44d0-809a-e6ccb181679c-kube-api-access-2fmsn\") pod \"e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-cmbdw\" (UID: \"8d335628-6e81-44d0-809a-e6ccb181679c\") " pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-cmbdw"
Nov 02 15:10:39.145657 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:39.145182    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d335628-6e81-44d0-809a-e6ccb181679c-istiod-ca-cert\") pod \"e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-cmbdw\" (UID: \"8d335628-6e81-44d0-809a-e6ccb181679c\") " pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-cmbdw"
Nov 02 15:10:39.145657 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:39.145222    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d335628-6e81-44d0-809a-e6ccb181679c-workload-certs\") pod \"e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-cmbdw\" (UID: \"8d335628-6e81-44d0-809a-e6ccb181679c\") " pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-cmbdw"
Nov 02 15:10:39.145657 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:39.145265    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d335628-6e81-44d0-809a-e6ccb181679c-workload-socket\") pod \"e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-cmbdw\" (UID: \"8d335628-6e81-44d0-809a-e6ccb181679c\") " pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-cmbdw"
Nov 02 15:10:39.145657 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:39.145360    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d335628-6e81-44d0-809a-e6ccb181679c-credential-socket\") pod \"e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-cmbdw\" (UID: \"8d335628-6e81-44d0-809a-e6ccb181679c\") " pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-cmbdw"
Nov 02 15:10:39.145657 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:39.145445    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d335628-6e81-44d0-809a-e6ccb181679c-istio-token\") pod \"e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-cmbdw\" (UID: \"8d335628-6e81-44d0-809a-e6ccb181679c\") " pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-cmbdw"
Nov 02 15:10:39.146245 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:39.145671    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d335628-6e81-44d0-809a-e6ccb181679c-workload-socket\") pod \"e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-cmbdw\" (UID: \"8d335628-6e81-44d0-809a-e6ccb181679c\") " pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-cmbdw"
Nov 02 15:10:39.146245 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:39.146229    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d335628-6e81-44d0-809a-e6ccb181679c-istio-data\") pod \"e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-cmbdw\" (UID: \"8d335628-6e81-44d0-809a-e6ccb181679c\") " pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-cmbdw"
Nov 02 15:10:39.146824 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:39.146799    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d335628-6e81-44d0-809a-e6ccb181679c-istiod-ca-cert\") pod \"e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-cmbdw\" (UID: \"8d335628-6e81-44d0-809a-e6ccb181679c\") " pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-cmbdw"
Nov 02 15:10:39.147364 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:39.147343    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d335628-6e81-44d0-809a-e6ccb181679c-workload-certs\") pod \"e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-cmbdw\" (UID: \"8d335628-6e81-44d0-809a-e6ccb181679c\") " pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-cmbdw"
Nov 02 15:10:39.151418 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:39.151392    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d335628-6e81-44d0-809a-e6ccb181679c-istio-envoy\") pod \"e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-cmbdw\" (UID: \"8d335628-6e81-44d0-809a-e6ccb181679c\") " pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-cmbdw"
Nov 02 15:10:39.151643 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:39.151608    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d335628-6e81-44d0-809a-e6ccb181679c-istio-podinfo\") pod \"e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-cmbdw\" (UID: \"8d335628-6e81-44d0-809a-e6ccb181679c\") " pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-cmbdw"
Nov 02 15:10:39.170507 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:39.170483    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-dq45c"
Nov 02 15:10:39.170795 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:39.170775    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d335628-6e81-44d0-809a-e6ccb181679c-istio-token\") pod \"e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-cmbdw\" (UID: \"8d335628-6e81-44d0-809a-e6ccb181679c\") " pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-cmbdw"
Nov 02 15:10:39.170946 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:39.170924    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2fmsn\" (UniqueName: \"kubernetes.io/projected/8d335628-6e81-44d0-809a-e6ccb181679c-kube-api-access-2fmsn\") pod \"e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-cmbdw\" (UID: \"8d335628-6e81-44d0-809a-e6ccb181679c\") " pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-cmbdw"
Nov 02 15:10:39.218832 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:39.218794    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-576tm"
Nov 02 15:10:39.288096 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:39.287793    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-cmbdw"
Nov 02 15:10:39.504814 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:39.504772    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-dq45c"]
Nov 02 15:10:39.507248 ip-10-0-135-138 kubenswrapper[5304]: W1102 15:10:39.507219    5304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbef5be7_9697_4eb4_b8c7_cd985e58c6b6.slice/crio-0ecca7a206c56feb60c4a0b885cab95071d704242f28c977ed855da4a1105712 WatchSource:0}: Error finding container 0ecca7a206c56feb60c4a0b885cab95071d704242f28c977ed855da4a1105712: Status 404 returned error can't find the container with id 0ecca7a206c56feb60c4a0b885cab95071d704242f28c977ed855da4a1105712
Nov 02 15:10:39.509772 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:39.509737    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:10:39.509847 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:39.509805    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:10:39.509887 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:39.509851    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:10:39.553891 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:39.553867    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-576tm"]
Nov 02 15:10:39.569476 ip-10-0-135-138 kubenswrapper[5304]: W1102 15:10:39.569445    5304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfdbb90e5_42c8_4409_95e1_a44ed4abead2.slice/crio-bbf768ecf1d14b4a2bb82b2594d9d532b33f3d3b87d550d8090a74a0e0aa14f7 WatchSource:0}: Error finding container bbf768ecf1d14b4a2bb82b2594d9d532b33f3d3b87d550d8090a74a0e0aa14f7: Status 404 returned error can't find the container with id bbf768ecf1d14b4a2bb82b2594d9d532b33f3d3b87d550d8090a74a0e0aa14f7
Nov 02 15:10:39.572115 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:39.572077    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:10:39.572264 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:39.572188    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:10:39.572264 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:39.572231    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:10:39.623504 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:39.623477    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-cmbdw"]
Nov 02 15:10:39.632541 ip-10-0-135-138 kubenswrapper[5304]: W1102 15:10:39.632450    5304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d335628_6e81_44d0_809a_e6ccb181679c.slice/crio-7b1e1d9dfb177254163c7d93544adcec57c764fb449b35b6457e49ab4cfa6c85 WatchSource:0}: Error finding container 7b1e1d9dfb177254163c7d93544adcec57c764fb449b35b6457e49ab4cfa6c85: Status 404 returned error can't find the container with id 7b1e1d9dfb177254163c7d93544adcec57c764fb449b35b6457e49ab4cfa6c85
Nov 02 15:10:39.636854 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:39.636461    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:10:39.636854 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:39.636534    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:10:39.636854 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:39.636562    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:10:40.324517 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:40.324477    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-576tm" event={"ID":"fdbb90e5-42c8-4409-95e1-a44ed4abead2","Type":"ContainerStarted","Data":"9204f7292e82a2a289873d5feea1b81d704edcdf3b90e4892e8a5bc5bd64117d"}
Nov 02 15:10:40.324993 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:40.324548    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-576tm" event={"ID":"fdbb90e5-42c8-4409-95e1-a44ed4abead2","Type":"ContainerStarted","Data":"bbf768ecf1d14b4a2bb82b2594d9d532b33f3d3b87d550d8090a74a0e0aa14f7"}
Nov 02 15:10:40.326511 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:40.326479    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-cmbdw" event={"ID":"8d335628-6e81-44d0-809a-e6ccb181679c","Type":"ContainerStarted","Data":"50dd14419792d2806f01a0855390a3119fff13cd513b170c410d8c2e77a557ec"}
Nov 02 15:10:40.326662 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:40.326545    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-cmbdw" event={"ID":"8d335628-6e81-44d0-809a-e6ccb181679c","Type":"ContainerStarted","Data":"7b1e1d9dfb177254163c7d93544adcec57c764fb449b35b6457e49ab4cfa6c85"}
Nov 02 15:10:40.328721 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:40.328702    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-dq45c" event={"ID":"dbef5be7-9697-4eb4-b8c7-cd985e58c6b6","Type":"ContainerStarted","Data":"ac7a2102277c7c786d1251baf6db451b6db5785c9ae1f24d6e55a39c762bb01f"}
Nov 02 15:10:40.328830 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:40.328735    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-dq45c" event={"ID":"dbef5be7-9697-4eb4-b8c7-cd985e58c6b6","Type":"ContainerStarted","Data":"0ecca7a206c56feb60c4a0b885cab95071d704242f28c977ed855da4a1105712"}
Nov 02 15:10:40.378150 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:40.376320    5304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-576tm" podStartSLOduration=2.37629828 podStartE2EDuration="2.37629828s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:10:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:10:40.374341202 +0000 UTC m=+840.368165613" watchObservedRunningTime="2026-11-02 15:10:40.37629828 +0000 UTC m=+840.370122690"
Nov 02 15:10:40.428730 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:40.428674    5304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-cmbdw" podStartSLOduration=2.428659778 podStartE2EDuration="2.428659778s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:10:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:10:40.425492435 +0000 UTC m=+840.419316845" watchObservedRunningTime="2026-11-02 15:10:40.428659778 +0000 UTC m=+840.422484187"
Nov 02 15:10:40.498184 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:40.497795    5304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-dq45c" podStartSLOduration=2.497773902 podStartE2EDuration="2.497773902s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:10:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:10:40.47518073 +0000 UTC m=+840.469005142" watchObservedRunningTime="2026-11-02 15:10:40.497773902 +0000 UTC m=+840.491598312"
Nov 02 15:10:41.179149 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:41.173715    5304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-dq45c"
Nov 02 15:10:41.179149 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:41.177380    5304 patch_prober.go:29] interesting pod/e2e-worker-w3-624254-openshift-default-7988c9d8b5-dq45c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Nov 02 15:10:41.179149 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:41.177437    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-dq45c" podUID="dbef5be7-9697-4eb4-b8c7-cd985e58c6b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Nov 02 15:10:41.225150 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:41.224306    5304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-576tm"
Nov 02 15:10:41.248145 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:41.247563    5304 patch_prober.go:29] interesting pod/e2e-worker-w3-624254-openshift-default-7988c9d8b5-576tm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Nov 02 15:10:41.248145 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:41.247634    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-576tm" podUID="fdbb90e5-42c8-4409-95e1-a44ed4abead2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Nov 02 15:10:41.307153 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:41.304724    5304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-cmbdw"
Nov 02 15:10:41.329148 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:41.320276    5304 patch_prober.go:29] interesting pod/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-cmbdw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Nov 02 15:10:41.329148 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:41.320352    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-cmbdw" podUID="8d335628-6e81-44d0-809a-e6ccb181679c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Nov 02 15:10:42.177154 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:42.175231    5304 patch_prober.go:29] interesting pod/e2e-worker-w3-624254-openshift-default-7988c9d8b5-dq45c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Nov 02 15:10:42.177154 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:42.175313    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-dq45c" podUID="dbef5be7-9697-4eb4-b8c7-cd985e58c6b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Nov 02 15:10:42.219941 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:42.219893    5304 patch_prober.go:29] interesting pod/e2e-worker-w3-624254-openshift-default-7988c9d8b5-576tm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Nov 02 15:10:42.220108 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:42.219964    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-576tm" podUID="fdbb90e5-42c8-4409-95e1-a44ed4abead2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Nov 02 15:10:42.306154 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:42.305583    5304 patch_prober.go:29] interesting pod/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-cmbdw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Nov 02 15:10:42.306154 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:42.305645    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-cmbdw" podUID="8d335628-6e81-44d0-809a-e6ccb181679c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Nov 02 15:10:43.179898 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:43.179842    5304 patch_prober.go:29] interesting pod/e2e-worker-w3-624254-openshift-default-7988c9d8b5-dq45c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Nov 02 15:10:43.180532 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:43.180499    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-dq45c" podUID="dbef5be7-9697-4eb4-b8c7-cd985e58c6b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Nov 02 15:10:43.268148 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:43.267725    5304 patch_prober.go:29] interesting pod/e2e-worker-w3-624254-openshift-default-7988c9d8b5-576tm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Nov 02 15:10:43.268148 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:43.267845    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-576tm" podUID="fdbb90e5-42c8-4409-95e1-a44ed4abead2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Nov 02 15:10:43.328507 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:43.328461    5304 patch_prober.go:29] interesting pod/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-cmbdw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Nov 02 15:10:43.328690 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:43.328532    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-cmbdw" podUID="8d335628-6e81-44d0-809a-e6ccb181679c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Nov 02 15:10:44.179152 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:44.171545    5304 patch_prober.go:29] interesting pod/e2e-worker-w3-624254-openshift-default-7988c9d8b5-dq45c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Nov 02 15:10:44.179152 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:44.171607    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-dq45c" podUID="dbef5be7-9697-4eb4-b8c7-cd985e58c6b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Nov 02 15:10:44.251150 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:44.231303    5304 patch_prober.go:29] interesting pod/e2e-worker-w3-624254-openshift-default-7988c9d8b5-576tm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Nov 02 15:10:44.251150 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:44.231365    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-576tm" podUID="fdbb90e5-42c8-4409-95e1-a44ed4abead2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Nov 02 15:10:44.303150 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:44.288545    5304 patch_prober.go:29] interesting pod/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-cmbdw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Nov 02 15:10:44.303150 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:44.288612    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-cmbdw" podUID="8d335628-6e81-44d0-809a-e6ccb181679c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Nov 02 15:10:45.029153 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:45.028241    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nkbrk" podUID="1f68d837-fff2-418b-84d7-c2d7eb21a57e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.24:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:10:45.029153 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:45.028687    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5dd8e9/simulator-w1-5dd8e9-kserve-6f9f755d6f-gnw5d" podUID="66b75e64-cb24-40be-9076-4984c132e208" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:10:45.046150 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:45.040306    5304 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-5113c2/premium-w6-5113c2-kserve-b84945cb5-pc6f8" podUID="9fb3e992-7bb0-497a-a82c-6e396ba241c1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:10:45.128494 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:45.128255    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-739bf1/simulator-w2-739bf1-kserve-7c74f85465-ltk4t" podUID="cf6a1ded-28a3-49eb-878b-b4ad77a0bb5c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:10:45.173151 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:45.172561    5304 patch_prober.go:29] interesting pod/e2e-worker-w3-624254-openshift-default-7988c9d8b5-dq45c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Nov 02 15:10:45.173151 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:45.172624    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-dq45c" podUID="dbef5be7-9697-4eb4-b8c7-cd985e58c6b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Nov 02 15:10:45.233648 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:45.233594    5304 patch_prober.go:29] interesting pod/e2e-worker-w3-624254-openshift-default-7988c9d8b5-576tm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Nov 02 15:10:45.233941 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:45.233886    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-576tm" podUID="fdbb90e5-42c8-4409-95e1-a44ed4abead2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Nov 02 15:10:45.291351 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:45.291240    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-739bf1/distinct-2-w2-739bf1-kserve-5c58479c48-qgfkh" podUID="9746bf8a-91bd-4715-a02e-ae2e651ca54d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded"
Nov 02 15:10:45.294759 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:45.294728    5304 patch_prober.go:29] interesting pod/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-cmbdw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Nov 02 15:10:45.294945 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:45.294920    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-cmbdw" podUID="8d335628-6e81-44d0-809a-e6ccb181679c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Nov 02 15:10:45.314489 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:45.314442    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5dd8e9/premium-w1-5dd8e9-kserve-67588456fb-j6lph" podUID="4ccc607a-14e4-43ca-8999-78100d4b799f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:10:45.451152 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:45.448762    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6af50f/distinct-w0-6af50f-kserve-c6d994647-hbj9z" podUID="3dca5bf3-8e08-47a0-936e-205670d59ede" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:10:45.461262 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:45.461211    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5113c2/premium-w6-5113c2-kserve-b84945cb5-pc6f8" podUID="9fb3e992-7bb0-497a-a82c-6e396ba241c1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:10:45.885149 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:45.884273    5304 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-739bf1/simulator-w2-739bf1-kserve-7c74f85465-ltk4t" podUID="cf6a1ded-28a3-49eb-878b-b4ad77a0bb5c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:10:46.074151 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:46.033226    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5dd8e9/simulator-w1-5dd8e9-kserve-6f9f755d6f-gnw5d" podUID="66b75e64-cb24-40be-9076-4984c132e208" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:10:46.074151 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:46.033687    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nkbrk" podUID="1f68d837-fff2-418b-84d7-c2d7eb21a57e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.24:8000/ready\": context deadline exceeded"
Nov 02 15:10:46.074151 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:46.035096    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5dd8e9/simulator-w1-5dd8e9-kserve-6f9f755d6f-gnw5d"
Nov 02 15:10:46.074151 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:46.035214    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nkbrk"
Nov 02 15:10:46.074151 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:46.035267    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-624254/simulator-w3-624254-kserve-56866b7699-z47sd" podUID="345c710f-b414-4e62-86c7-d51ff4e5be15" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": context deadline exceeded"
Nov 02 15:10:46.099152 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:46.086468    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7ecfe1/distinct-w4-7ecfe1-kserve-6c847c647d-d2jvv" podUID="f1f1ca5e-cca7-4c54-939f-06f095defcdd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:10:46.143303 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:46.143207    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-624254/distinct-w3-624254-kserve-5465dc4765-dmq67" podUID="1a5aa910-a6de-417b-8638-81e4ec5c1825" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:10:46.171157 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:46.153844    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-739bf1/simulator-w2-739bf1-kserve-7c74f85465-ltk4t" podUID="cf6a1ded-28a3-49eb-878b-b4ad77a0bb5c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:10:46.201160 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:46.173319    5304 patch_prober.go:29] interesting pod/e2e-worker-w3-624254-openshift-default-7988c9d8b5-dq45c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Nov 02 15:10:46.201503 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:46.201462    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-dq45c" podUID="dbef5be7-9697-4eb4-b8c7-cd985e58c6b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Nov 02 15:10:46.239700 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:46.239661    5304 patch_prober.go:29] interesting pod/e2e-worker-w3-624254-openshift-default-7988c9d8b5-576tm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Nov 02 15:10:46.239973 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:46.239941    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-576tm" podUID="fdbb90e5-42c8-4409-95e1-a44ed4abead2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Nov 02 15:10:46.244562 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:46.244522    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1cc1b2/premium-w5-1cc1b2-kserve-58db46f97c-cgdqg" podUID="210f620a-8ae7-4e4c-acc3-46f75ddc6105" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:10:46.335598 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:46.335561    5304 patch_prober.go:29] interesting pod/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-cmbdw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Nov 02 15:10:46.421252 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:46.405289    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-cmbdw" podUID="8d335628-6e81-44d0-809a-e6ccb181679c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Nov 02 15:10:46.421252 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:46.375012    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5dd8e9/premium-w1-5dd8e9-kserve-67588456fb-j6lph" podUID="4ccc607a-14e4-43ca-8999-78100d4b799f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded"
Nov 02 15:10:46.421252 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:46.342838    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-739bf1/distinct-2-w2-739bf1-kserve-5c58479c48-qgfkh" podUID="9746bf8a-91bd-4715-a02e-ae2e651ca54d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:10:46.493311 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:46.493261    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5113c2/premium-w6-5113c2-kserve-b84945cb5-pc6f8" podUID="9fb3e992-7bb0-497a-a82c-6e396ba241c1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:10:46.501333 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:46.501288    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6af50f/distinct-w0-6af50f-kserve-c6d994647-hbj9z" podUID="3dca5bf3-8e08-47a0-936e-205670d59ede" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded"
Nov 02 15:10:46.748211 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:46.715622    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1cc1b2/distinct-w5-1cc1b2-kserve-5b7459bc6d-zl8xr" podUID="211ad15d-9234-4f57-b1b9-181dd2d74208" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded"
Nov 02 15:10:46.912951 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:46.912907    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-739bf1/simulator-w2-739bf1-kserve-7c74f85465-ltk4t"
Nov 02 15:10:46.926318 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:46.926289    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5dd8e9/premium-w1-5dd8e9-kserve-67588456fb-j6lph"
Nov 02 15:10:46.952272 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:46.952237    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-739bf1/distinct-2-w2-739bf1-kserve-5c58479c48-qgfkh"
Nov 02 15:10:46.952568 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:46.952548    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-5113c2/premium-w6-5113c2-kserve-b84945cb5-pc6f8"
Nov 02 15:10:46.981151 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:46.956201    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-6af50f/distinct-w0-6af50f-kserve-c6d994647-hbj9z"
Nov 02 15:10:47.059210 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:47.047788    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-624254/simulator-w3-624254-kserve-56866b7699-z47sd" podUID="345c710f-b414-4e62-86c7-d51ff4e5be15" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:10:47.060943 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:47.049170    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nkbrk" podUID="1f68d837-fff2-418b-84d7-c2d7eb21a57e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.24:8000/ready\": context deadline exceeded"
Nov 02 15:10:47.062118 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:47.049204    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5dd8e9/simulator-w1-5dd8e9-kserve-6f9f755d6f-gnw5d" podUID="66b75e64-cb24-40be-9076-4984c132e208" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": context deadline exceeded"
Nov 02 15:10:47.063240 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:47.062101    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-624254/simulator-w3-624254-kserve-56866b7699-z47sd"
Nov 02 15:10:47.089423 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:47.086741    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7ecfe1/distinct-w4-7ecfe1-kserve-6c847c647d-d2jvv" podUID="f1f1ca5e-cca7-4c54-939f-06f095defcdd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:10:47.092914 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:47.092848    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-7ecfe1/distinct-w4-7ecfe1-kserve-6c847c647d-d2jvv"
Nov 02 15:10:47.149154 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:47.149094    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-624254/distinct-w3-624254-kserve-5465dc4765-dmq67" podUID="1a5aa910-a6de-417b-8638-81e4ec5c1825" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:10:47.154831 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:47.154681    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-739bf1/simulator-w2-739bf1-kserve-7c74f85465-ltk4t" podUID="cf6a1ded-28a3-49eb-878b-b4ad77a0bb5c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:10:47.169315 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:47.163198    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-624254/distinct-w3-624254-kserve-5465dc4765-dmq67"
Nov 02 15:10:47.204339 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:47.204290    5304 patch_prober.go:29] interesting pod/e2e-worker-w3-624254-openshift-default-7988c9d8b5-dq45c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Nov 02 15:10:47.204674 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:47.204620    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-dq45c" podUID="dbef5be7-9697-4eb4-b8c7-cd985e58c6b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Nov 02 15:10:47.266197 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:47.265474    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1cc1b2/premium-w5-1cc1b2-kserve-58db46f97c-cgdqg" podUID="210f620a-8ae7-4e4c-acc3-46f75ddc6105" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:10:47.287797 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:47.285204    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-1cc1b2/premium-w5-1cc1b2-kserve-58db46f97c-cgdqg"
Nov 02 15:10:47.289245 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:47.289118    5304 patch_prober.go:29] interesting pod/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-cmbdw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Nov 02 15:10:47.289245 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:47.289188    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-cmbdw" podUID="8d335628-6e81-44d0-809a-e6ccb181679c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Nov 02 15:10:47.312895 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:47.298355    5304 patch_prober.go:29] interesting pod/e2e-worker-w3-624254-openshift-default-7988c9d8b5-576tm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Nov 02 15:10:47.312895 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:47.298404    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-576tm" podUID="fdbb90e5-42c8-4409-95e1-a44ed4abead2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Nov 02 15:10:47.433192 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:47.422072    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-739bf1/distinct-2-w2-739bf1-kserve-5c58479c48-qgfkh" podUID="9746bf8a-91bd-4715-a02e-ae2e651ca54d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:10:47.691151 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:47.452169    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5dd8e9/premium-w1-5dd8e9-kserve-67588456fb-j6lph" podUID="4ccc607a-14e4-43ca-8999-78100d4b799f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:10:47.705831 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:47.526804    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6af50f/distinct-w0-6af50f-kserve-c6d994647-hbj9z" podUID="3dca5bf3-8e08-47a0-936e-205670d59ede" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:10:47.736610 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:47.526839    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5113c2/premium-w6-5113c2-kserve-b84945cb5-pc6f8" podUID="9fb3e992-7bb0-497a-a82c-6e396ba241c1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded"
Nov 02 15:10:47.817171 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:47.806225    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1cc1b2/distinct-w5-1cc1b2-kserve-5b7459bc6d-zl8xr" podUID="211ad15d-9234-4f57-b1b9-181dd2d74208" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:10:47.817171 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:47.806661    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-1cc1b2/distinct-w5-1cc1b2-kserve-5b7459bc6d-zl8xr"
Nov 02 15:10:48.061339 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:48.061224    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-624254/simulator-w3-624254-kserve-56866b7699-z47sd" podUID="345c710f-b414-4e62-86c7-d51ff4e5be15" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:10:48.070281 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:48.070237    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5dd8e9/simulator-w1-5dd8e9-kserve-6f9f755d6f-gnw5d" podUID="66b75e64-cb24-40be-9076-4984c132e208" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:10:48.127926 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:48.121932    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7ecfe1/distinct-w4-7ecfe1-kserve-6c847c647d-d2jvv" podUID="f1f1ca5e-cca7-4c54-939f-06f095defcdd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded"
Nov 02 15:10:48.171261 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:48.171209    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-739bf1/simulator-w2-739bf1-kserve-7c74f85465-ltk4t" podUID="cf6a1ded-28a3-49eb-878b-b4ad77a0bb5c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded"
Nov 02 15:10:48.241230 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:48.183911    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-624254/distinct-w3-624254-kserve-5465dc4765-dmq67" podUID="1a5aa910-a6de-417b-8638-81e4ec5c1825" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded"
Nov 02 15:10:48.241917 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:48.223258    5304 patch_prober.go:29] interesting pod/e2e-worker-w3-624254-openshift-default-7988c9d8b5-dq45c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Nov 02 15:10:48.249146 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:48.245609    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-dq45c" podUID="dbef5be7-9697-4eb4-b8c7-cd985e58c6b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Nov 02 15:10:48.249339 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:48.224946    5304 patch_prober.go:29] interesting pod/e2e-worker-w3-624254-openshift-default-7988c9d8b5-576tm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Nov 02 15:10:48.249462 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:48.249437    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-576tm" podUID="fdbb90e5-42c8-4409-95e1-a44ed4abead2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Nov 02 15:10:48.254252 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:48.254219    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nkbrk" podUID="1f68d837-fff2-418b-84d7-c2d7eb21a57e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.24:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:10:48.306151 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:48.306072    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1cc1b2/premium-w5-1cc1b2-kserve-58db46f97c-cgdqg" podUID="210f620a-8ae7-4e4c-acc3-46f75ddc6105" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:10:48.324206 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:48.308340    5304 patch_prober.go:29] interesting pod/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-cmbdw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Nov 02 15:10:48.324206 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:48.308387    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-cmbdw" podUID="8d335628-6e81-44d0-809a-e6ccb181679c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Nov 02 15:10:48.732151 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:48.689237    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-739bf1/distinct-2-w2-739bf1-kserve-5c58479c48-qgfkh" podUID="9746bf8a-91bd-4715-a02e-ae2e651ca54d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:10:48.732151 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:48.690479    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5dd8e9/premium-w1-5dd8e9-kserve-67588456fb-j6lph" podUID="4ccc607a-14e4-43ca-8999-78100d4b799f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded"
Nov 02 15:10:48.733881 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:48.733840    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6af50f/distinct-w0-6af50f-kserve-c6d994647-hbj9z" podUID="3dca5bf3-8e08-47a0-936e-205670d59ede" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:10:48.819374 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:48.819321    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1cc1b2/distinct-w5-1cc1b2-kserve-5b7459bc6d-zl8xr" podUID="211ad15d-9234-4f57-b1b9-181dd2d74208" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:10:48.819955 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:48.783349    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5113c2/premium-w6-5113c2-kserve-b84945cb5-pc6f8" podUID="9fb3e992-7bb0-497a-a82c-6e396ba241c1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:10:49.064341 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:49.064229    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-624254/simulator-w3-624254-kserve-56866b7699-z47sd" podUID="345c710f-b414-4e62-86c7-d51ff4e5be15" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": context deadline exceeded"
Nov 02 15:10:49.142174 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:49.125227    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5dd8e9/simulator-w1-5dd8e9-kserve-6f9f755d6f-gnw5d" podUID="66b75e64-cb24-40be-9076-4984c132e208" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:10:49.142174 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:49.140512    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7ecfe1/distinct-w4-7ecfe1-kserve-6c847c647d-d2jvv" podUID="f1f1ca5e-cca7-4c54-939f-06f095defcdd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded"
Nov 02 15:10:49.142174 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:49.141567    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-624254/simulator-w3-624254-kserve-56866b7699-z47sd"
Nov 02 15:10:49.162290 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:49.162259    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-7ecfe1/distinct-w4-7ecfe1-kserve-6c847c647d-d2jvv"
Nov 02 15:10:49.171494 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:49.171459    5304 patch_prober.go:29] interesting pod/e2e-worker-w3-624254-openshift-default-7988c9d8b5-dq45c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Nov 02 15:10:49.171748 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:49.171722    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-dq45c" podUID="dbef5be7-9697-4eb4-b8c7-cd985e58c6b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Nov 02 15:10:49.252485 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:49.252431    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-624254/distinct-w3-624254-kserve-5465dc4765-dmq67" podUID="1a5aa910-a6de-417b-8638-81e4ec5c1825" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:10:49.252800 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:49.252776    5304 patch_prober.go:29] interesting pod/e2e-worker-w3-624254-openshift-default-7988c9d8b5-576tm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Nov 02 15:10:49.253007 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:49.252980    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-576tm" podUID="fdbb90e5-42c8-4409-95e1-a44ed4abead2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Nov 02 15:10:49.253247 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:49.253222    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-739bf1/simulator-w2-739bf1-kserve-7c74f85465-ltk4t" podUID="cf6a1ded-28a3-49eb-878b-b4ad77a0bb5c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:10:49.288311 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:49.253704    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5dd8e9/simulator-w1-5dd8e9-kserve-6f9f755d6f-gnw5d"
Nov 02 15:10:49.288311 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:49.279358    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nkbrk" podUID="1f68d837-fff2-418b-84d7-c2d7eb21a57e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.24:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:10:49.310276 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:49.310235    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1cc1b2/premium-w5-1cc1b2-kserve-58db46f97c-cgdqg" podUID="210f620a-8ae7-4e4c-acc3-46f75ddc6105" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:10:49.346151 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:49.328763    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-6af50f/distinct-w0-6af50f-kserve-c6d994647-hbj9z"
Nov 02 15:10:49.346151 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:49.334766    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-1cc1b2/distinct-w5-1cc1b2-kserve-5b7459bc6d-zl8xr"
Nov 02 15:10:49.346151 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:49.336310    5304 patch_prober.go:29] interesting pod/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-cmbdw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Nov 02 15:10:49.346151 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:49.336369    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-cmbdw" podUID="8d335628-6e81-44d0-809a-e6ccb181679c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Nov 02 15:10:49.346151 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:49.343739    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-624254/distinct-w3-624254-kserve-5465dc4765-dmq67"
Nov 02 15:10:49.380189 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:49.380153    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-739bf1/distinct-2-w2-739bf1-kserve-5c58479c48-qgfkh"
Nov 02 15:10:49.401632 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:49.401603    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nkbrk"
Nov 02 15:10:49.468686 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:49.468646    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-739bf1/simulator-w2-739bf1-kserve-7c74f85465-ltk4t"
Nov 02 15:10:49.478481 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:49.478452    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-1cc1b2/premium-w5-1cc1b2-kserve-58db46f97c-cgdqg"
Nov 02 15:10:49.481523 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:49.481490    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5dd8e9/premium-w1-5dd8e9-kserve-67588456fb-j6lph"
Nov 02 15:10:49.516624 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:49.516594    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-5113c2/premium-w6-5113c2-kserve-b84945cb5-pc6f8"
Nov 02 15:10:50.288980 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:50.288538    5304 patch_prober.go:29] interesting pod/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-cmbdw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Nov 02 15:10:50.288980 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:50.288598    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-cmbdw" podUID="8d335628-6e81-44d0-809a-e6ccb181679c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Nov 02 15:10:50.644346 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:50.644291    5304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-576tm"
Nov 02 15:10:50.646339 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:50.646298    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-576tm"
Nov 02 15:10:50.647322 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:50.647305    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-576tm"
Nov 02 15:10:50.684207 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:50.681758    5304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-dq45c"
Nov 02 15:10:50.684207 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:50.682585    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-dq45c"
Nov 02 15:10:50.689169 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:50.685813    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-dq45c"
Nov 02 15:10:51.333182 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:51.333150    5304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-cmbdw"
Nov 02 15:10:51.333689 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:51.333571    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-cmbdw"
Nov 02 15:10:51.334525 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:10:51.334500    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-cmbdw"
Nov 02 15:10:54.799102 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:10:54.799056    5304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k6fjk" containerName="extractor"
Nov 02 15:11:03.618902 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:03.618865    5304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-9gbvl"]
Nov 02 15:11:03.622983 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:03.622967    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-9gbvl"
Nov 02 15:11:03.652628 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:03.652595    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a8625f9-7163-4e64-8e52-6537ebbe404c-workload-socket\") pod \"e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-9gbvl\" (UID: \"7a8625f9-7163-4e64-8e52-6537ebbe404c\") " pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-9gbvl"
Nov 02 15:11:03.652786 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:03.652661    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a8625f9-7163-4e64-8e52-6537ebbe404c-istio-envoy\") pod \"e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-9gbvl\" (UID: \"7a8625f9-7163-4e64-8e52-6537ebbe404c\") " pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-9gbvl"
Nov 02 15:11:03.652786 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:03.652742    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a8625f9-7163-4e64-8e52-6537ebbe404c-istio-token\") pod \"e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-9gbvl\" (UID: \"7a8625f9-7163-4e64-8e52-6537ebbe404c\") " pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-9gbvl"
Nov 02 15:11:03.652900 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:03.652791    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a8625f9-7163-4e64-8e52-6537ebbe404c-istiod-ca-cert\") pod \"e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-9gbvl\" (UID: \"7a8625f9-7163-4e64-8e52-6537ebbe404c\") " pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-9gbvl"
Nov 02 15:11:03.652900 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:03.652861    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a8625f9-7163-4e64-8e52-6537ebbe404c-workload-certs\") pod \"e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-9gbvl\" (UID: \"7a8625f9-7163-4e64-8e52-6537ebbe404c\") " pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-9gbvl"
Nov 02 15:11:03.653053 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:03.653031    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a8625f9-7163-4e64-8e52-6537ebbe404c-istio-data\") pod \"e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-9gbvl\" (UID: \"7a8625f9-7163-4e64-8e52-6537ebbe404c\") " pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-9gbvl"
Nov 02 15:11:03.653189 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:03.653063    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x4v96\" (UniqueName: \"kubernetes.io/projected/7a8625f9-7163-4e64-8e52-6537ebbe404c-kube-api-access-x4v96\") pod \"e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-9gbvl\" (UID: \"7a8625f9-7163-4e64-8e52-6537ebbe404c\") " pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-9gbvl"
Nov 02 15:11:03.653286 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:03.653250    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a8625f9-7163-4e64-8e52-6537ebbe404c-istio-podinfo\") pod \"e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-9gbvl\" (UID: \"7a8625f9-7163-4e64-8e52-6537ebbe404c\") " pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-9gbvl"
Nov 02 15:11:03.653337 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:03.653305    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a8625f9-7163-4e64-8e52-6537ebbe404c-credential-socket\") pod \"e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-9gbvl\" (UID: \"7a8625f9-7163-4e64-8e52-6537ebbe404c\") " pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-9gbvl"
Nov 02 15:11:03.661266 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:03.661239    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-9gbvl"]
Nov 02 15:11:03.754824 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:03.754794    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a8625f9-7163-4e64-8e52-6537ebbe404c-istio-data\") pod \"e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-9gbvl\" (UID: \"7a8625f9-7163-4e64-8e52-6537ebbe404c\") " pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-9gbvl"
Nov 02 15:11:03.754824 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:03.754827    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x4v96\" (UniqueName: \"kubernetes.io/projected/7a8625f9-7163-4e64-8e52-6537ebbe404c-kube-api-access-x4v96\") pod \"e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-9gbvl\" (UID: \"7a8625f9-7163-4e64-8e52-6537ebbe404c\") " pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-9gbvl"
Nov 02 15:11:03.755068 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:03.754905    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a8625f9-7163-4e64-8e52-6537ebbe404c-istio-podinfo\") pod \"e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-9gbvl\" (UID: \"7a8625f9-7163-4e64-8e52-6537ebbe404c\") " pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-9gbvl"
Nov 02 15:11:03.755068 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:03.754925    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a8625f9-7163-4e64-8e52-6537ebbe404c-credential-socket\") pod \"e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-9gbvl\" (UID: \"7a8625f9-7163-4e64-8e52-6537ebbe404c\") " pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-9gbvl"
Nov 02 15:11:03.755068 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:03.755019    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a8625f9-7163-4e64-8e52-6537ebbe404c-workload-socket\") pod \"e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-9gbvl\" (UID: \"7a8625f9-7163-4e64-8e52-6537ebbe404c\") " pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-9gbvl"
Nov 02 15:11:03.755068 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:03.755064    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a8625f9-7163-4e64-8e52-6537ebbe404c-istio-envoy\") pod \"e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-9gbvl\" (UID: \"7a8625f9-7163-4e64-8e52-6537ebbe404c\") " pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-9gbvl"
Nov 02 15:11:03.755310 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:03.755107    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a8625f9-7163-4e64-8e52-6537ebbe404c-istio-token\") pod \"e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-9gbvl\" (UID: \"7a8625f9-7163-4e64-8e52-6537ebbe404c\") " pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-9gbvl"
Nov 02 15:11:03.755310 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:03.755157    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a8625f9-7163-4e64-8e52-6537ebbe404c-istiod-ca-cert\") pod \"e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-9gbvl\" (UID: \"7a8625f9-7163-4e64-8e52-6537ebbe404c\") " pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-9gbvl"
Nov 02 15:11:03.755310 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:03.755186    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a8625f9-7163-4e64-8e52-6537ebbe404c-workload-certs\") pod \"e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-9gbvl\" (UID: \"7a8625f9-7163-4e64-8e52-6537ebbe404c\") " pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-9gbvl"
Nov 02 15:11:03.755310 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:03.755254    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a8625f9-7163-4e64-8e52-6537ebbe404c-istio-data\") pod \"e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-9gbvl\" (UID: \"7a8625f9-7163-4e64-8e52-6537ebbe404c\") " pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-9gbvl"
Nov 02 15:11:03.755515 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:03.755353    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a8625f9-7163-4e64-8e52-6537ebbe404c-credential-socket\") pod \"e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-9gbvl\" (UID: \"7a8625f9-7163-4e64-8e52-6537ebbe404c\") " pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-9gbvl"
Nov 02 15:11:03.755612 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:03.755587    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a8625f9-7163-4e64-8e52-6537ebbe404c-workload-socket\") pod \"e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-9gbvl\" (UID: \"7a8625f9-7163-4e64-8e52-6537ebbe404c\") " pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-9gbvl"
Nov 02 15:11:03.755703 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:03.755677    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a8625f9-7163-4e64-8e52-6537ebbe404c-workload-certs\") pod \"e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-9gbvl\" (UID: \"7a8625f9-7163-4e64-8e52-6537ebbe404c\") " pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-9gbvl"
Nov 02 15:11:03.756530 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:03.756504    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a8625f9-7163-4e64-8e52-6537ebbe404c-istiod-ca-cert\") pod \"e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-9gbvl\" (UID: \"7a8625f9-7163-4e64-8e52-6537ebbe404c\") " pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-9gbvl"
Nov 02 15:11:03.757448 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:03.757431    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a8625f9-7163-4e64-8e52-6537ebbe404c-istio-envoy\") pod \"e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-9gbvl\" (UID: \"7a8625f9-7163-4e64-8e52-6537ebbe404c\") " pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-9gbvl"
Nov 02 15:11:03.757565 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:03.757548    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a8625f9-7163-4e64-8e52-6537ebbe404c-istio-podinfo\") pod \"e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-9gbvl\" (UID: \"7a8625f9-7163-4e64-8e52-6537ebbe404c\") " pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-9gbvl"
Nov 02 15:11:03.778938 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:03.778905    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a8625f9-7163-4e64-8e52-6537ebbe404c-istio-token\") pod \"e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-9gbvl\" (UID: \"7a8625f9-7163-4e64-8e52-6537ebbe404c\") " pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-9gbvl"
Nov 02 15:11:03.779190 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:03.779158    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x4v96\" (UniqueName: \"kubernetes.io/projected/7a8625f9-7163-4e64-8e52-6537ebbe404c-kube-api-access-x4v96\") pod \"e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-9gbvl\" (UID: \"7a8625f9-7163-4e64-8e52-6537ebbe404c\") " pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-9gbvl"
Nov 02 15:11:03.937413 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:03.937319    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-9gbvl"
Nov 02 15:11:04.273584 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:04.273554    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-9gbvl"]
Nov 02 15:11:04.276613 ip-10-0-135-138 kubenswrapper[5304]: W1102 15:11:04.276586    5304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a8625f9_7163_4e64_8e52_6537ebbe404c.slice/crio-77359c238e91c6da5374d370a28848fdd8f918e1400c28e44ffbf125c1991887 WatchSource:0}: Error finding container 77359c238e91c6da5374d370a28848fdd8f918e1400c28e44ffbf125c1991887: Status 404 returned error can't find the container with id 77359c238e91c6da5374d370a28848fdd8f918e1400c28e44ffbf125c1991887
Nov 02 15:11:04.279343 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:04.279302    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:11:04.279416 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:04.279400    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:11:04.279467 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:04.279450    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:11:04.455718 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:04.455681    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-9gbvl" event={"ID":"7a8625f9-7163-4e64-8e52-6537ebbe404c","Type":"ContainerStarted","Data":"1e6f17353a53455da6b25fef982bf958f8bf3f468acd79fe65343e426bda100c"}
Nov 02 15:11:04.455903 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:04.455745    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-9gbvl" event={"ID":"7a8625f9-7163-4e64-8e52-6537ebbe404c","Type":"ContainerStarted","Data":"77359c238e91c6da5374d370a28848fdd8f918e1400c28e44ffbf125c1991887"}
Nov 02 15:11:04.503732 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:04.503675    5304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-9gbvl" podStartSLOduration=1.50365314 podStartE2EDuration="1.50365314s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:11:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:11:04.50097054 +0000 UTC m=+864.494794961" watchObservedRunningTime="2026-11-02 15:11:04.50365314 +0000 UTC m=+864.497477551"
Nov 02 15:11:04.950208 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:04.943233    5304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-9gbvl"
Nov 02 15:11:05.953150 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:05.946184    5304 patch_prober.go:29] interesting pod/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-9gbvl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Nov 02 15:11:05.953150 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:05.946245    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-9gbvl" podUID="7a8625f9-7163-4e64-8e52-6537ebbe404c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Nov 02 15:11:06.938589 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:06.938551    5304 patch_prober.go:29] interesting pod/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-9gbvl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Nov 02 15:11:06.938866 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:06.938837    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-9gbvl" podUID="7a8625f9-7163-4e64-8e52-6537ebbe404c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Nov 02 15:11:07.079151 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:07.078238    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-624254/simulator-w3-624254-kserve-56866b7699-z47sd" podUID="345c710f-b414-4e62-86c7-d51ff4e5be15" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": context deadline exceeded"
Nov 02 15:11:07.079151 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:07.078647    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5dd8e9/simulator-w1-5dd8e9-kserve-6f9f755d6f-gnw5d" podUID="66b75e64-cb24-40be-9076-4984c132e208" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": context deadline exceeded"
Nov 02 15:11:07.079151 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:07.078914    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nkbrk" podUID="1f68d837-fff2-418b-84d7-c2d7eb21a57e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.24:8000/ready\": context deadline exceeded"
Nov 02 15:11:07.138814 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:07.138772    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7ecfe1/distinct-w4-7ecfe1-kserve-6c847c647d-d2jvv" podUID="f1f1ca5e-cca7-4c54-939f-06f095defcdd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:11:07.251212 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:07.248685    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-624254/distinct-w3-624254-kserve-5465dc4765-dmq67" podUID="1a5aa910-a6de-417b-8638-81e4ec5c1825" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:11:07.326153 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:07.321235    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1cc1b2/premium-w5-1cc1b2-kserve-58db46f97c-cgdqg" podUID="210f620a-8ae7-4e4c-acc3-46f75ddc6105" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:11:07.328485 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:07.328441    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-739bf1/simulator-w2-739bf1-kserve-7c74f85465-ltk4t" podUID="cf6a1ded-28a3-49eb-878b-b4ad77a0bb5c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:11:07.351279 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:07.351221    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1cc1b2/distinct-w5-1cc1b2-kserve-5b7459bc6d-zl8xr" podUID="211ad15d-9234-4f57-b1b9-181dd2d74208" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:11:07.697271 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:07.697219    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5dd8e9/premium-w1-5dd8e9-kserve-67588456fb-j6lph" podUID="4ccc607a-14e4-43ca-8999-78100d4b799f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:11:07.697668 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:07.697537    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-739bf1/distinct-2-w2-739bf1-kserve-5c58479c48-qgfkh" podUID="9746bf8a-91bd-4715-a02e-ae2e651ca54d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded"
Nov 02 15:11:07.711264 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:07.711217    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6af50f/distinct-w0-6af50f-kserve-c6d994647-hbj9z" podUID="3dca5bf3-8e08-47a0-936e-205670d59ede" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:11:07.850178 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:07.836806    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5113c2/premium-w6-5113c2-kserve-b84945cb5-pc6f8" podUID="9fb3e992-7bb0-497a-a82c-6e396ba241c1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:11:07.938947 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:07.938505    5304 patch_prober.go:29] interesting pod/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-9gbvl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Nov 02 15:11:07.938947 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:07.938568    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-9gbvl" podUID="7a8625f9-7163-4e64-8e52-6537ebbe404c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Nov 02 15:11:08.199206 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:08.092349    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nkbrk" podUID="1f68d837-fff2-418b-84d7-c2d7eb21a57e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.24:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:11:08.199206 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:08.092794    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-624254/simulator-w3-624254-kserve-56866b7699-z47sd" podUID="345c710f-b414-4e62-86c7-d51ff4e5be15" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": context deadline exceeded"
Nov 02 15:11:08.199206 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:08.093081    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5dd8e9/simulator-w1-5dd8e9-kserve-6f9f755d6f-gnw5d" podUID="66b75e64-cb24-40be-9076-4984c132e208" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": context deadline exceeded"
Nov 02 15:11:08.239218 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:08.200111    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7ecfe1/distinct-w4-7ecfe1-kserve-6c847c647d-d2jvv" podUID="f1f1ca5e-cca7-4c54-939f-06f095defcdd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:11:08.239218 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:08.200910    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nkbrk"
Nov 02 15:11:08.239218 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:08.200972    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-624254/simulator-w3-624254-kserve-56866b7699-z47sd"
Nov 02 15:11:08.239218 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:08.201038    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5dd8e9/simulator-w1-5dd8e9-kserve-6f9f755d6f-gnw5d"
Nov 02 15:11:08.239218 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:08.201073    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-7ecfe1/distinct-w4-7ecfe1-kserve-6c847c647d-d2jvv"
Nov 02 15:11:08.319332 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:08.295273    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-624254/distinct-w3-624254-kserve-5465dc4765-dmq67" podUID="1a5aa910-a6de-417b-8638-81e4ec5c1825" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:11:08.379646 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:08.347944    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-739bf1/simulator-w2-739bf1-kserve-7c74f85465-ltk4t" podUID="cf6a1ded-28a3-49eb-878b-b4ad77a0bb5c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:11:08.388379 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:08.380926    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1cc1b2/premium-w5-1cc1b2-kserve-58db46f97c-cgdqg" podUID="210f620a-8ae7-4e4c-acc3-46f75ddc6105" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded"
Nov 02 15:11:08.388379 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:08.382576    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1cc1b2/distinct-w5-1cc1b2-kserve-5b7459bc6d-zl8xr" podUID="211ad15d-9234-4f57-b1b9-181dd2d74208" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:11:08.618581 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:08.618541    5304 patch_prober.go:29] interesting pod/e2e-worker-w3-624254-openshift-default-7988c9d8b5-cp5ms container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 15:11:08.622149 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:08.618831    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-cp5ms" podUID="11a65ba1-0a55-4478-b043-637c0a9d6d64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:11:08.695151 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:08.669831    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-624254/distinct-w3-624254-kserve-5465dc4765-dmq67"
Nov 02 15:11:08.792832 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:08.792804    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-739bf1/simulator-w2-739bf1-kserve-7c74f85465-ltk4t"
Nov 02 15:11:08.794010 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:08.793988    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-1cc1b2/premium-w5-1cc1b2-kserve-58db46f97c-cgdqg"
Nov 02 15:11:08.829157 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:08.828189    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-1cc1b2/distinct-w5-1cc1b2-kserve-5b7459bc6d-zl8xr"
Nov 02 15:11:08.829157 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:08.746885    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6af50f/distinct-w0-6af50f-kserve-c6d994647-hbj9z" podUID="3dca5bf3-8e08-47a0-936e-205670d59ede" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded"
Nov 02 15:11:08.864154 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:08.840877    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5113c2/premium-w6-5113c2-kserve-b84945cb5-pc6f8" podUID="9fb3e992-7bb0-497a-a82c-6e396ba241c1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:11:08.864154 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:08.842047    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-6af50f/distinct-w0-6af50f-kserve-c6d994647-hbj9z"
Nov 02 15:11:08.864154 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:08.842111    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-5113c2/premium-w6-5113c2-kserve-b84945cb5-pc6f8"
Nov 02 15:11:08.864154 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:08.732258    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5dd8e9/premium-w1-5dd8e9-kserve-67588456fb-j6lph" podUID="4ccc607a-14e4-43ca-8999-78100d4b799f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:11:08.881205 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:08.867647    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5dd8e9/premium-w1-5dd8e9-kserve-67588456fb-j6lph"
Nov 02 15:11:08.881205 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:08.732297    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-739bf1/distinct-2-w2-739bf1-kserve-5c58479c48-qgfkh" podUID="9746bf8a-91bd-4715-a02e-ae2e651ca54d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded"
Nov 02 15:11:08.881205 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:08.869332    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-739bf1/distinct-2-w2-739bf1-kserve-5c58479c48-qgfkh"
Nov 02 15:11:09.038150 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:09.033256    5304 patch_prober.go:29] interesting pod/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-9gbvl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Nov 02 15:11:09.038150 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:09.033315    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-9gbvl" podUID="7a8625f9-7163-4e64-8e52-6537ebbe404c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Nov 02 15:11:09.113562 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:09.113510    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5dd8e9/simulator-w1-5dd8e9-kserve-6f9f755d6f-gnw5d" podUID="66b75e64-cb24-40be-9076-4984c132e208" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:11:09.118150 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:09.114148    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nkbrk" podUID="1f68d837-fff2-418b-84d7-c2d7eb21a57e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.24:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:11:09.130153 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:09.126226    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-624254/simulator-w3-624254-kserve-56866b7699-z47sd" podUID="345c710f-b414-4e62-86c7-d51ff4e5be15" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:11:09.251209 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:09.224085    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7ecfe1/distinct-w4-7ecfe1-kserve-6c847c647d-d2jvv" podUID="f1f1ca5e-cca7-4c54-939f-06f095defcdd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded"
Nov 02 15:11:09.277307 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:09.277252    5304 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-5dd8e9/simulator-w1-5dd8e9-kserve-6f9f755d6f-gnw5d" podUID="66b75e64-cb24-40be-9076-4984c132e208" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:11:09.282154 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:09.231418    5304 patch_prober.go:29] interesting pod/e2e-worker-w0-6af50f-openshift-default-8545fb68f-nrs9j container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 15:11:09.282154 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:09.277607    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-nrs9j" podUID="c1051c4a-02ca-40b8-b8de-180bd2ca598b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:11:09.297287 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:09.297235    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-624254/distinct-w3-624254-kserve-5465dc4765-dmq67" podUID="1a5aa910-a6de-417b-8638-81e4ec5c1825" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded"
Nov 02 15:11:09.373494 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:09.356638    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-739bf1/simulator-w2-739bf1-kserve-7c74f85465-ltk4t" podUID="cf6a1ded-28a3-49eb-878b-b4ad77a0bb5c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:11:09.382140 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:09.382084    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1cc1b2/premium-w5-1cc1b2-kserve-58db46f97c-cgdqg" podUID="210f620a-8ae7-4e4c-acc3-46f75ddc6105" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:11:09.382810 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:09.382781    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1cc1b2/distinct-w5-1cc1b2-kserve-5b7459bc6d-zl8xr" podUID="211ad15d-9234-4f57-b1b9-181dd2d74208" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:11:10.056450 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:10.056393    5304 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-624254/distinct-w3-624254-kserve-5465dc4765-dmq67" podUID="1a5aa910-a6de-417b-8638-81e4ec5c1825" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/health\": context deadline exceeded"
Nov 02 15:11:10.128155 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:10.126706    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-739bf1/distinct-2-w2-739bf1-kserve-5c58479c48-qgfkh" podUID="9746bf8a-91bd-4715-a02e-ae2e651ca54d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:11:10.128155 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:10.127732    5304 patch_prober.go:29] interesting pod/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-pshqj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 15:11:10.128155 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:10.127792    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-pshqj" podUID="89710086-0e78-4395-b8f9-e47cce4f7dfe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:11:10.179072 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:10.162683    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6af50f/distinct-w0-6af50f-kserve-c6d994647-hbj9z" podUID="3dca5bf3-8e08-47a0-936e-205670d59ede" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded"
Nov 02 15:11:10.416749 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:10.232676    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5dd8e9/premium-w1-5dd8e9-kserve-67588456fb-j6lph" podUID="4ccc607a-14e4-43ca-8999-78100d4b799f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:11:10.469707 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:10.232765    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5113c2/premium-w6-5113c2-kserve-b84945cb5-pc6f8" podUID="9fb3e992-7bb0-497a-a82c-6e396ba241c1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded"
Nov 02 15:11:10.494264 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:10.320755    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7ecfe1/distinct-w4-7ecfe1-kserve-6c847c647d-d2jvv" podUID="f1f1ca5e-cca7-4c54-939f-06f095defcdd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded"
Nov 02 15:11:10.495014 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:10.402105    5304 patch_prober.go:29] interesting pod/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-9gbvl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Nov 02 15:11:10.700219 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:10.700083    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-9gbvl" podUID="7a8625f9-7163-4e64-8e52-6537ebbe404c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Nov 02 15:11:10.700487 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:10.404283    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5dd8e9/simulator-w1-5dd8e9-kserve-6f9f755d6f-gnw5d" podUID="66b75e64-cb24-40be-9076-4984c132e208" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": context deadline exceeded"
Nov 02 15:11:10.701705 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:10.404248    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-624254/simulator-w3-624254-kserve-56866b7699-z47sd" podUID="345c710f-b414-4e62-86c7-d51ff4e5be15" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:11:10.702891 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:10.404318    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nkbrk" podUID="1f68d837-fff2-418b-84d7-c2d7eb21a57e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.24:8000/ready\": context deadline exceeded"
Nov 02 15:11:10.704090 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:10.436243    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1cc1b2/distinct-w5-1cc1b2-kserve-5b7459bc6d-zl8xr" podUID="211ad15d-9234-4f57-b1b9-181dd2d74208" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:11:10.791423 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:10.436331    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1cc1b2/premium-w5-1cc1b2-kserve-58db46f97c-cgdqg" podUID="210f620a-8ae7-4e4c-acc3-46f75ddc6105" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded"
Nov 02 15:11:10.819216 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:10.567452    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-739bf1/simulator-w2-739bf1-kserve-7c74f85465-ltk4t" podUID="cf6a1ded-28a3-49eb-878b-b4ad77a0bb5c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:11:10.847148 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:10.436309    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-624254/distinct-w3-624254-kserve-5465dc4765-dmq67" podUID="1a5aa910-a6de-417b-8638-81e4ec5c1825" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:11:11.010206 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:11.007610    5304 patch_prober.go:29] interesting pod/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-9gbvl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Nov 02 15:11:11.010206 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:11.007669    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-9gbvl" podUID="7a8625f9-7163-4e64-8e52-6537ebbe404c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Nov 02 15:11:11.093907 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:11.093878    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-624254/distinct-w3-624254-kserve-5465dc4765-dmq67"
Nov 02 15:11:11.112013 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:11.111987    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nkbrk"
Nov 02 15:11:11.127038 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:11.127004    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-739bf1/distinct-2-w2-739bf1-kserve-5c58479c48-qgfkh" podUID="9746bf8a-91bd-4715-a02e-ae2e651ca54d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:11:11.156995 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:11.156966    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5dd8e9/premium-w1-5dd8e9-kserve-67588456fb-j6lph"
Nov 02 15:11:11.187003 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:11.186972    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-739bf1/distinct-2-w2-739bf1-kserve-5c58479c48-qgfkh"
Nov 02 15:11:11.193819 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:11.193796    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-5113c2/premium-w6-5113c2-kserve-b84945cb5-pc6f8"
Nov 02 15:11:11.206669 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:11.201080    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-6af50f/distinct-w0-6af50f-kserve-c6d994647-hbj9z"
Nov 02 15:11:11.206669 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:11.201167    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-7ecfe1/distinct-w4-7ecfe1-kserve-6c847c647d-d2jvv"
Nov 02 15:11:11.208765 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:11.208725    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-624254/simulator-w3-624254-kserve-56866b7699-z47sd"
Nov 02 15:11:11.209229 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:11.209206    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-1cc1b2/premium-w5-1cc1b2-kserve-58db46f97c-cgdqg"
Nov 02 15:11:11.286210 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:11.284456    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5dd8e9/simulator-w1-5dd8e9-kserve-6f9f755d6f-gnw5d"
Nov 02 15:11:11.291144 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:11.288108    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-739bf1/simulator-w2-739bf1-kserve-7c74f85465-ltk4t"
Nov 02 15:11:11.292822 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:11.292802    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-1cc1b2/distinct-w5-1cc1b2-kserve-5b7459bc6d-zl8xr"
Nov 02 15:11:11.950150 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:11.943486    5304 patch_prober.go:29] interesting pod/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-9gbvl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Nov 02 15:11:11.950150 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:11.943552    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-9gbvl" podUID="7a8625f9-7163-4e64-8e52-6537ebbe404c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Nov 02 15:11:12.952600 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:12.952561    5304 patch_prober.go:29] interesting pod/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-9gbvl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Nov 02 15:11:12.953257 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:12.953214    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-9gbvl" podUID="7a8625f9-7163-4e64-8e52-6537ebbe404c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Nov 02 15:11:13.938284 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:13.938245    5304 patch_prober.go:29] interesting pod/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-9gbvl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Nov 02 15:11:13.938468 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:13.938300    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-9gbvl" podUID="7a8625f9-7163-4e64-8e52-6537ebbe404c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Nov 02 15:11:15.980307 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:15.980262    5304 patch_prober.go:29] interesting pod/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-9gbvl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 15:11:15.980856 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:15.980336    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-9gbvl" podUID="7a8625f9-7163-4e64-8e52-6537ebbe404c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:11:16.167477 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:16.167442    5304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-9gbvl"
Nov 02 15:11:16.168320 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:16.168302    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-9gbvl"
Nov 02 15:11:16.169182 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:16.169155    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-9gbvl"
Nov 02 15:11:18.662453 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:18.662420    5304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-6gnx4"]
Nov 02 15:11:18.667584 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:18.667566    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-6gnx4"
Nov 02 15:11:18.688213 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:18.688180    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-6gnx4"]
Nov 02 15:11:18.747747 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:18.747698    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2afb5fd0-d309-42eb-bd1e-9092ddeb166a-credential-socket\") pod \"e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-6gnx4\" (UID: \"2afb5fd0-d309-42eb-bd1e-9092ddeb166a\") " pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-6gnx4"
Nov 02 15:11:18.747747 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:18.747748    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2afb5fd0-d309-42eb-bd1e-9092ddeb166a-istiod-ca-cert\") pod \"e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-6gnx4\" (UID: \"2afb5fd0-d309-42eb-bd1e-9092ddeb166a\") " pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-6gnx4"
Nov 02 15:11:18.747986 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:18.747882    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2afb5fd0-d309-42eb-bd1e-9092ddeb166a-istio-podinfo\") pod \"e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-6gnx4\" (UID: \"2afb5fd0-d309-42eb-bd1e-9092ddeb166a\") " pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-6gnx4"
Nov 02 15:11:18.747986 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:18.747946    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2afb5fd0-d309-42eb-bd1e-9092ddeb166a-istio-envoy\") pod \"e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-6gnx4\" (UID: \"2afb5fd0-d309-42eb-bd1e-9092ddeb166a\") " pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-6gnx4"
Nov 02 15:11:18.748103 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:18.748057    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2afb5fd0-d309-42eb-bd1e-9092ddeb166a-workload-certs\") pod \"e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-6gnx4\" (UID: \"2afb5fd0-d309-42eb-bd1e-9092ddeb166a\") " pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-6gnx4"
Nov 02 15:11:18.748184 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:18.748165    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tqcjm\" (UniqueName: \"kubernetes.io/projected/2afb5fd0-d309-42eb-bd1e-9092ddeb166a-kube-api-access-tqcjm\") pod \"e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-6gnx4\" (UID: \"2afb5fd0-d309-42eb-bd1e-9092ddeb166a\") " pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-6gnx4"
Nov 02 15:11:18.748227 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:18.748214    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2afb5fd0-d309-42eb-bd1e-9092ddeb166a-workload-socket\") pod \"e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-6gnx4\" (UID: \"2afb5fd0-d309-42eb-bd1e-9092ddeb166a\") " pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-6gnx4"
Nov 02 15:11:18.748319 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:18.748306    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2afb5fd0-d309-42eb-bd1e-9092ddeb166a-istio-data\") pod \"e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-6gnx4\" (UID: \"2afb5fd0-d309-42eb-bd1e-9092ddeb166a\") " pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-6gnx4"
Nov 02 15:11:18.748377 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:18.748326    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2afb5fd0-d309-42eb-bd1e-9092ddeb166a-istio-token\") pod \"e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-6gnx4\" (UID: \"2afb5fd0-d309-42eb-bd1e-9092ddeb166a\") " pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-6gnx4"
Nov 02 15:11:18.850257 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:18.850217    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tqcjm\" (UniqueName: \"kubernetes.io/projected/2afb5fd0-d309-42eb-bd1e-9092ddeb166a-kube-api-access-tqcjm\") pod \"e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-6gnx4\" (UID: \"2afb5fd0-d309-42eb-bd1e-9092ddeb166a\") " pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-6gnx4"
Nov 02 15:11:18.850453 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:18.850276    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2afb5fd0-d309-42eb-bd1e-9092ddeb166a-workload-socket\") pod \"e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-6gnx4\" (UID: \"2afb5fd0-d309-42eb-bd1e-9092ddeb166a\") " pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-6gnx4"
Nov 02 15:11:18.850453 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:18.850339    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2afb5fd0-d309-42eb-bd1e-9092ddeb166a-istio-data\") pod \"e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-6gnx4\" (UID: \"2afb5fd0-d309-42eb-bd1e-9092ddeb166a\") " pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-6gnx4"
Nov 02 15:11:18.850453 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:18.850355    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2afb5fd0-d309-42eb-bd1e-9092ddeb166a-istio-token\") pod \"e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-6gnx4\" (UID: \"2afb5fd0-d309-42eb-bd1e-9092ddeb166a\") " pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-6gnx4"
Nov 02 15:11:18.850453 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:18.850441    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2afb5fd0-d309-42eb-bd1e-9092ddeb166a-credential-socket\") pod \"e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-6gnx4\" (UID: \"2afb5fd0-d309-42eb-bd1e-9092ddeb166a\") " pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-6gnx4"
Nov 02 15:11:18.850682 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:18.850458    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2afb5fd0-d309-42eb-bd1e-9092ddeb166a-istiod-ca-cert\") pod \"e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-6gnx4\" (UID: \"2afb5fd0-d309-42eb-bd1e-9092ddeb166a\") " pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-6gnx4"
Nov 02 15:11:18.850682 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:18.850494    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2afb5fd0-d309-42eb-bd1e-9092ddeb166a-istio-podinfo\") pod \"e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-6gnx4\" (UID: \"2afb5fd0-d309-42eb-bd1e-9092ddeb166a\") " pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-6gnx4"
Nov 02 15:11:18.850682 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:18.850563    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2afb5fd0-d309-42eb-bd1e-9092ddeb166a-istio-envoy\") pod \"e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-6gnx4\" (UID: \"2afb5fd0-d309-42eb-bd1e-9092ddeb166a\") " pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-6gnx4"
Nov 02 15:11:18.850682 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:18.850627    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2afb5fd0-d309-42eb-bd1e-9092ddeb166a-workload-certs\") pod \"e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-6gnx4\" (UID: \"2afb5fd0-d309-42eb-bd1e-9092ddeb166a\") " pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-6gnx4"
Nov 02 15:11:18.850866 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:18.850786    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2afb5fd0-d309-42eb-bd1e-9092ddeb166a-workload-socket\") pod \"e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-6gnx4\" (UID: \"2afb5fd0-d309-42eb-bd1e-9092ddeb166a\") " pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-6gnx4"
Nov 02 15:11:18.850866 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:18.850842    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2afb5fd0-d309-42eb-bd1e-9092ddeb166a-istio-data\") pod \"e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-6gnx4\" (UID: \"2afb5fd0-d309-42eb-bd1e-9092ddeb166a\") " pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-6gnx4"
Nov 02 15:11:18.850942 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:18.850878    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2afb5fd0-d309-42eb-bd1e-9092ddeb166a-credential-socket\") pod \"e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-6gnx4\" (UID: \"2afb5fd0-d309-42eb-bd1e-9092ddeb166a\") " pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-6gnx4"
Nov 02 15:11:18.851094 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:18.851073    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2afb5fd0-d309-42eb-bd1e-9092ddeb166a-workload-certs\") pod \"e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-6gnx4\" (UID: \"2afb5fd0-d309-42eb-bd1e-9092ddeb166a\") " pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-6gnx4"
Nov 02 15:11:18.851365 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:18.851339    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2afb5fd0-d309-42eb-bd1e-9092ddeb166a-istiod-ca-cert\") pod \"e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-6gnx4\" (UID: \"2afb5fd0-d309-42eb-bd1e-9092ddeb166a\") " pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-6gnx4"
Nov 02 15:11:18.853077 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:18.853056    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2afb5fd0-d309-42eb-bd1e-9092ddeb166a-istio-envoy\") pod \"e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-6gnx4\" (UID: \"2afb5fd0-d309-42eb-bd1e-9092ddeb166a\") " pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-6gnx4"
Nov 02 15:11:18.853203 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:18.853099    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2afb5fd0-d309-42eb-bd1e-9092ddeb166a-istio-podinfo\") pod \"e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-6gnx4\" (UID: \"2afb5fd0-d309-42eb-bd1e-9092ddeb166a\") " pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-6gnx4"
Nov 02 15:11:18.874098 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:18.874067    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2afb5fd0-d309-42eb-bd1e-9092ddeb166a-istio-token\") pod \"e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-6gnx4\" (UID: \"2afb5fd0-d309-42eb-bd1e-9092ddeb166a\") " pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-6gnx4"
Nov 02 15:11:18.874232 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:18.874140    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tqcjm\" (UniqueName: \"kubernetes.io/projected/2afb5fd0-d309-42eb-bd1e-9092ddeb166a-kube-api-access-tqcjm\") pod \"e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-6gnx4\" (UID: \"2afb5fd0-d309-42eb-bd1e-9092ddeb166a\") " pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-6gnx4"
Nov 02 15:11:18.981574 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:18.981472    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-6gnx4"
Nov 02 15:11:19.302335 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:19.302285    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-6gnx4"]
Nov 02 15:11:19.305264 ip-10-0-135-138 kubenswrapper[5304]: W1102 15:11:19.305234    5304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2afb5fd0_d309_42eb_bd1e_9092ddeb166a.slice/crio-71f8bf70c2f9b7bab586387920e077e2eff4cb4c1dc646be4792ed251c1aba2d WatchSource:0}: Error finding container 71f8bf70c2f9b7bab586387920e077e2eff4cb4c1dc646be4792ed251c1aba2d: Status 404 returned error can't find the container with id 71f8bf70c2f9b7bab586387920e077e2eff4cb4c1dc646be4792ed251c1aba2d
Nov 02 15:11:19.307696 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:19.307659    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:11:19.307794 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:19.307766    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:11:19.307858 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:19.307817    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:11:19.422960 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:19.422923    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-6gnx4" event={"ID":"2afb5fd0-d309-42eb-bd1e-9092ddeb166a","Type":"ContainerStarted","Data":"62ba7ae8f5ec5e03bc76221962b6d5947c0efeb6b89e89f81fff181293061504"}
Nov 02 15:11:19.423280 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:19.423248    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-6gnx4" event={"ID":"2afb5fd0-d309-42eb-bd1e-9092ddeb166a","Type":"ContainerStarted","Data":"71f8bf70c2f9b7bab586387920e077e2eff4cb4c1dc646be4792ed251c1aba2d"}
Nov 02 15:11:19.469955 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:19.469904    5304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-6gnx4" podStartSLOduration=1.469889084 podStartE2EDuration="1.469889084s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:11:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:11:19.467351324 +0000 UTC m=+879.461175733" watchObservedRunningTime="2026-11-02 15:11:19.469889084 +0000 UTC m=+879.463713493"
Nov 02 15:11:19.981811 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:19.981772    5304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-6gnx4"
Nov 02 15:11:20.983633 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:20.983593    5304 patch_prober.go:29] interesting pod/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-6gnx4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body=
Nov 02 15:11:20.984047 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:20.983666    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-6gnx4" podUID="2afb5fd0-d309-42eb-bd1e-9092ddeb166a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused"
Nov 02 15:11:21.982565 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:21.982531    5304 patch_prober.go:29] interesting pod/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-6gnx4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body=
Nov 02 15:11:21.982759 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:21.982582    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-6gnx4" podUID="2afb5fd0-d309-42eb-bd1e-9092ddeb166a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused"
Nov 02 15:11:22.981868 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:22.981832    5304 patch_prober.go:29] interesting pod/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-6gnx4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body=
Nov 02 15:11:22.982300 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:22.981888    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-6gnx4" podUID="2afb5fd0-d309-42eb-bd1e-9092ddeb166a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused"
Nov 02 15:11:23.906210 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:23.906170    5304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-9gswz"]
Nov 02 15:11:23.910338 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:23.910316    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-9gswz"
Nov 02 15:11:23.967212 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:23.967179    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-9gswz"]
Nov 02 15:11:23.985583 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:23.985548    5304 patch_prober.go:29] interesting pod/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-6gnx4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body=
Nov 02 15:11:23.985971 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:23.985607    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-6gnx4" podUID="2afb5fd0-d309-42eb-bd1e-9092ddeb166a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused"
Nov 02 15:11:24.026565 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:24.026533    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45f29144-431b-4948-abfa-a1fc5a6847ef-workload-socket\") pod \"e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-9gswz\" (UID: \"45f29144-431b-4948-abfa-a1fc5a6847ef\") " pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-9gswz"
Nov 02 15:11:24.026776 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:24.026596    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45f29144-431b-4948-abfa-a1fc5a6847ef-istio-data\") pod \"e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-9gswz\" (UID: \"45f29144-431b-4948-abfa-a1fc5a6847ef\") " pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-9gswz"
Nov 02 15:11:24.026855 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:24.026765    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45f29144-431b-4948-abfa-a1fc5a6847ef-istio-envoy\") pod \"e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-9gswz\" (UID: \"45f29144-431b-4948-abfa-a1fc5a6847ef\") " pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-9gswz"
Nov 02 15:11:24.026855 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:24.026847    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45f29144-431b-4948-abfa-a1fc5a6847ef-istiod-ca-cert\") pod \"e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-9gswz\" (UID: \"45f29144-431b-4948-abfa-a1fc5a6847ef\") " pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-9gswz"
Nov 02 15:11:24.026957 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:24.026904    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45f29144-431b-4948-abfa-a1fc5a6847ef-workload-certs\") pod \"e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-9gswz\" (UID: \"45f29144-431b-4948-abfa-a1fc5a6847ef\") " pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-9gswz"
Nov 02 15:11:24.027007 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:24.026987    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45f29144-431b-4948-abfa-a1fc5a6847ef-istio-podinfo\") pod \"e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-9gswz\" (UID: \"45f29144-431b-4948-abfa-a1fc5a6847ef\") " pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-9gswz"
Nov 02 15:11:24.027065 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:24.027023    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45f29144-431b-4948-abfa-a1fc5a6847ef-credential-socket\") pod \"e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-9gswz\" (UID: \"45f29144-431b-4948-abfa-a1fc5a6847ef\") " pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-9gswz"
Nov 02 15:11:24.027120 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:24.027066    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lztbm\" (UniqueName: \"kubernetes.io/projected/45f29144-431b-4948-abfa-a1fc5a6847ef-kube-api-access-lztbm\") pod \"e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-9gswz\" (UID: \"45f29144-431b-4948-abfa-a1fc5a6847ef\") " pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-9gswz"
Nov 02 15:11:24.027268 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:24.027249    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45f29144-431b-4948-abfa-a1fc5a6847ef-istio-token\") pod \"e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-9gswz\" (UID: \"45f29144-431b-4948-abfa-a1fc5a6847ef\") " pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-9gswz"
Nov 02 15:11:24.128684 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:24.128651    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45f29144-431b-4948-abfa-a1fc5a6847ef-istio-envoy\") pod \"e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-9gswz\" (UID: \"45f29144-431b-4948-abfa-a1fc5a6847ef\") " pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-9gswz"
Nov 02 15:11:24.128874 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:24.128736    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45f29144-431b-4948-abfa-a1fc5a6847ef-istiod-ca-cert\") pod \"e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-9gswz\" (UID: \"45f29144-431b-4948-abfa-a1fc5a6847ef\") " pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-9gswz"
Nov 02 15:11:24.128874 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:24.128793    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45f29144-431b-4948-abfa-a1fc5a6847ef-workload-certs\") pod \"e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-9gswz\" (UID: \"45f29144-431b-4948-abfa-a1fc5a6847ef\") " pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-9gswz"
Nov 02 15:11:24.128874 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:24.128860    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45f29144-431b-4948-abfa-a1fc5a6847ef-istio-podinfo\") pod \"e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-9gswz\" (UID: \"45f29144-431b-4948-abfa-a1fc5a6847ef\") " pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-9gswz"
Nov 02 15:11:24.129063 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:24.128894    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45f29144-431b-4948-abfa-a1fc5a6847ef-credential-socket\") pod \"e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-9gswz\" (UID: \"45f29144-431b-4948-abfa-a1fc5a6847ef\") " pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-9gswz"
Nov 02 15:11:24.129063 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:24.128934    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lztbm\" (UniqueName: \"kubernetes.io/projected/45f29144-431b-4948-abfa-a1fc5a6847ef-kube-api-access-lztbm\") pod \"e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-9gswz\" (UID: \"45f29144-431b-4948-abfa-a1fc5a6847ef\") " pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-9gswz"
Nov 02 15:11:24.129063 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:24.128996    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45f29144-431b-4948-abfa-a1fc5a6847ef-istio-token\") pod \"e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-9gswz\" (UID: \"45f29144-431b-4948-abfa-a1fc5a6847ef\") " pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-9gswz"
Nov 02 15:11:24.129240 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:24.129073    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45f29144-431b-4948-abfa-a1fc5a6847ef-workload-socket\") pod \"e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-9gswz\" (UID: \"45f29144-431b-4948-abfa-a1fc5a6847ef\") " pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-9gswz"
Nov 02 15:11:24.129240 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:24.129175    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45f29144-431b-4948-abfa-a1fc5a6847ef-istio-data\") pod \"e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-9gswz\" (UID: \"45f29144-431b-4948-abfa-a1fc5a6847ef\") " pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-9gswz"
Nov 02 15:11:24.129240 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:24.129214    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45f29144-431b-4948-abfa-a1fc5a6847ef-workload-certs\") pod \"e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-9gswz\" (UID: \"45f29144-431b-4948-abfa-a1fc5a6847ef\") " pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-9gswz"
Nov 02 15:11:24.129557 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:24.129538    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45f29144-431b-4948-abfa-a1fc5a6847ef-workload-socket\") pod \"e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-9gswz\" (UID: \"45f29144-431b-4948-abfa-a1fc5a6847ef\") " pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-9gswz"
Nov 02 15:11:24.129732 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:24.129565    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45f29144-431b-4948-abfa-a1fc5a6847ef-istiod-ca-cert\") pod \"e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-9gswz\" (UID: \"45f29144-431b-4948-abfa-a1fc5a6847ef\") " pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-9gswz"
Nov 02 15:11:24.129815 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:24.129797    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45f29144-431b-4948-abfa-a1fc5a6847ef-istio-data\") pod \"e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-9gswz\" (UID: \"45f29144-431b-4948-abfa-a1fc5a6847ef\") " pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-9gswz"
Nov 02 15:11:24.129879 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:24.129858    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45f29144-431b-4948-abfa-a1fc5a6847ef-credential-socket\") pod \"e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-9gswz\" (UID: \"45f29144-431b-4948-abfa-a1fc5a6847ef\") " pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-9gswz"
Nov 02 15:11:24.155993 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:24.155790    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45f29144-431b-4948-abfa-a1fc5a6847ef-istio-podinfo\") pod \"e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-9gswz\" (UID: \"45f29144-431b-4948-abfa-a1fc5a6847ef\") " pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-9gswz"
Nov 02 15:11:24.157665 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:24.157430    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45f29144-431b-4948-abfa-a1fc5a6847ef-istio-envoy\") pod \"e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-9gswz\" (UID: \"45f29144-431b-4948-abfa-a1fc5a6847ef\") " pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-9gswz"
Nov 02 15:11:24.165171 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:24.165146    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45f29144-431b-4948-abfa-a1fc5a6847ef-istio-token\") pod \"e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-9gswz\" (UID: \"45f29144-431b-4948-abfa-a1fc5a6847ef\") " pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-9gswz"
Nov 02 15:11:24.166164 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:24.166119    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lztbm\" (UniqueName: \"kubernetes.io/projected/45f29144-431b-4948-abfa-a1fc5a6847ef-kube-api-access-lztbm\") pod \"e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-9gswz\" (UID: \"45f29144-431b-4948-abfa-a1fc5a6847ef\") " pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-9gswz"
Nov 02 15:11:24.228370 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:24.228336    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-9gswz"
Nov 02 15:11:24.428025 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:24.427901    5304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-qqqdn"]
Nov 02 15:11:24.486138 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:24.486087    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-qqqdn"]
Nov 02 15:11:24.487210 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:24.486428    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-qqqdn"
Nov 02 15:11:24.491714 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:24.491688    5304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-64phg"]
Nov 02 15:11:24.505172 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:24.500374    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-64phg"
Nov 02 15:11:24.519547 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:24.519515    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-64phg"]
Nov 02 15:11:24.573064 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:24.573025    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9xr5q\" (UniqueName: \"kubernetes.io/projected/59fee359-49bd-4a5f-b5da-e9e22c9328f1-kube-api-access-9xr5q\") pod \"e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-qqqdn\" (UID: \"59fee359-49bd-4a5f-b5da-e9e22c9328f1\") " pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-qqqdn"
Nov 02 15:11:24.573278 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:24.573078    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59fee359-49bd-4a5f-b5da-e9e22c9328f1-istio-envoy\") pod \"e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-qqqdn\" (UID: \"59fee359-49bd-4a5f-b5da-e9e22c9328f1\") " pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-qqqdn"
Nov 02 15:11:24.573278 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:24.573180    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e60cf723-3d4a-4bcc-bc72-0c984eba466b-workload-certs\") pod \"e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-64phg\" (UID: \"e60cf723-3d4a-4bcc-bc72-0c984eba466b\") " pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-64phg"
Nov 02 15:11:24.573278 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:24.573211    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59fee359-49bd-4a5f-b5da-e9e22c9328f1-workload-certs\") pod \"e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-qqqdn\" (UID: \"59fee359-49bd-4a5f-b5da-e9e22c9328f1\") " pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-qqqdn"
Nov 02 15:11:24.573463 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:24.573281    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e60cf723-3d4a-4bcc-bc72-0c984eba466b-istio-data\") pod \"e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-64phg\" (UID: \"e60cf723-3d4a-4bcc-bc72-0c984eba466b\") " pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-64phg"
Nov 02 15:11:24.573463 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:24.573334    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e60cf723-3d4a-4bcc-bc72-0c984eba466b-credential-socket\") pod \"e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-64phg\" (UID: \"e60cf723-3d4a-4bcc-bc72-0c984eba466b\") " pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-64phg"
Nov 02 15:11:24.573463 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:24.573412    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e60cf723-3d4a-4bcc-bc72-0c984eba466b-istio-envoy\") pod \"e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-64phg\" (UID: \"e60cf723-3d4a-4bcc-bc72-0c984eba466b\") " pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-64phg"
Nov 02 15:11:24.573613 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:24.573462    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e60cf723-3d4a-4bcc-bc72-0c984eba466b-workload-socket\") pod \"e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-64phg\" (UID: \"e60cf723-3d4a-4bcc-bc72-0c984eba466b\") " pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-64phg"
Nov 02 15:11:24.573613 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:24.573565    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59fee359-49bd-4a5f-b5da-e9e22c9328f1-credential-socket\") pod \"e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-qqqdn\" (UID: \"59fee359-49bd-4a5f-b5da-e9e22c9328f1\") " pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-qqqdn"
Nov 02 15:11:24.573613 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:24.573594    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59fee359-49bd-4a5f-b5da-e9e22c9328f1-istio-token\") pod \"e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-qqqdn\" (UID: \"59fee359-49bd-4a5f-b5da-e9e22c9328f1\") " pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-qqqdn"
Nov 02 15:11:24.573767 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:24.573625    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59fee359-49bd-4a5f-b5da-e9e22c9328f1-istio-podinfo\") pod \"e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-qqqdn\" (UID: \"59fee359-49bd-4a5f-b5da-e9e22c9328f1\") " pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-qqqdn"
Nov 02 15:11:24.573767 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:24.573713    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e60cf723-3d4a-4bcc-bc72-0c984eba466b-istio-podinfo\") pod \"e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-64phg\" (UID: \"e60cf723-3d4a-4bcc-bc72-0c984eba466b\") " pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-64phg"
Nov 02 15:11:24.573767 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:24.573755    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4wbfr\" (UniqueName: \"kubernetes.io/projected/e60cf723-3d4a-4bcc-bc72-0c984eba466b-kube-api-access-4wbfr\") pod \"e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-64phg\" (UID: \"e60cf723-3d4a-4bcc-bc72-0c984eba466b\") " pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-64phg"
Nov 02 15:11:24.573913 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:24.573829    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59fee359-49bd-4a5f-b5da-e9e22c9328f1-istiod-ca-cert\") pod \"e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-qqqdn\" (UID: \"59fee359-49bd-4a5f-b5da-e9e22c9328f1\") " pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-qqqdn"
Nov 02 15:11:24.573913 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:24.573846    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e60cf723-3d4a-4bcc-bc72-0c984eba466b-istiod-ca-cert\") pod \"e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-64phg\" (UID: \"e60cf723-3d4a-4bcc-bc72-0c984eba466b\") " pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-64phg"
Nov 02 15:11:24.573913 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:24.573882    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59fee359-49bd-4a5f-b5da-e9e22c9328f1-istio-data\") pod \"e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-qqqdn\" (UID: \"59fee359-49bd-4a5f-b5da-e9e22c9328f1\") " pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-qqqdn"
Nov 02 15:11:24.574070 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:24.573954    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59fee359-49bd-4a5f-b5da-e9e22c9328f1-workload-socket\") pod \"e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-qqqdn\" (UID: \"59fee359-49bd-4a5f-b5da-e9e22c9328f1\") " pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-qqqdn"
Nov 02 15:11:24.574070 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:24.574060    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e60cf723-3d4a-4bcc-bc72-0c984eba466b-istio-token\") pod \"e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-64phg\" (UID: \"e60cf723-3d4a-4bcc-bc72-0c984eba466b\") " pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-64phg"
Nov 02 15:11:24.602112 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:24.602089    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-9gswz"]
Nov 02 15:11:24.606430 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:24.606398    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:11:24.606501 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:24.606466    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:11:24.606544 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:24.606499    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:11:24.675833 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:24.675808    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e60cf723-3d4a-4bcc-bc72-0c984eba466b-istio-envoy\") pod \"e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-64phg\" (UID: \"e60cf723-3d4a-4bcc-bc72-0c984eba466b\") " pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-64phg"
Nov 02 15:11:24.676072 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:24.675859    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e60cf723-3d4a-4bcc-bc72-0c984eba466b-workload-socket\") pod \"e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-64phg\" (UID: \"e60cf723-3d4a-4bcc-bc72-0c984eba466b\") " pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-64phg"
Nov 02 15:11:24.676072 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:24.675914    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59fee359-49bd-4a5f-b5da-e9e22c9328f1-credential-socket\") pod \"e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-qqqdn\" (UID: \"59fee359-49bd-4a5f-b5da-e9e22c9328f1\") " pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-qqqdn"
Nov 02 15:11:24.676072 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:24.675935    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59fee359-49bd-4a5f-b5da-e9e22c9328f1-istio-token\") pod \"e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-qqqdn\" (UID: \"59fee359-49bd-4a5f-b5da-e9e22c9328f1\") " pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-qqqdn"
Nov 02 15:11:24.676072 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:24.675975    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59fee359-49bd-4a5f-b5da-e9e22c9328f1-istio-podinfo\") pod \"e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-qqqdn\" (UID: \"59fee359-49bd-4a5f-b5da-e9e22c9328f1\") " pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-qqqdn"
Nov 02 15:11:24.676072 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:24.676006    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e60cf723-3d4a-4bcc-bc72-0c984eba466b-istio-podinfo\") pod \"e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-64phg\" (UID: \"e60cf723-3d4a-4bcc-bc72-0c984eba466b\") " pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-64phg"
Nov 02 15:11:24.676072 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:24.676064    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4wbfr\" (UniqueName: \"kubernetes.io/projected/e60cf723-3d4a-4bcc-bc72-0c984eba466b-kube-api-access-4wbfr\") pod \"e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-64phg\" (UID: \"e60cf723-3d4a-4bcc-bc72-0c984eba466b\") " pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-64phg"
Nov 02 15:11:24.676288 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:24.676108    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59fee359-49bd-4a5f-b5da-e9e22c9328f1-istiod-ca-cert\") pod \"e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-qqqdn\" (UID: \"59fee359-49bd-4a5f-b5da-e9e22c9328f1\") " pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-qqqdn"
Nov 02 15:11:24.676288 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:24.676146    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e60cf723-3d4a-4bcc-bc72-0c984eba466b-istiod-ca-cert\") pod \"e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-64phg\" (UID: \"e60cf723-3d4a-4bcc-bc72-0c984eba466b\") " pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-64phg"
Nov 02 15:11:24.676288 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:24.676190    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59fee359-49bd-4a5f-b5da-e9e22c9328f1-istio-data\") pod \"e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-qqqdn\" (UID: \"59fee359-49bd-4a5f-b5da-e9e22c9328f1\") " pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-qqqdn"
Nov 02 15:11:24.676288 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:24.676222    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59fee359-49bd-4a5f-b5da-e9e22c9328f1-workload-socket\") pod \"e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-qqqdn\" (UID: \"59fee359-49bd-4a5f-b5da-e9e22c9328f1\") " pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-qqqdn"
Nov 02 15:11:24.676288 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:24.676270    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e60cf723-3d4a-4bcc-bc72-0c984eba466b-istio-token\") pod \"e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-64phg\" (UID: \"e60cf723-3d4a-4bcc-bc72-0c984eba466b\") " pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-64phg"
Nov 02 15:11:24.676489 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:24.676319    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9xr5q\" (UniqueName: \"kubernetes.io/projected/59fee359-49bd-4a5f-b5da-e9e22c9328f1-kube-api-access-9xr5q\") pod \"e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-qqqdn\" (UID: \"59fee359-49bd-4a5f-b5da-e9e22c9328f1\") " pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-qqqdn"
Nov 02 15:11:24.676489 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:24.676366    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59fee359-49bd-4a5f-b5da-e9e22c9328f1-istio-envoy\") pod \"e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-qqqdn\" (UID: \"59fee359-49bd-4a5f-b5da-e9e22c9328f1\") " pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-qqqdn"
Nov 02 15:11:24.676489 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:24.676399    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e60cf723-3d4a-4bcc-bc72-0c984eba466b-workload-certs\") pod \"e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-64phg\" (UID: \"e60cf723-3d4a-4bcc-bc72-0c984eba466b\") " pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-64phg"
Nov 02 15:11:24.676489 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:24.676442    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59fee359-49bd-4a5f-b5da-e9e22c9328f1-workload-certs\") pod \"e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-qqqdn\" (UID: \"59fee359-49bd-4a5f-b5da-e9e22c9328f1\") " pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-qqqdn"
Nov 02 15:11:24.676489 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:24.676459    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e60cf723-3d4a-4bcc-bc72-0c984eba466b-workload-socket\") pod \"e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-64phg\" (UID: \"e60cf723-3d4a-4bcc-bc72-0c984eba466b\") " pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-64phg"
Nov 02 15:11:24.676489 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:24.676477    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e60cf723-3d4a-4bcc-bc72-0c984eba466b-istio-data\") pod \"e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-64phg\" (UID: \"e60cf723-3d4a-4bcc-bc72-0c984eba466b\") " pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-64phg"
Nov 02 15:11:24.676777 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:24.676535    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e60cf723-3d4a-4bcc-bc72-0c984eba466b-credential-socket\") pod \"e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-64phg\" (UID: \"e60cf723-3d4a-4bcc-bc72-0c984eba466b\") " pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-64phg"
Nov 02 15:11:24.676777 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:24.676622    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59fee359-49bd-4a5f-b5da-e9e22c9328f1-credential-socket\") pod \"e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-qqqdn\" (UID: \"59fee359-49bd-4a5f-b5da-e9e22c9328f1\") " pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-qqqdn"
Nov 02 15:11:24.676777 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:24.676723    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e60cf723-3d4a-4bcc-bc72-0c984eba466b-istio-data\") pod \"e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-64phg\" (UID: \"e60cf723-3d4a-4bcc-bc72-0c984eba466b\") " pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-64phg"
Nov 02 15:11:24.677143 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:24.677108    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59fee359-49bd-4a5f-b5da-e9e22c9328f1-istiod-ca-cert\") pod \"e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-qqqdn\" (UID: \"59fee359-49bd-4a5f-b5da-e9e22c9328f1\") " pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-qqqdn"
Nov 02 15:11:24.677417 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:24.677393    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59fee359-49bd-4a5f-b5da-e9e22c9328f1-istio-data\") pod \"e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-qqqdn\" (UID: \"59fee359-49bd-4a5f-b5da-e9e22c9328f1\") " pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-qqqdn"
Nov 02 15:11:24.677649 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:24.677616    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59fee359-49bd-4a5f-b5da-e9e22c9328f1-workload-certs\") pod \"e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-qqqdn\" (UID: \"59fee359-49bd-4a5f-b5da-e9e22c9328f1\") " pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-qqqdn"
Nov 02 15:11:24.677730 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:24.677650    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e60cf723-3d4a-4bcc-bc72-0c984eba466b-workload-certs\") pod \"e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-64phg\" (UID: \"e60cf723-3d4a-4bcc-bc72-0c984eba466b\") " pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-64phg"
Nov 02 15:11:24.677788 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:24.677748    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e60cf723-3d4a-4bcc-bc72-0c984eba466b-istiod-ca-cert\") pod \"e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-64phg\" (UID: \"e60cf723-3d4a-4bcc-bc72-0c984eba466b\") " pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-64phg"
Nov 02 15:11:24.677931 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:24.677907    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59fee359-49bd-4a5f-b5da-e9e22c9328f1-workload-socket\") pod \"e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-qqqdn\" (UID: \"59fee359-49bd-4a5f-b5da-e9e22c9328f1\") " pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-qqqdn"
Nov 02 15:11:24.678083 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:24.678030    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e60cf723-3d4a-4bcc-bc72-0c984eba466b-credential-socket\") pod \"e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-64phg\" (UID: \"e60cf723-3d4a-4bcc-bc72-0c984eba466b\") " pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-64phg"
Nov 02 15:11:24.678760 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:24.678732    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e60cf723-3d4a-4bcc-bc72-0c984eba466b-istio-envoy\") pod \"e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-64phg\" (UID: \"e60cf723-3d4a-4bcc-bc72-0c984eba466b\") " pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-64phg"
Nov 02 15:11:24.680092 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:24.679988    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e60cf723-3d4a-4bcc-bc72-0c984eba466b-istio-podinfo\") pod \"e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-64phg\" (UID: \"e60cf723-3d4a-4bcc-bc72-0c984eba466b\") " pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-64phg"
Nov 02 15:11:24.680206 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:24.680095    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59fee359-49bd-4a5f-b5da-e9e22c9328f1-istio-podinfo\") pod \"e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-qqqdn\" (UID: \"59fee359-49bd-4a5f-b5da-e9e22c9328f1\") " pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-qqqdn"
Nov 02 15:11:24.680283 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:24.680216    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59fee359-49bd-4a5f-b5da-e9e22c9328f1-istio-envoy\") pod \"e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-qqqdn\" (UID: \"59fee359-49bd-4a5f-b5da-e9e22c9328f1\") " pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-qqqdn"
Nov 02 15:11:24.700631 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:24.700572    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9xr5q\" (UniqueName: \"kubernetes.io/projected/59fee359-49bd-4a5f-b5da-e9e22c9328f1-kube-api-access-9xr5q\") pod \"e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-qqqdn\" (UID: \"59fee359-49bd-4a5f-b5da-e9e22c9328f1\") " pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-qqqdn"
Nov 02 15:11:24.700747 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:24.700668    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59fee359-49bd-4a5f-b5da-e9e22c9328f1-istio-token\") pod \"e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-qqqdn\" (UID: \"59fee359-49bd-4a5f-b5da-e9e22c9328f1\") " pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-qqqdn"
Nov 02 15:11:24.701802 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:24.701756    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4wbfr\" (UniqueName: \"kubernetes.io/projected/e60cf723-3d4a-4bcc-bc72-0c984eba466b-kube-api-access-4wbfr\") pod \"e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-64phg\" (UID: \"e60cf723-3d4a-4bcc-bc72-0c984eba466b\") " pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-64phg"
Nov 02 15:11:24.702373 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:24.702334    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e60cf723-3d4a-4bcc-bc72-0c984eba466b-istio-token\") pod \"e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-64phg\" (UID: \"e60cf723-3d4a-4bcc-bc72-0c984eba466b\") " pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-64phg"
Nov 02 15:11:24.820167 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:24.819620    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-qqqdn"
Nov 02 15:11:24.826569 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:11:24.826532    5304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k6fjk" containerName="extractor"
Nov 02 15:11:24.864187 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:24.864142    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-64phg"
Nov 02 15:11:24.987065 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:24.986629    5304 patch_prober.go:29] interesting pod/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-6gnx4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body=
Nov 02 15:11:24.987065 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:24.986686    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-6gnx4" podUID="2afb5fd0-d309-42eb-bd1e-9092ddeb166a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused"
Nov 02 15:11:25.195707 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:25.195677    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-qqqdn"]
Nov 02 15:11:25.195865 ip-10-0-135-138 kubenswrapper[5304]: W1102 15:11:25.195784    5304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59fee359_49bd_4a5f_b5da_e9e22c9328f1.slice/crio-32184248414a7e61c1bffda7ecc38e9e69aac76e3d6bc80d6c7570ea0f8dae8b WatchSource:0}: Error finding container 32184248414a7e61c1bffda7ecc38e9e69aac76e3d6bc80d6c7570ea0f8dae8b: Status 404 returned error can't find the container with id 32184248414a7e61c1bffda7ecc38e9e69aac76e3d6bc80d6c7570ea0f8dae8b
Nov 02 15:11:25.198936 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:25.198901    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:11:25.199030 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:25.198970    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:11:25.199030 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:25.198999    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:11:25.267549 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:25.267402    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-64phg"]
Nov 02 15:11:25.275240 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:25.274923    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:11:25.275240 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:25.274999    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:11:25.275240 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:25.275032    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:11:25.521307 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:25.521217    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-9gswz" event={"ID":"45f29144-431b-4948-abfa-a1fc5a6847ef","Type":"ContainerStarted","Data":"24ded508397143b4d07251767075b3096cfcb6d5ce566922551c62edbdb45520"}
Nov 02 15:11:25.521566 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:25.521541    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-9gswz" event={"ID":"45f29144-431b-4948-abfa-a1fc5a6847ef","Type":"ContainerStarted","Data":"6d6813ca8ce5380ebe01f8933dc97d4a813a676a5124524623f22eda8ff0046f"}
Nov 02 15:11:25.524164 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:25.524102    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-64phg" event={"ID":"e60cf723-3d4a-4bcc-bc72-0c984eba466b","Type":"ContainerStarted","Data":"06e5c446c033df27152ed7a158a49a0d299c958e4afe3e00a3cb047bfdf5315d"}
Nov 02 15:11:25.524280 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:25.524181    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-64phg" event={"ID":"e60cf723-3d4a-4bcc-bc72-0c984eba466b","Type":"ContainerStarted","Data":"5c01915173619b9f33e66de69a407baae4f0718870e919607a2c261d05f71137"}
Nov 02 15:11:25.527002 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:25.526972    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-qqqdn" event={"ID":"59fee359-49bd-4a5f-b5da-e9e22c9328f1","Type":"ContainerStarted","Data":"564c908aa5ce1b803c9b8f326631537260db9011013e1d0d4f96662f86df5638"}
Nov 02 15:11:25.527102 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:25.527022    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-qqqdn" event={"ID":"59fee359-49bd-4a5f-b5da-e9e22c9328f1","Type":"ContainerStarted","Data":"32184248414a7e61c1bffda7ecc38e9e69aac76e3d6bc80d6c7570ea0f8dae8b"}
Nov 02 15:11:25.576011 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:25.575869    5304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-9gswz" podStartSLOduration=2.575849451 podStartE2EDuration="2.575849451s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:11:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:11:25.575676315 +0000 UTC m=+885.569500725" watchObservedRunningTime="2026-11-02 15:11:25.575849451 +0000 UTC m=+885.569673862"
Nov 02 15:11:25.655150 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:25.654727    5304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-64phg" podStartSLOduration=1.6547074670000002 podStartE2EDuration="1.654707467s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:11:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:11:25.629448772 +0000 UTC m=+885.623273184" watchObservedRunningTime="2026-11-02 15:11:25.654707467 +0000 UTC m=+885.648531878"
Nov 02 15:11:25.683657 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:25.683582    5304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-qqqdn" podStartSLOduration=1.683560807 podStartE2EDuration="1.683560807s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:11:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:11:25.676692524 +0000 UTC m=+885.670516935" watchObservedRunningTime="2026-11-02 15:11:25.683560807 +0000 UTC m=+885.677385218"
Nov 02 15:11:25.821327 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:25.821233    5304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-qqqdn"
Nov 02 15:11:25.871895 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:25.868141    5304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-64phg"
Nov 02 15:11:25.985338 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:25.984810    5304 patch_prober.go:29] interesting pod/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-6gnx4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body=
Nov 02 15:11:25.985338 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:25.984878    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-6gnx4" podUID="2afb5fd0-d309-42eb-bd1e-9092ddeb166a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused"
Nov 02 15:11:26.231144 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:26.230883    5304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-9gswz"
Nov 02 15:11:26.232285 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:26.232256    5304 patch_prober.go:29] interesting pod/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-9gswz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Nov 02 15:11:26.232522 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:26.232494    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-9gswz" podUID="45f29144-431b-4948-abfa-a1fc5a6847ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Nov 02 15:11:26.847887 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:26.847853    5304 patch_prober.go:29] interesting pod/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-qqqdn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Nov 02 15:11:26.848222 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:26.848169    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-qqqdn" podUID="59fee359-49bd-4a5f-b5da-e9e22c9328f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Nov 02 15:11:26.880004 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:26.879969    5304 patch_prober.go:29] interesting pod/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-64phg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Nov 02 15:11:26.880276 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:26.880246    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-64phg" podUID="e60cf723-3d4a-4bcc-bc72-0c984eba466b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Nov 02 15:11:27.231367 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:27.231272    5304 patch_prober.go:29] interesting pod/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-9gswz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Nov 02 15:11:27.231942 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:27.231893    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-9gswz" podUID="45f29144-431b-4948-abfa-a1fc5a6847ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Nov 02 15:11:27.270272 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:27.270229    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6af50f/distinct-w0-6af50f-kserve-c6d994647-hbj9z" podUID="3dca5bf3-8e08-47a0-936e-205670d59ede" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:11:27.424322 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:27.424265    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-739bf1/simulator-w2-739bf1-kserve-7c74f85465-ltk4t" podUID="cf6a1ded-28a3-49eb-878b-b4ad77a0bb5c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:11:27.509599 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:27.505269    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5113c2/premium-w6-5113c2-kserve-b84945cb5-pc6f8" podUID="9fb3e992-7bb0-497a-a82c-6e396ba241c1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:11:27.665514 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:27.665478    5304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-6gnx4"
Nov 02 15:11:27.680148 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:27.667249    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-6gnx4"
Nov 02 15:11:27.690154 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:27.686404    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-6gnx4"
Nov 02 15:11:27.837187 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:27.827613    5304 patch_prober.go:29] interesting pod/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-qqqdn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Nov 02 15:11:27.837187 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:27.827674    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-qqqdn" podUID="59fee359-49bd-4a5f-b5da-e9e22c9328f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Nov 02 15:11:28.023482 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:28.023364    5304 patch_prober.go:29] interesting pod/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-64phg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Nov 02 15:11:28.023780 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:28.023735    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-64phg" podUID="e60cf723-3d4a-4bcc-bc72-0c984eba466b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Nov 02 15:11:28.229463 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:28.229429    5304 patch_prober.go:29] interesting pod/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-9gswz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Nov 02 15:11:28.229639 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:28.229488    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-9gswz" podUID="45f29144-431b-4948-abfa-a1fc5a6847ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Nov 02 15:11:29.026153 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:29.025364    5304 patch_prober.go:29] interesting pod/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-64phg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Nov 02 15:11:29.026153 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:29.025425    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-64phg" podUID="e60cf723-3d4a-4bcc-bc72-0c984eba466b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Nov 02 15:11:29.026745 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:29.026529    5304 patch_prober.go:29] interesting pod/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-qqqdn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Nov 02 15:11:29.026745 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:29.026578    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-qqqdn" podUID="59fee359-49bd-4a5f-b5da-e9e22c9328f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Nov 02 15:11:29.229185 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:29.229146    5304 patch_prober.go:29] interesting pod/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-9gswz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Nov 02 15:11:29.229375 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:29.229202    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-9gswz" podUID="45f29144-431b-4948-abfa-a1fc5a6847ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Nov 02 15:11:29.823220 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:29.823187    5304 patch_prober.go:29] interesting pod/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-qqqdn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Nov 02 15:11:29.823496 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:29.823470    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-qqqdn" podUID="59fee359-49bd-4a5f-b5da-e9e22c9328f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Nov 02 15:11:29.918523 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:29.918487    5304 patch_prober.go:29] interesting pod/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-64phg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Nov 02 15:11:29.918778 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:29.918754    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-64phg" podUID="e60cf723-3d4a-4bcc-bc72-0c984eba466b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Nov 02 15:11:30.231533 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:30.231496    5304 patch_prober.go:29] interesting pod/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-9gswz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Nov 02 15:11:30.250490 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:30.250440    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-9gswz" podUID="45f29144-431b-4948-abfa-a1fc5a6847ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Nov 02 15:11:30.820464 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:30.820430    5304 patch_prober.go:29] interesting pod/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-qqqdn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Nov 02 15:11:30.820654 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:30.820480    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-qqqdn" podUID="59fee359-49bd-4a5f-b5da-e9e22c9328f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Nov 02 15:11:30.870361 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:30.870318    5304 patch_prober.go:29] interesting pod/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-64phg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Nov 02 15:11:30.870639 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:30.870590    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-64phg" podUID="e60cf723-3d4a-4bcc-bc72-0c984eba466b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Nov 02 15:11:31.229568 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:31.229535    5304 patch_prober.go:29] interesting pod/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-9gswz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Nov 02 15:11:31.229784 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:31.229616    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-9gswz" podUID="45f29144-431b-4948-abfa-a1fc5a6847ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Nov 02 15:11:31.820727 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:31.820681    5304 patch_prober.go:29] interesting pod/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-qqqdn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Nov 02 15:11:31.820727 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:31.820733    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-qqqdn" podUID="59fee359-49bd-4a5f-b5da-e9e22c9328f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Nov 02 15:11:31.864327 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:31.864278    5304 patch_prober.go:29] interesting pod/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-64phg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Nov 02 15:11:31.864523 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:31.864355    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-64phg" podUID="e60cf723-3d4a-4bcc-bc72-0c984eba466b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Nov 02 15:11:32.672614 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:32.672567    5304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-9gswz"
Nov 02 15:11:32.673474 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:32.673448    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-9gswz"
Nov 02 15:11:32.674735 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:32.674716    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-9gswz"
Nov 02 15:11:32.864263 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:32.864227    5304 patch_prober.go:29] interesting pod/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-64phg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Nov 02 15:11:32.864780 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:32.864286    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-64phg" podUID="e60cf723-3d4a-4bcc-bc72-0c984eba466b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Nov 02 15:11:33.360560 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:33.360528    5304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-qqqdn"
Nov 02 15:11:33.365190 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:33.364920    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-qqqdn"
Nov 02 15:11:33.368763 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:33.368736    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-qqqdn"
Nov 02 15:11:33.868218 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:33.868189    5304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-64phg"
Nov 02 15:11:33.868705 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:33.868539    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-64phg"
Nov 02 15:11:33.869286 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:33.869266    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-64phg"
Nov 02 15:11:42.103153 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:42.101438    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-624254/distinct-w3-624254-kserve-5465dc4765-dmq67" podUID="1a5aa910-a6de-417b-8638-81e4ec5c1825" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:11:42.130230 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:42.125211    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5dd8e9/simulator-w1-5dd8e9-kserve-6f9f755d6f-gnw5d" podUID="66b75e64-cb24-40be-9076-4984c132e208" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": context deadline exceeded"
Nov 02 15:11:42.153153 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:42.139212    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-739bf1/distinct-2-w2-739bf1-kserve-5c58479c48-qgfkh" podUID="9746bf8a-91bd-4715-a02e-ae2e651ca54d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:11:42.187386 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:42.187240    5304 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-624254/simulator-w3-624254-kserve-56866b7699-z47sd" podUID="345c710f-b414-4e62-86c7-d51ff4e5be15" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:11:42.188180 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:42.187666    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5dd8e9/premium-w1-5dd8e9-kserve-67588456fb-j6lph" podUID="4ccc607a-14e4-43ca-8999-78100d4b799f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:11:42.217144 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:42.204321    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6af50f/distinct-w0-6af50f-kserve-c6d994647-hbj9z" podUID="3dca5bf3-8e08-47a0-936e-205670d59ede" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:11:42.304315 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:42.304263    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7ecfe1/distinct-w4-7ecfe1-kserve-6c847c647d-d2jvv" podUID="f1f1ca5e-cca7-4c54-939f-06f095defcdd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:11:42.364946 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:42.364837    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-739bf1/simulator-w2-739bf1-kserve-7c74f85465-ltk4t" podUID="cf6a1ded-28a3-49eb-878b-b4ad77a0bb5c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:11:42.396181 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:42.391542    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1cc1b2/premium-w5-1cc1b2-kserve-58db46f97c-cgdqg" podUID="210f620a-8ae7-4e4c-acc3-46f75ddc6105" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:11:42.481283 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:42.481236    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5113c2/premium-w6-5113c2-kserve-b84945cb5-pc6f8" podUID="9fb3e992-7bb0-497a-a82c-6e396ba241c1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded"
Nov 02 15:11:42.707215 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:42.702296    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-624254/simulator-w3-624254-kserve-56866b7699-z47sd" podUID="345c710f-b414-4e62-86c7-d51ff4e5be15" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:11:42.708633 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:42.707830    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nkbrk" podUID="1f68d837-fff2-418b-84d7-c2d7eb21a57e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.24:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:11:42.767990 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:42.767581    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1cc1b2/distinct-w5-1cc1b2-kserve-5b7459bc6d-zl8xr" podUID="211ad15d-9234-4f57-b1b9-181dd2d74208" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded"
Nov 02 15:11:42.958222 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:42.944320    5304 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-7ecfe1/distinct-w4-7ecfe1-kserve-6c847c647d-d2jvv" podUID="f1f1ca5e-cca7-4c54-939f-06f095defcdd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:11:43.072998 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:43.072944    5304 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nkbrk" podUID="1f68d837-fff2-418b-84d7-c2d7eb21a57e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.24:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:11:43.164163 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:43.156748    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-624254/distinct-w3-624254-kserve-5465dc4765-dmq67" podUID="1a5aa910-a6de-417b-8638-81e4ec5c1825" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:11:43.181335 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:43.181281    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5dd8e9/simulator-w1-5dd8e9-kserve-6f9f755d6f-gnw5d" podUID="66b75e64-cb24-40be-9076-4984c132e208" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": context deadline exceeded"
Nov 02 15:11:43.181703 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:43.181678    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-739bf1/distinct-2-w2-739bf1-kserve-5c58479c48-qgfkh" podUID="9746bf8a-91bd-4715-a02e-ae2e651ca54d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded"
Nov 02 15:11:43.182318 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:43.182300    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-624254/distinct-w3-624254-kserve-5465dc4765-dmq67"
Nov 02 15:11:43.198831 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:43.198804    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5dd8e9/simulator-w1-5dd8e9-kserve-6f9f755d6f-gnw5d"
Nov 02 15:11:43.245687 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:43.217278    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6af50f/distinct-w0-6af50f-kserve-c6d994647-hbj9z" podUID="3dca5bf3-8e08-47a0-936e-205670d59ede" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:11:43.245687 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:43.217781    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-739bf1/distinct-2-w2-739bf1-kserve-5c58479c48-qgfkh"
Nov 02 15:11:43.245687 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:43.220197    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-6af50f/distinct-w0-6af50f-kserve-c6d994647-hbj9z"
Nov 02 15:11:43.321155 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:43.301374    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5dd8e9/premium-w1-5dd8e9-kserve-67588456fb-j6lph" podUID="4ccc607a-14e4-43ca-8999-78100d4b799f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:11:43.321155 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:43.301798    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5dd8e9/premium-w1-5dd8e9-kserve-67588456fb-j6lph"
Nov 02 15:11:43.332282 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:43.321630    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7ecfe1/distinct-w4-7ecfe1-kserve-6c847c647d-d2jvv" podUID="f1f1ca5e-cca7-4c54-939f-06f095defcdd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded"
Nov 02 15:11:43.332282 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:43.322114    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-7ecfe1/distinct-w4-7ecfe1-kserve-6c847c647d-d2jvv"
Nov 02 15:11:43.439221 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:43.416338    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-739bf1/simulator-w2-739bf1-kserve-7c74f85465-ltk4t" podUID="cf6a1ded-28a3-49eb-878b-b4ad77a0bb5c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded"
Nov 02 15:11:43.439221 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:43.416587    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-739bf1/simulator-w2-739bf1-kserve-7c74f85465-ltk4t"
Nov 02 15:11:43.450189 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:43.450149    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1cc1b2/premium-w5-1cc1b2-kserve-58db46f97c-cgdqg" podUID="210f620a-8ae7-4e4c-acc3-46f75ddc6105" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded"
Nov 02 15:11:43.451620 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:43.451595    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-1cc1b2/premium-w5-1cc1b2-kserve-58db46f97c-cgdqg"
Nov 02 15:11:43.494492 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:43.494446    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5113c2/premium-w6-5113c2-kserve-b84945cb5-pc6f8" podUID="9fb3e992-7bb0-497a-a82c-6e396ba241c1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:11:43.501208 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:43.497523    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-5113c2/premium-w6-5113c2-kserve-b84945cb5-pc6f8"
Nov 02 15:11:43.856216 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:43.741365    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-624254/simulator-w3-624254-kserve-56866b7699-z47sd" podUID="345c710f-b414-4e62-86c7-d51ff4e5be15" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:11:43.870147 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:43.743727    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nkbrk" podUID="1f68d837-fff2-418b-84d7-c2d7eb21a57e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.24:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:11:43.891911 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:43.891871    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-624254/simulator-w3-624254-kserve-56866b7699-z47sd"
Nov 02 15:11:43.892247 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:43.892228    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nkbrk"
Nov 02 15:11:43.893554 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:43.783068    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1cc1b2/distinct-w5-1cc1b2-kserve-5b7459bc6d-zl8xr" podUID="211ad15d-9234-4f57-b1b9-181dd2d74208" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:11:43.895802 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:43.895785    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-1cc1b2/distinct-w5-1cc1b2-kserve-5b7459bc6d-zl8xr"
Nov 02 15:11:43.949812 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:43.949761    5304 patch_prober.go:29] interesting pod/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-6gnx4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 15:11:43.950119 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:43.950074    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-6gnx4" podUID="2afb5fd0-d309-42eb-bd1e-9092ddeb166a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:11:43.980699 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:43.980658    5304 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-5dd8e9/premium-w1-5dd8e9-kserve-67588456fb-j6lph" podUID="4ccc607a-14e4-43ca-8999-78100d4b799f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:11:44.079446 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:44.079299    5304 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-1cc1b2/distinct-w5-1cc1b2-kserve-5b7459bc6d-zl8xr" podUID="211ad15d-9234-4f57-b1b9-181dd2d74208" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:11:44.184726 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:44.184626    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-624254/distinct-w3-624254-kserve-5465dc4765-dmq67" podUID="1a5aa910-a6de-417b-8638-81e4ec5c1825" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:11:44.508537 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:44.226508    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5dd8e9/simulator-w1-5dd8e9-kserve-6f9f755d6f-gnw5d" podUID="66b75e64-cb24-40be-9076-4984c132e208" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": context deadline exceeded"
Nov 02 15:11:44.862039 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:44.236674    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6af50f/distinct-w0-6af50f-kserve-c6d994647-hbj9z" podUID="3dca5bf3-8e08-47a0-936e-205670d59ede" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded"
Nov 02 15:11:44.892148 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:44.236743    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-739bf1/distinct-2-w2-739bf1-kserve-5c58479c48-qgfkh" podUID="9746bf8a-91bd-4715-a02e-ae2e651ca54d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:11:44.934723 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:44.347071    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7ecfe1/distinct-w4-7ecfe1-kserve-6c847c647d-d2jvv" podUID="f1f1ca5e-cca7-4c54-939f-06f095defcdd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded"
Nov 02 15:11:45.089150 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:44.347103    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5dd8e9/premium-w1-5dd8e9-kserve-67588456fb-j6lph" podUID="4ccc607a-14e4-43ca-8999-78100d4b799f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded"
Nov 02 15:11:45.089150 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:44.415338    5304 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-739bf1/distinct-2-w2-739bf1-kserve-5c58479c48-qgfkh" podUID="9746bf8a-91bd-4715-a02e-ae2e651ca54d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:11:45.089150 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:44.425664    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-739bf1/simulator-w2-739bf1-kserve-7c74f85465-ltk4t" podUID="cf6a1ded-28a3-49eb-878b-b4ad77a0bb5c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded"
Nov 02 15:11:45.089150 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:44.547258    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5113c2/premium-w6-5113c2-kserve-b84945cb5-pc6f8" podUID="9fb3e992-7bb0-497a-a82c-6e396ba241c1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:11:45.089150 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:44.547333    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1cc1b2/premium-w5-1cc1b2-kserve-58db46f97c-cgdqg" podUID="210f620a-8ae7-4e4c-acc3-46f75ddc6105" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded"
Nov 02 15:11:45.089150 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:44.893109    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nkbrk" podUID="1f68d837-fff2-418b-84d7-c2d7eb21a57e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.24:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:11:45.089150 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:44.893166    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-624254/simulator-w3-624254-kserve-56866b7699-z47sd" podUID="345c710f-b414-4e62-86c7-d51ff4e5be15" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": context deadline exceeded"
Nov 02 15:11:45.089150 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:44.893846    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1cc1b2/distinct-w5-1cc1b2-kserve-5b7459bc6d-zl8xr" podUID="211ad15d-9234-4f57-b1b9-181dd2d74208" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded"
Nov 02 15:11:45.089150 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:45.016017    5304 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-5113c2/premium-w6-5113c2-kserve-b84945cb5-pc6f8" podUID="9fb3e992-7bb0-497a-a82c-6e396ba241c1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:11:45.212443 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:11:45.212338    5304 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.085s"
Nov 02 15:11:45.579380 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:45.579254    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-624254/distinct-w3-624254-kserve-5465dc4765-dmq67" podUID="1a5aa910-a6de-417b-8638-81e4ec5c1825" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:11:45.907308 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:45.907260    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6af50f/distinct-w0-6af50f-kserve-c6d994647-hbj9z" podUID="3dca5bf3-8e08-47a0-936e-205670d59ede" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded"
Nov 02 15:11:45.932268 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:45.927305    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5dd8e9/simulator-w1-5dd8e9-kserve-6f9f755d6f-gnw5d" podUID="66b75e64-cb24-40be-9076-4984c132e208" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:11:45.932618 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:45.927609    5304 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-739bf1/simulator-w2-739bf1-kserve-7c74f85465-ltk4t" podUID="cf6a1ded-28a3-49eb-878b-b4ad77a0bb5c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:11:45.941105 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:45.936107    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-739bf1/distinct-2-w2-739bf1-kserve-5c58479c48-qgfkh" podUID="9746bf8a-91bd-4715-a02e-ae2e651ca54d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:11:46.093513 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:46.093443    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7ecfe1/distinct-w4-7ecfe1-kserve-6c847c647d-d2jvv" podUID="f1f1ca5e-cca7-4c54-939f-06f095defcdd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded"
Nov 02 15:11:46.123806 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:46.123763    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1cc1b2/premium-w5-1cc1b2-kserve-58db46f97c-cgdqg" podUID="210f620a-8ae7-4e4c-acc3-46f75ddc6105" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded"
Nov 02 15:11:46.133287 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:46.133220    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5dd8e9/premium-w1-5dd8e9-kserve-67588456fb-j6lph" podUID="4ccc607a-14e4-43ca-8999-78100d4b799f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded"
Nov 02 15:11:46.134396 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:46.134348    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-739bf1/simulator-w2-739bf1-kserve-7c74f85465-ltk4t" podUID="cf6a1ded-28a3-49eb-878b-b4ad77a0bb5c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded"
Nov 02 15:11:46.188264 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:46.134589    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5113c2/premium-w6-5113c2-kserve-b84945cb5-pc6f8" podUID="9fb3e992-7bb0-497a-a82c-6e396ba241c1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded"
Nov 02 15:11:46.188264 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:46.138186    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-624254/simulator-w3-624254-kserve-56866b7699-z47sd" podUID="345c710f-b414-4e62-86c7-d51ff4e5be15" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": context deadline exceeded"
Nov 02 15:11:46.188264 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:46.138242    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nkbrk" podUID="1f68d837-fff2-418b-84d7-c2d7eb21a57e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.24:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:11:46.203255 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:46.203207    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1cc1b2/distinct-w5-1cc1b2-kserve-5b7459bc6d-zl8xr" podUID="211ad15d-9234-4f57-b1b9-181dd2d74208" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:11:46.603214 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:46.582009    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-624254/distinct-w3-624254-kserve-5465dc4765-dmq67" podUID="1a5aa910-a6de-417b-8638-81e4ec5c1825" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded"
Nov 02 15:11:46.951681 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:46.951566    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6af50f/distinct-w0-6af50f-kserve-c6d994647-hbj9z" podUID="3dca5bf3-8e08-47a0-936e-205670d59ede" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:11:46.972203 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:46.952182    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5dd8e9/simulator-w1-5dd8e9-kserve-6f9f755d6f-gnw5d" podUID="66b75e64-cb24-40be-9076-4984c132e208" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:11:46.972203 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:46.953108    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-739bf1/distinct-2-w2-739bf1-kserve-5c58479c48-qgfkh" podUID="9746bf8a-91bd-4715-a02e-ae2e651ca54d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded"
Nov 02 15:11:47.099552 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:47.098807    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7ecfe1/distinct-w4-7ecfe1-kserve-6c847c647d-d2jvv" podUID="f1f1ca5e-cca7-4c54-939f-06f095defcdd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:11:47.156080 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:47.142949    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-739bf1/simulator-w2-739bf1-kserve-7c74f85465-ltk4t" podUID="cf6a1ded-28a3-49eb-878b-b4ad77a0bb5c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:11:47.156080 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:47.143352    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1cc1b2/premium-w5-1cc1b2-kserve-58db46f97c-cgdqg" podUID="210f620a-8ae7-4e4c-acc3-46f75ddc6105" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded"
Nov 02 15:11:47.156080 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:47.143624    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5dd8e9/premium-w1-5dd8e9-kserve-67588456fb-j6lph" podUID="4ccc607a-14e4-43ca-8999-78100d4b799f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded"
Nov 02 15:11:47.158837 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:47.158382    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nkbrk" podUID="1f68d837-fff2-418b-84d7-c2d7eb21a57e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.24:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:11:47.158837 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:47.158432    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-624254/simulator-w3-624254-kserve-56866b7699-z47sd" podUID="345c710f-b414-4e62-86c7-d51ff4e5be15" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": context deadline exceeded"
Nov 02 15:11:47.158837 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:47.158506    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5113c2/premium-w6-5113c2-kserve-b84945cb5-pc6f8" podUID="9fb3e992-7bb0-497a-a82c-6e396ba241c1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded"
Nov 02 15:11:47.220210 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:47.214895    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1cc1b2/distinct-w5-1cc1b2-kserve-5b7459bc6d-zl8xr" podUID="211ad15d-9234-4f57-b1b9-181dd2d74208" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded"
Nov 02 15:11:47.273082 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:47.272949    5304 patch_prober.go:29] interesting pod/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-9gbvl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 15:11:47.273082 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:47.273008    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-9gbvl" podUID="7a8625f9-7163-4e64-8e52-6537ebbe404c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:11:47.675185 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:47.661223    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-624254/distinct-w3-624254-kserve-5465dc4765-dmq67" podUID="1a5aa910-a6de-417b-8638-81e4ec5c1825" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded"
Nov 02 15:11:47.953382 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:47.953276    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-739bf1/distinct-2-w2-739bf1-kserve-5c58479c48-qgfkh" podUID="9746bf8a-91bd-4715-a02e-ae2e651ca54d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:11:47.958243 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:47.958201    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6af50f/distinct-w0-6af50f-kserve-c6d994647-hbj9z" podUID="3dca5bf3-8e08-47a0-936e-205670d59ede" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:11:47.980153 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:47.967227    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5dd8e9/simulator-w1-5dd8e9-kserve-6f9f755d6f-gnw5d" podUID="66b75e64-cb24-40be-9076-4984c132e208" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": context deadline exceeded"
Nov 02 15:11:48.100149 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:48.099641    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7ecfe1/distinct-w4-7ecfe1-kserve-6c847c647d-d2jvv" podUID="f1f1ca5e-cca7-4c54-939f-06f095defcdd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:11:48.100353 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:48.100247    5304 patch_prober.go:29] interesting pod/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-25flx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 15:11:48.100353 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:48.100307    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-25flx" podUID="6f80b62a-49ba-4922-8f3e-804d6bdcc46f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:11:48.181152 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:48.154427    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5dd8e9/premium-w1-5dd8e9-kserve-67588456fb-j6lph" podUID="4ccc607a-14e4-43ca-8999-78100d4b799f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:11:48.181152 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:48.159231    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-739bf1/simulator-w2-739bf1-kserve-7c74f85465-ltk4t" podUID="cf6a1ded-28a3-49eb-878b-b4ad77a0bb5c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded"
Nov 02 15:11:48.181152 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:48.172350    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1cc1b2/premium-w5-1cc1b2-kserve-58db46f97c-cgdqg" podUID="210f620a-8ae7-4e4c-acc3-46f75ddc6105" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded"
Nov 02 15:11:48.184155 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:48.184097    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5113c2/premium-w6-5113c2-kserve-b84945cb5-pc6f8" podUID="9fb3e992-7bb0-497a-a82c-6e396ba241c1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:11:48.215101 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:48.191074    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nkbrk" podUID="1f68d837-fff2-418b-84d7-c2d7eb21a57e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.24:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:11:48.217549 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:48.217505    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-624254/simulator-w3-624254-kserve-56866b7699-z47sd" podUID="345c710f-b414-4e62-86c7-d51ff4e5be15" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:11:48.269152 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:48.248026    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1cc1b2/distinct-w5-1cc1b2-kserve-5b7459bc6d-zl8xr" podUID="211ad15d-9234-4f57-b1b9-181dd2d74208" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded"
Nov 02 15:11:48.708152 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:48.706229    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-624254/distinct-w3-624254-kserve-5465dc4765-dmq67" podUID="1a5aa910-a6de-417b-8638-81e4ec5c1825" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded"
Nov 02 15:11:48.810149 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:48.765258    5304 patch_prober.go:29] interesting pod/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-9gswz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 15:11:48.810149 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:48.765329    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-9gswz" podUID="45f29144-431b-4948-abfa-a1fc5a6847ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:11:48.972203 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:48.954771    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-739bf1/distinct-2-w2-739bf1-kserve-5c58479c48-qgfkh" podUID="9746bf8a-91bd-4715-a02e-ae2e651ca54d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:11:49.069398 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:49.052913    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5dd8e9/simulator-w1-5dd8e9-kserve-6f9f755d6f-gnw5d" podUID="66b75e64-cb24-40be-9076-4984c132e208" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:11:49.097526 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:49.052966    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6af50f/distinct-w0-6af50f-kserve-c6d994647-hbj9z" podUID="3dca5bf3-8e08-47a0-936e-205670d59ede" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:11:49.104520 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:49.104482    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7ecfe1/distinct-w4-7ecfe1-kserve-6c847c647d-d2jvv" podUID="f1f1ca5e-cca7-4c54-939f-06f095defcdd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded"
Nov 02 15:11:49.160091 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:49.160039    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-739bf1/simulator-w2-739bf1-kserve-7c74f85465-ltk4t" podUID="cf6a1ded-28a3-49eb-878b-b4ad77a0bb5c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded"
Nov 02 15:11:49.160743 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:49.160117    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5dd8e9/premium-w1-5dd8e9-kserve-67588456fb-j6lph" podUID="4ccc607a-14e4-43ca-8999-78100d4b799f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:11:49.194323 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:49.194277    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5113c2/premium-w6-5113c2-kserve-b84945cb5-pc6f8" podUID="9fb3e992-7bb0-497a-a82c-6e396ba241c1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:11:49.246886 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:49.230754    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nkbrk" podUID="1f68d837-fff2-418b-84d7-c2d7eb21a57e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.24:8000/ready\": context deadline exceeded"
Nov 02 15:11:49.246886 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:49.231276    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-624254/simulator-w3-624254-kserve-56866b7699-z47sd" podUID="345c710f-b414-4e62-86c7-d51ff4e5be15" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": context deadline exceeded"
Nov 02 15:11:49.246886 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:49.232176    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1cc1b2/premium-w5-1cc1b2-kserve-58db46f97c-cgdqg" podUID="210f620a-8ae7-4e4c-acc3-46f75ddc6105" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:11:49.276162 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:49.272229    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1cc1b2/distinct-w5-1cc1b2-kserve-5b7459bc6d-zl8xr" podUID="211ad15d-9234-4f57-b1b9-181dd2d74208" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded"
Nov 02 15:11:49.290153 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:49.289313    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5dd8e9/premium-w1-5dd8e9-kserve-67588456fb-j6lph"
Nov 02 15:11:49.344565 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:49.342911    5304 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-5dd8e9/simulator-w1-5dd8e9-kserve-6f9f755d6f-gnw5d" podUID="66b75e64-cb24-40be-9076-4984c132e208" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:11:49.349308 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:49.346867    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nkbrk"
Nov 02 15:11:49.366537 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:49.366511    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-739bf1/distinct-2-w2-739bf1-kserve-5c58479c48-qgfkh"
Nov 02 15:11:49.401799 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:49.401767    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-624254/simulator-w3-624254-kserve-56866b7699-z47sd"
Nov 02 15:11:49.405682 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:49.405656    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5dd8e9/simulator-w1-5dd8e9-kserve-6f9f755d6f-gnw5d"
Nov 02 15:11:49.419316 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:49.419293    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-1cc1b2/premium-w5-1cc1b2-kserve-58db46f97c-cgdqg"
Nov 02 15:11:49.419554 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:49.419539    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-7ecfe1/distinct-w4-7ecfe1-kserve-6c847c647d-d2jvv"
Nov 02 15:11:49.421312 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:49.421291    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-6af50f/distinct-w0-6af50f-kserve-c6d994647-hbj9z"
Nov 02 15:11:49.423045 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:49.423022    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-739bf1/simulator-w2-739bf1-kserve-7c74f85465-ltk4t"
Nov 02 15:11:49.424088 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:49.424066    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-5113c2/premium-w6-5113c2-kserve-b84945cb5-pc6f8"
Nov 02 15:11:49.428310 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:49.428291    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-1cc1b2/distinct-w5-1cc1b2-kserve-5b7459bc6d-zl8xr"
Nov 02 15:11:49.429273 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:49.429252    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-624254/distinct-w3-624254-kserve-5465dc4765-dmq67"
Nov 02 15:11:54.206183 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:54.206074    5304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-lphrz"]
Nov 02 15:11:54.210685 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:54.210660    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-lphrz"
Nov 02 15:11:54.244450 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:54.244394    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-lphrz"]
Nov 02 15:11:54.323104 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:54.323073    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/474f879d-3a52-4e47-8eb0-95a4fcfbe399-workload-socket\") pod \"e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-lphrz\" (UID: \"474f879d-3a52-4e47-8eb0-95a4fcfbe399\") " pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-lphrz"
Nov 02 15:11:54.323289 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:54.323240    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/474f879d-3a52-4e47-8eb0-95a4fcfbe399-credential-socket\") pod \"e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-lphrz\" (UID: \"474f879d-3a52-4e47-8eb0-95a4fcfbe399\") " pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-lphrz"
Nov 02 15:11:54.323343 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:54.323328    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/474f879d-3a52-4e47-8eb0-95a4fcfbe399-istio-envoy\") pod \"e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-lphrz\" (UID: \"474f879d-3a52-4e47-8eb0-95a4fcfbe399\") " pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-lphrz"
Nov 02 15:11:54.323380 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:54.323368    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/474f879d-3a52-4e47-8eb0-95a4fcfbe399-workload-certs\") pod \"e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-lphrz\" (UID: \"474f879d-3a52-4e47-8eb0-95a4fcfbe399\") " pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-lphrz"
Nov 02 15:11:54.323413 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:54.323385    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/474f879d-3a52-4e47-8eb0-95a4fcfbe399-istio-token\") pod \"e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-lphrz\" (UID: \"474f879d-3a52-4e47-8eb0-95a4fcfbe399\") " pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-lphrz"
Nov 02 15:11:54.323483 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:54.323471    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ncrxc\" (UniqueName: \"kubernetes.io/projected/474f879d-3a52-4e47-8eb0-95a4fcfbe399-kube-api-access-ncrxc\") pod \"e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-lphrz\" (UID: \"474f879d-3a52-4e47-8eb0-95a4fcfbe399\") " pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-lphrz"
Nov 02 15:11:54.323522 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:54.323510    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/474f879d-3a52-4e47-8eb0-95a4fcfbe399-istio-data\") pod \"e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-lphrz\" (UID: \"474f879d-3a52-4e47-8eb0-95a4fcfbe399\") " pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-lphrz"
Nov 02 15:11:54.323556 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:54.323548    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/474f879d-3a52-4e47-8eb0-95a4fcfbe399-istiod-ca-cert\") pod \"e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-lphrz\" (UID: \"474f879d-3a52-4e47-8eb0-95a4fcfbe399\") " pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-lphrz"
Nov 02 15:11:54.323671 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:54.323647    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/474f879d-3a52-4e47-8eb0-95a4fcfbe399-istio-podinfo\") pod \"e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-lphrz\" (UID: \"474f879d-3a52-4e47-8eb0-95a4fcfbe399\") " pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-lphrz"
Nov 02 15:11:54.419787 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:54.419751    5304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-k2p6m"]
Nov 02 15:11:54.424755 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:54.424721    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-k2p6m"
Nov 02 15:11:54.425465 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:54.425429    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/474f879d-3a52-4e47-8eb0-95a4fcfbe399-workload-certs\") pod \"e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-lphrz\" (UID: \"474f879d-3a52-4e47-8eb0-95a4fcfbe399\") " pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-lphrz"
Nov 02 15:11:54.425465 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:54.425479    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/474f879d-3a52-4e47-8eb0-95a4fcfbe399-istio-token\") pod \"e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-lphrz\" (UID: \"474f879d-3a52-4e47-8eb0-95a4fcfbe399\") " pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-lphrz"
Nov 02 15:11:54.425800 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:54.425594    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ncrxc\" (UniqueName: \"kubernetes.io/projected/474f879d-3a52-4e47-8eb0-95a4fcfbe399-kube-api-access-ncrxc\") pod \"e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-lphrz\" (UID: \"474f879d-3a52-4e47-8eb0-95a4fcfbe399\") " pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-lphrz"
Nov 02 15:11:54.425800 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:54.425661    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/474f879d-3a52-4e47-8eb0-95a4fcfbe399-istio-data\") pod \"e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-lphrz\" (UID: \"474f879d-3a52-4e47-8eb0-95a4fcfbe399\") " pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-lphrz"
Nov 02 15:11:54.425800 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:54.425728    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/474f879d-3a52-4e47-8eb0-95a4fcfbe399-istiod-ca-cert\") pod \"e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-lphrz\" (UID: \"474f879d-3a52-4e47-8eb0-95a4fcfbe399\") " pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-lphrz"
Nov 02 15:11:54.425800 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:54.425760    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/474f879d-3a52-4e47-8eb0-95a4fcfbe399-istio-podinfo\") pod \"e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-lphrz\" (UID: \"474f879d-3a52-4e47-8eb0-95a4fcfbe399\") " pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-lphrz"
Nov 02 15:11:54.426090 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:54.425811    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/474f879d-3a52-4e47-8eb0-95a4fcfbe399-workload-socket\") pod \"e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-lphrz\" (UID: \"474f879d-3a52-4e47-8eb0-95a4fcfbe399\") " pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-lphrz"
Nov 02 15:11:54.426090 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:54.425958    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/474f879d-3a52-4e47-8eb0-95a4fcfbe399-credential-socket\") pod \"e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-lphrz\" (UID: \"474f879d-3a52-4e47-8eb0-95a4fcfbe399\") " pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-lphrz"
Nov 02 15:11:54.426090 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:54.426050    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/474f879d-3a52-4e47-8eb0-95a4fcfbe399-istio-envoy\") pod \"e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-lphrz\" (UID: \"474f879d-3a52-4e47-8eb0-95a4fcfbe399\") " pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-lphrz"
Nov 02 15:11:54.426594 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:54.426570    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/474f879d-3a52-4e47-8eb0-95a4fcfbe399-workload-socket\") pod \"e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-lphrz\" (UID: \"474f879d-3a52-4e47-8eb0-95a4fcfbe399\") " pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-lphrz"
Nov 02 15:11:54.426809 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:54.426772    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/474f879d-3a52-4e47-8eb0-95a4fcfbe399-credential-socket\") pod \"e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-lphrz\" (UID: \"474f879d-3a52-4e47-8eb0-95a4fcfbe399\") " pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-lphrz"
Nov 02 15:11:54.426880 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:54.426801    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/474f879d-3a52-4e47-8eb0-95a4fcfbe399-istio-data\") pod \"e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-lphrz\" (UID: \"474f879d-3a52-4e47-8eb0-95a4fcfbe399\") " pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-lphrz"
Nov 02 15:11:54.426880 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:54.426852    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/474f879d-3a52-4e47-8eb0-95a4fcfbe399-istiod-ca-cert\") pod \"e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-lphrz\" (UID: \"474f879d-3a52-4e47-8eb0-95a4fcfbe399\") " pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-lphrz"
Nov 02 15:11:54.427197 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:54.427075    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/474f879d-3a52-4e47-8eb0-95a4fcfbe399-workload-certs\") pod \"e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-lphrz\" (UID: \"474f879d-3a52-4e47-8eb0-95a4fcfbe399\") " pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-lphrz"
Nov 02 15:11:54.429613 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:54.429584    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/474f879d-3a52-4e47-8eb0-95a4fcfbe399-istio-podinfo\") pod \"e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-lphrz\" (UID: \"474f879d-3a52-4e47-8eb0-95a4fcfbe399\") " pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-lphrz"
Nov 02 15:11:54.436267 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:54.436116    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/474f879d-3a52-4e47-8eb0-95a4fcfbe399-istio-envoy\") pod \"e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-lphrz\" (UID: \"474f879d-3a52-4e47-8eb0-95a4fcfbe399\") " pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-lphrz"
Nov 02 15:11:54.448292 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:54.448256    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-k2p6m"]
Nov 02 15:11:54.455777 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:54.455746    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/474f879d-3a52-4e47-8eb0-95a4fcfbe399-istio-token\") pod \"e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-lphrz\" (UID: \"474f879d-3a52-4e47-8eb0-95a4fcfbe399\") " pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-lphrz"
Nov 02 15:11:54.456836 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:54.456764    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ncrxc\" (UniqueName: \"kubernetes.io/projected/474f879d-3a52-4e47-8eb0-95a4fcfbe399-kube-api-access-ncrxc\") pod \"e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-lphrz\" (UID: \"474f879d-3a52-4e47-8eb0-95a4fcfbe399\") " pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-lphrz"
Nov 02 15:11:54.524803 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:54.524774    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-lphrz"
Nov 02 15:11:54.527436 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:54.527395    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j4v6m\" (UniqueName: \"kubernetes.io/projected/1bd23848-eb1d-45fd-9d87-c11302b3c111-kube-api-access-j4v6m\") pod \"e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-k2p6m\" (UID: \"1bd23848-eb1d-45fd-9d87-c11302b3c111\") " pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-k2p6m"
Nov 02 15:11:54.527578 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:54.527440    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1bd23848-eb1d-45fd-9d87-c11302b3c111-istio-data\") pod \"e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-k2p6m\" (UID: \"1bd23848-eb1d-45fd-9d87-c11302b3c111\") " pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-k2p6m"
Nov 02 15:11:54.527578 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:54.527554    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bd23848-eb1d-45fd-9d87-c11302b3c111-credential-socket\") pod \"e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-k2p6m\" (UID: \"1bd23848-eb1d-45fd-9d87-c11302b3c111\") " pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-k2p6m"
Nov 02 15:11:54.527709 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:54.527612    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1bd23848-eb1d-45fd-9d87-c11302b3c111-workload-certs\") pod \"e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-k2p6m\" (UID: \"1bd23848-eb1d-45fd-9d87-c11302b3c111\") " pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-k2p6m"
Nov 02 15:11:54.527709 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:54.527638    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1bd23848-eb1d-45fd-9d87-c11302b3c111-istiod-ca-cert\") pod \"e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-k2p6m\" (UID: \"1bd23848-eb1d-45fd-9d87-c11302b3c111\") " pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-k2p6m"
Nov 02 15:11:54.527822 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:54.527768    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1bd23848-eb1d-45fd-9d87-c11302b3c111-istio-podinfo\") pod \"e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-k2p6m\" (UID: \"1bd23848-eb1d-45fd-9d87-c11302b3c111\") " pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-k2p6m"
Nov 02 15:11:54.528396 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:54.527895    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1bd23848-eb1d-45fd-9d87-c11302b3c111-istio-token\") pod \"e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-k2p6m\" (UID: \"1bd23848-eb1d-45fd-9d87-c11302b3c111\") " pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-k2p6m"
Nov 02 15:11:54.528396 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:54.527925    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bd23848-eb1d-45fd-9d87-c11302b3c111-workload-socket\") pod \"e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-k2p6m\" (UID: \"1bd23848-eb1d-45fd-9d87-c11302b3c111\") " pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-k2p6m"
Nov 02 15:11:54.528396 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:54.528317    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1bd23848-eb1d-45fd-9d87-c11302b3c111-istio-envoy\") pod \"e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-k2p6m\" (UID: \"1bd23848-eb1d-45fd-9d87-c11302b3c111\") " pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-k2p6m"
Nov 02 15:11:54.630621 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:54.630585    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1bd23848-eb1d-45fd-9d87-c11302b3c111-istio-envoy\") pod \"e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-k2p6m\" (UID: \"1bd23848-eb1d-45fd-9d87-c11302b3c111\") " pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-k2p6m"
Nov 02 15:11:54.630797 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:54.630656    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j4v6m\" (UniqueName: \"kubernetes.io/projected/1bd23848-eb1d-45fd-9d87-c11302b3c111-kube-api-access-j4v6m\") pod \"e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-k2p6m\" (UID: \"1bd23848-eb1d-45fd-9d87-c11302b3c111\") " pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-k2p6m"
Nov 02 15:11:54.630797 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:54.630700    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1bd23848-eb1d-45fd-9d87-c11302b3c111-istio-data\") pod \"e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-k2p6m\" (UID: \"1bd23848-eb1d-45fd-9d87-c11302b3c111\") " pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-k2p6m"
Nov 02 15:11:54.630797 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:54.630779    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bd23848-eb1d-45fd-9d87-c11302b3c111-credential-socket\") pod \"e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-k2p6m\" (UID: \"1bd23848-eb1d-45fd-9d87-c11302b3c111\") " pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-k2p6m"
Nov 02 15:11:54.630972 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:54.630830    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1bd23848-eb1d-45fd-9d87-c11302b3c111-workload-certs\") pod \"e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-k2p6m\" (UID: \"1bd23848-eb1d-45fd-9d87-c11302b3c111\") " pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-k2p6m"
Nov 02 15:11:54.630972 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:54.630857    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1bd23848-eb1d-45fd-9d87-c11302b3c111-istiod-ca-cert\") pod \"e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-k2p6m\" (UID: \"1bd23848-eb1d-45fd-9d87-c11302b3c111\") " pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-k2p6m"
Nov 02 15:11:54.630972 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:54.630916    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1bd23848-eb1d-45fd-9d87-c11302b3c111-istio-podinfo\") pod \"e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-k2p6m\" (UID: \"1bd23848-eb1d-45fd-9d87-c11302b3c111\") " pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-k2p6m"
Nov 02 15:11:54.631179 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:54.631013    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1bd23848-eb1d-45fd-9d87-c11302b3c111-istio-token\") pod \"e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-k2p6m\" (UID: \"1bd23848-eb1d-45fd-9d87-c11302b3c111\") " pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-k2p6m"
Nov 02 15:11:54.631179 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:54.631047    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bd23848-eb1d-45fd-9d87-c11302b3c111-workload-socket\") pod \"e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-k2p6m\" (UID: \"1bd23848-eb1d-45fd-9d87-c11302b3c111\") " pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-k2p6m"
Nov 02 15:11:54.631319 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:54.631290    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bd23848-eb1d-45fd-9d87-c11302b3c111-credential-socket\") pod \"e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-k2p6m\" (UID: \"1bd23848-eb1d-45fd-9d87-c11302b3c111\") " pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-k2p6m"
Nov 02 15:11:54.631382 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:54.631363    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1bd23848-eb1d-45fd-9d87-c11302b3c111-istio-data\") pod \"e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-k2p6m\" (UID: \"1bd23848-eb1d-45fd-9d87-c11302b3c111\") " pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-k2p6m"
Nov 02 15:11:54.632248 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:54.632002    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bd23848-eb1d-45fd-9d87-c11302b3c111-workload-socket\") pod \"e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-k2p6m\" (UID: \"1bd23848-eb1d-45fd-9d87-c11302b3c111\") " pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-k2p6m"
Nov 02 15:11:54.632248 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:54.632203    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1bd23848-eb1d-45fd-9d87-c11302b3c111-workload-certs\") pod \"e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-k2p6m\" (UID: \"1bd23848-eb1d-45fd-9d87-c11302b3c111\") " pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-k2p6m"
Nov 02 15:11:54.632628 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:54.632602    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1bd23848-eb1d-45fd-9d87-c11302b3c111-istiod-ca-cert\") pod \"e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-k2p6m\" (UID: \"1bd23848-eb1d-45fd-9d87-c11302b3c111\") " pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-k2p6m"
Nov 02 15:11:54.634073 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:54.634048    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1bd23848-eb1d-45fd-9d87-c11302b3c111-istio-envoy\") pod \"e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-k2p6m\" (UID: \"1bd23848-eb1d-45fd-9d87-c11302b3c111\") " pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-k2p6m"
Nov 02 15:11:54.634313 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:54.634290    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1bd23848-eb1d-45fd-9d87-c11302b3c111-istio-podinfo\") pod \"e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-k2p6m\" (UID: \"1bd23848-eb1d-45fd-9d87-c11302b3c111\") " pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-k2p6m"
Nov 02 15:11:54.655619 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:54.655566    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1bd23848-eb1d-45fd-9d87-c11302b3c111-istio-token\") pod \"e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-k2p6m\" (UID: \"1bd23848-eb1d-45fd-9d87-c11302b3c111\") " pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-k2p6m"
Nov 02 15:11:54.655791 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:54.655766    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j4v6m\" (UniqueName: \"kubernetes.io/projected/1bd23848-eb1d-45fd-9d87-c11302b3c111-kube-api-access-j4v6m\") pod \"e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-k2p6m\" (UID: \"1bd23848-eb1d-45fd-9d87-c11302b3c111\") " pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-k2p6m"
Nov 02 15:11:54.791148 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:54.791066    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-k2p6m"
Nov 02 15:11:54.801745 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:11:54.801715    5304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k6fjk" containerName="extractor"
Nov 02 15:11:54.858955 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:54.858923    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-lphrz"]
Nov 02 15:11:54.869673 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:54.869628    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:11:54.869792 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:54.869725    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:11:54.869792 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:54.869780    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:11:54.895717 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:54.895678    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-lphrz" event={"ID":"474f879d-3a52-4e47-8eb0-95a4fcfbe399","Type":"ContainerStarted","Data":"006118828f472196afdf4a95e8db0498abb6b757ec29eaa3655abb58ee27e317"}
Nov 02 15:11:55.105992 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:55.105961    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-k2p6m"]
Nov 02 15:11:55.110915 ip-10-0-135-138 kubenswrapper[5304]: W1102 15:11:55.110883    5304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1bd23848_eb1d_45fd_9d87_c11302b3c111.slice/crio-a19541c6415fcf38aa1ec0e52f1368b479c5275c975d1b462a73ba7a905e650e WatchSource:0}: Error finding container a19541c6415fcf38aa1ec0e52f1368b479c5275c975d1b462a73ba7a905e650e: Status 404 returned error can't find the container with id a19541c6415fcf38aa1ec0e52f1368b479c5275c975d1b462a73ba7a905e650e
Nov 02 15:11:55.114351 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:55.114315    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:11:55.114462 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:55.114399    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:11:55.114462 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:55.114445    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:11:55.905522 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:55.905476    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-k2p6m" event={"ID":"1bd23848-eb1d-45fd-9d87-c11302b3c111","Type":"ContainerStarted","Data":"d6bc4523c8b8264ec504666897583f330e20731aee7d610dbbf90cd79d0807b9"}
Nov 02 15:11:55.906035 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:55.905554    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-k2p6m" event={"ID":"1bd23848-eb1d-45fd-9d87-c11302b3c111","Type":"ContainerStarted","Data":"a19541c6415fcf38aa1ec0e52f1368b479c5275c975d1b462a73ba7a905e650e"}
Nov 02 15:11:55.907487 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:55.907457    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-lphrz" event={"ID":"474f879d-3a52-4e47-8eb0-95a4fcfbe399","Type":"ContainerStarted","Data":"43cb8a845d15be46fe6bb789d7bb2ed4ae2799cb345e07749961896e872f4875"}
Nov 02 15:11:55.988070 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:55.987996    5304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-k2p6m" podStartSLOduration=1.987974657 podStartE2EDuration="1.987974657s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:11:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:11:55.973337543 +0000 UTC m=+915.967161952" watchObservedRunningTime="2026-11-02 15:11:55.987974657 +0000 UTC m=+915.981799068"
Nov 02 15:11:56.020886 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:56.020836    5304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-lphrz" podStartSLOduration=2.020820494 podStartE2EDuration="2.020820494s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:11:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:11:56.018524058 +0000 UTC m=+916.012348467" watchObservedRunningTime="2026-11-02 15:11:56.020820494 +0000 UTC m=+916.014644911"
Nov 02 15:11:56.525009 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:56.524970    5304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-lphrz"
Nov 02 15:11:56.526537 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:56.526509    5304 patch_prober.go:29] interesting pod/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-lphrz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Nov 02 15:11:56.526649 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:56.526560    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-lphrz" podUID="474f879d-3a52-4e47-8eb0-95a4fcfbe399" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Nov 02 15:11:56.791661 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:56.791352    5304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-k2p6m"
Nov 02 15:11:56.793312 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:56.793282    5304 patch_prober.go:29] interesting pod/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-k2p6m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Nov 02 15:11:56.793472 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:56.793341    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-k2p6m" podUID="1bd23848-eb1d-45fd-9d87-c11302b3c111" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Nov 02 15:11:57.525215 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:57.525177    5304 patch_prober.go:29] interesting pod/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-lphrz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Nov 02 15:11:57.525720 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:57.525236    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-lphrz" podUID="474f879d-3a52-4e47-8eb0-95a4fcfbe399" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Nov 02 15:11:57.793422 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:57.793291    5304 patch_prober.go:29] interesting pod/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-k2p6m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Nov 02 15:11:57.793422 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:57.793363    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-k2p6m" podUID="1bd23848-eb1d-45fd-9d87-c11302b3c111" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Nov 02 15:11:58.525945 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:58.525911    5304 patch_prober.go:29] interesting pod/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-lphrz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Nov 02 15:11:58.526346 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:58.525979    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-lphrz" podUID="474f879d-3a52-4e47-8eb0-95a4fcfbe399" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Nov 02 15:11:58.793561 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:58.793459    5304 patch_prober.go:29] interesting pod/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-k2p6m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Nov 02 15:11:58.793561 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:58.793521    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-k2p6m" podUID="1bd23848-eb1d-45fd-9d87-c11302b3c111" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Nov 02 15:11:59.528563 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:59.528513    5304 patch_prober.go:29] interesting pod/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-lphrz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Nov 02 15:11:59.529221 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:59.529184    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-lphrz" podUID="474f879d-3a52-4e47-8eb0-95a4fcfbe399" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Nov 02 15:11:59.791802 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:59.791719    5304 patch_prober.go:29] interesting pod/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-k2p6m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Nov 02 15:11:59.792020 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:11:59.791997    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-k2p6m" podUID="1bd23848-eb1d-45fd-9d87-c11302b3c111" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Nov 02 15:12:00.526273 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:12:00.526242    5304 patch_prober.go:29] interesting pod/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-lphrz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Nov 02 15:12:00.526525 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:12:00.526498    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-lphrz" podUID="474f879d-3a52-4e47-8eb0-95a4fcfbe399" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Nov 02 15:12:00.793068 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:12:00.792975    5304 patch_prober.go:29] interesting pod/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-k2p6m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Nov 02 15:12:00.793614 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:12:00.793573    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-k2p6m" podUID="1bd23848-eb1d-45fd-9d87-c11302b3c111" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Nov 02 15:12:01.526820 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:12:01.526785    5304 patch_prober.go:29] interesting pod/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-lphrz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Nov 02 15:12:01.527008 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:12:01.526851    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-lphrz" podUID="474f879d-3a52-4e47-8eb0-95a4fcfbe399" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Nov 02 15:12:01.792179 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:12:01.792085    5304 patch_prober.go:29] interesting pod/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-k2p6m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Nov 02 15:12:01.792179 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:12:01.792163    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-k2p6m" podUID="1bd23848-eb1d-45fd-9d87-c11302b3c111" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Nov 02 15:12:02.532102 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:12:02.532065    5304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-lphrz"
Nov 02 15:12:02.532790 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:12:02.532770    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-lphrz"
Nov 02 15:12:02.533549 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:12:02.533525    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-lphrz"
Nov 02 15:12:02.797059 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:12:02.796984    5304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-k2p6m"
Nov 02 15:12:02.797441 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:12:02.797421    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-k2p6m"
Nov 02 15:12:02.798222 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:12:02.798201    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-k2p6m"
Nov 02 15:12:09.854951 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:12:09.854918    5304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-xcfbs"]
Nov 02 15:12:09.860449 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:12:09.860427    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-xcfbs"
Nov 02 15:12:09.894973 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:12:09.894944    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-xcfbs"]
Nov 02 15:12:09.967893 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:12:09.967853    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a395ec29-6257-43f1-a2f0-1ec07bf215ad-istio-podinfo\") pod \"e2e-worker-w0-6af50f-openshift-default-8545fb68f-xcfbs\" (UID: \"a395ec29-6257-43f1-a2f0-1ec07bf215ad\") " pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-xcfbs"
Nov 02 15:12:09.968158 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:12:09.967950    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a395ec29-6257-43f1-a2f0-1ec07bf215ad-workload-certs\") pod \"e2e-worker-w0-6af50f-openshift-default-8545fb68f-xcfbs\" (UID: \"a395ec29-6257-43f1-a2f0-1ec07bf215ad\") " pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-xcfbs"
Nov 02 15:12:09.968158 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:12:09.968046    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a395ec29-6257-43f1-a2f0-1ec07bf215ad-credential-socket\") pod \"e2e-worker-w0-6af50f-openshift-default-8545fb68f-xcfbs\" (UID: \"a395ec29-6257-43f1-a2f0-1ec07bf215ad\") " pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-xcfbs"
Nov 02 15:12:09.968237 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:12:09.968189    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a395ec29-6257-43f1-a2f0-1ec07bf215ad-workload-socket\") pod \"e2e-worker-w0-6af50f-openshift-default-8545fb68f-xcfbs\" (UID: \"a395ec29-6257-43f1-a2f0-1ec07bf215ad\") " pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-xcfbs"
Nov 02 15:12:09.968344 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:12:09.968328    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a395ec29-6257-43f1-a2f0-1ec07bf215ad-istio-data\") pod \"e2e-worker-w0-6af50f-openshift-default-8545fb68f-xcfbs\" (UID: \"a395ec29-6257-43f1-a2f0-1ec07bf215ad\") " pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-xcfbs"
Nov 02 15:12:09.968439 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:12:09.968415    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a395ec29-6257-43f1-a2f0-1ec07bf215ad-istio-envoy\") pod \"e2e-worker-w0-6af50f-openshift-default-8545fb68f-xcfbs\" (UID: \"a395ec29-6257-43f1-a2f0-1ec07bf215ad\") " pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-xcfbs"
Nov 02 15:12:09.968481 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:12:09.968458    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zs497\" (UniqueName: \"kubernetes.io/projected/a395ec29-6257-43f1-a2f0-1ec07bf215ad-kube-api-access-zs497\") pod \"e2e-worker-w0-6af50f-openshift-default-8545fb68f-xcfbs\" (UID: \"a395ec29-6257-43f1-a2f0-1ec07bf215ad\") " pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-xcfbs"
Nov 02 15:12:09.968552 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:12:09.968532    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a395ec29-6257-43f1-a2f0-1ec07bf215ad-istiod-ca-cert\") pod \"e2e-worker-w0-6af50f-openshift-default-8545fb68f-xcfbs\" (UID: \"a395ec29-6257-43f1-a2f0-1ec07bf215ad\") " pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-xcfbs"
Nov 02 15:12:09.968698 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:12:09.968666    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a395ec29-6257-43f1-a2f0-1ec07bf215ad-istio-token\") pod \"e2e-worker-w0-6af50f-openshift-default-8545fb68f-xcfbs\" (UID: \"a395ec29-6257-43f1-a2f0-1ec07bf215ad\") " pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-xcfbs"
Nov 02 15:12:10.069605 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:12:10.069560    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a395ec29-6257-43f1-a2f0-1ec07bf215ad-istio-envoy\") pod \"e2e-worker-w0-6af50f-openshift-default-8545fb68f-xcfbs\" (UID: \"a395ec29-6257-43f1-a2f0-1ec07bf215ad\") " pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-xcfbs"
Nov 02 15:12:10.069789 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:12:10.069677    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zs497\" (UniqueName: \"kubernetes.io/projected/a395ec29-6257-43f1-a2f0-1ec07bf215ad-kube-api-access-zs497\") pod \"e2e-worker-w0-6af50f-openshift-default-8545fb68f-xcfbs\" (UID: \"a395ec29-6257-43f1-a2f0-1ec07bf215ad\") " pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-xcfbs"
Nov 02 15:12:10.069789 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:12:10.069713    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a395ec29-6257-43f1-a2f0-1ec07bf215ad-istiod-ca-cert\") pod \"e2e-worker-w0-6af50f-openshift-default-8545fb68f-xcfbs\" (UID: \"a395ec29-6257-43f1-a2f0-1ec07bf215ad\") " pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-xcfbs"
Nov 02 15:12:10.069907 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:12:10.069841    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a395ec29-6257-43f1-a2f0-1ec07bf215ad-istio-token\") pod \"e2e-worker-w0-6af50f-openshift-default-8545fb68f-xcfbs\" (UID: \"a395ec29-6257-43f1-a2f0-1ec07bf215ad\") " pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-xcfbs"
Nov 02 15:12:10.069907 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:12:10.069872    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a395ec29-6257-43f1-a2f0-1ec07bf215ad-istio-podinfo\") pod \"e2e-worker-w0-6af50f-openshift-default-8545fb68f-xcfbs\" (UID: \"a395ec29-6257-43f1-a2f0-1ec07bf215ad\") " pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-xcfbs"
Nov 02 15:12:10.070005 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:12:10.069924    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a395ec29-6257-43f1-a2f0-1ec07bf215ad-workload-certs\") pod \"e2e-worker-w0-6af50f-openshift-default-8545fb68f-xcfbs\" (UID: \"a395ec29-6257-43f1-a2f0-1ec07bf215ad\") " pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-xcfbs"
Nov 02 15:12:10.070005 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:12:10.069988    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a395ec29-6257-43f1-a2f0-1ec07bf215ad-credential-socket\") pod \"e2e-worker-w0-6af50f-openshift-default-8545fb68f-xcfbs\" (UID: \"a395ec29-6257-43f1-a2f0-1ec07bf215ad\") " pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-xcfbs"
Nov 02 15:12:10.070142 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:12:10.070111    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a395ec29-6257-43f1-a2f0-1ec07bf215ad-workload-socket\") pod \"e2e-worker-w0-6af50f-openshift-default-8545fb68f-xcfbs\" (UID: \"a395ec29-6257-43f1-a2f0-1ec07bf215ad\") " pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-xcfbs"
Nov 02 15:12:10.070298 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:12:10.070282    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a395ec29-6257-43f1-a2f0-1ec07bf215ad-istio-data\") pod \"e2e-worker-w0-6af50f-openshift-default-8545fb68f-xcfbs\" (UID: \"a395ec29-6257-43f1-a2f0-1ec07bf215ad\") " pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-xcfbs"
Nov 02 15:12:10.070638 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:12:10.070613    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a395ec29-6257-43f1-a2f0-1ec07bf215ad-workload-certs\") pod \"e2e-worker-w0-6af50f-openshift-default-8545fb68f-xcfbs\" (UID: \"a395ec29-6257-43f1-a2f0-1ec07bf215ad\") " pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-xcfbs"
Nov 02 15:12:10.070861 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:12:10.070835    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a395ec29-6257-43f1-a2f0-1ec07bf215ad-istiod-ca-cert\") pod \"e2e-worker-w0-6af50f-openshift-default-8545fb68f-xcfbs\" (UID: \"a395ec29-6257-43f1-a2f0-1ec07bf215ad\") " pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-xcfbs"
Nov 02 15:12:10.071176 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:12:10.071151    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a395ec29-6257-43f1-a2f0-1ec07bf215ad-istio-data\") pod \"e2e-worker-w0-6af50f-openshift-default-8545fb68f-xcfbs\" (UID: \"a395ec29-6257-43f1-a2f0-1ec07bf215ad\") " pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-xcfbs"
Nov 02 15:12:10.071490 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:12:10.071462    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a395ec29-6257-43f1-a2f0-1ec07bf215ad-credential-socket\") pod \"e2e-worker-w0-6af50f-openshift-default-8545fb68f-xcfbs\" (UID: \"a395ec29-6257-43f1-a2f0-1ec07bf215ad\") " pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-xcfbs"
Nov 02 15:12:10.071767 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:12:10.071744    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a395ec29-6257-43f1-a2f0-1ec07bf215ad-workload-socket\") pod \"e2e-worker-w0-6af50f-openshift-default-8545fb68f-xcfbs\" (UID: \"a395ec29-6257-43f1-a2f0-1ec07bf215ad\") " pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-xcfbs"
Nov 02 15:12:10.073091 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:12:10.073059    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a395ec29-6257-43f1-a2f0-1ec07bf215ad-istio-envoy\") pod \"e2e-worker-w0-6af50f-openshift-default-8545fb68f-xcfbs\" (UID: \"a395ec29-6257-43f1-a2f0-1ec07bf215ad\") " pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-xcfbs"
Nov 02 15:12:10.080139 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:12:10.074320    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a395ec29-6257-43f1-a2f0-1ec07bf215ad-istio-podinfo\") pod \"e2e-worker-w0-6af50f-openshift-default-8545fb68f-xcfbs\" (UID: \"a395ec29-6257-43f1-a2f0-1ec07bf215ad\") " pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-xcfbs"
Nov 02 15:12:10.094166 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:12:10.094116    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a395ec29-6257-43f1-a2f0-1ec07bf215ad-istio-token\") pod \"e2e-worker-w0-6af50f-openshift-default-8545fb68f-xcfbs\" (UID: \"a395ec29-6257-43f1-a2f0-1ec07bf215ad\") " pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-xcfbs"
Nov 02 15:12:10.097421 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:12:10.097393    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zs497\" (UniqueName: \"kubernetes.io/projected/a395ec29-6257-43f1-a2f0-1ec07bf215ad-kube-api-access-zs497\") pod \"e2e-worker-w0-6af50f-openshift-default-8545fb68f-xcfbs\" (UID: \"a395ec29-6257-43f1-a2f0-1ec07bf215ad\") " pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-xcfbs"
Nov 02 15:12:10.175364 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:12:10.175278    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-xcfbs"
Nov 02 15:12:10.500719 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:12:10.500691    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-xcfbs"]
Nov 02 15:12:10.502791 ip-10-0-135-138 kubenswrapper[5304]: W1102 15:12:10.502763    5304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda395ec29_6257_43f1_a2f0_1ec07bf215ad.slice/crio-ad18961e6c5af7315acd59e55cc42053649a1abb7945dba189403e58cfbe2af5 WatchSource:0}: Error finding container ad18961e6c5af7315acd59e55cc42053649a1abb7945dba189403e58cfbe2af5: Status 404 returned error can't find the container with id ad18961e6c5af7315acd59e55cc42053649a1abb7945dba189403e58cfbe2af5
Nov 02 15:12:10.505339 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:12:10.505303    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:12:10.505442 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:12:10.505382    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:12:10.505442 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:12:10.505431    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:12:11.060473 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:12:11.060437    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-xcfbs" event={"ID":"a395ec29-6257-43f1-a2f0-1ec07bf215ad","Type":"ContainerStarted","Data":"a075c313dd476af7b38cb47ac46b708c29575c832eae7c3eca5e503e0f1927ee"}
Nov 02 15:12:11.060938 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:12:11.060497    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-xcfbs" event={"ID":"a395ec29-6257-43f1-a2f0-1ec07bf215ad","Type":"ContainerStarted","Data":"ad18961e6c5af7315acd59e55cc42053649a1abb7945dba189403e58cfbe2af5"}
Nov 02 15:12:11.120738 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:12:11.120676    5304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-xcfbs" podStartSLOduration=2.120659865 podStartE2EDuration="2.120659865s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:12:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:12:11.1173929 +0000 UTC m=+931.111217310" watchObservedRunningTime="2026-11-02 15:12:11.120659865 +0000 UTC m=+931.114484273"
Nov 02 15:12:11.176333 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:12:11.176297    5304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-xcfbs"
Nov 02 15:12:11.177797 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:12:11.177772    5304 patch_prober.go:29] interesting pod/e2e-worker-w0-6af50f-openshift-default-8545fb68f-xcfbs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Nov 02 15:12:11.177903 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:12:11.177825    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-xcfbs" podUID="a395ec29-6257-43f1-a2f0-1ec07bf215ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Nov 02 15:12:12.176011 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:12:12.175980    5304 patch_prober.go:29] interesting pod/e2e-worker-w0-6af50f-openshift-default-8545fb68f-xcfbs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Nov 02 15:12:12.176443 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:12:12.176031    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-xcfbs" podUID="a395ec29-6257-43f1-a2f0-1ec07bf215ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Nov 02 15:12:13.176238 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:12:13.176193    5304 patch_prober.go:29] interesting pod/e2e-worker-w0-6af50f-openshift-default-8545fb68f-xcfbs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Nov 02 15:12:13.176638 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:12:13.176255    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-xcfbs" podUID="a395ec29-6257-43f1-a2f0-1ec07bf215ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Nov 02 15:12:14.176357 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:12:14.176323    5304 patch_prober.go:29] interesting pod/e2e-worker-w0-6af50f-openshift-default-8545fb68f-xcfbs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Nov 02 15:12:14.176757 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:12:14.176376    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-xcfbs" podUID="a395ec29-6257-43f1-a2f0-1ec07bf215ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Nov 02 15:12:15.176608 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:12:15.176572    5304 patch_prober.go:29] interesting pod/e2e-worker-w0-6af50f-openshift-default-8545fb68f-xcfbs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Nov 02 15:12:15.177224 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:12:15.177191    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-xcfbs" podUID="a395ec29-6257-43f1-a2f0-1ec07bf215ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Nov 02 15:12:16.216116 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:12:16.216076    5304 patch_prober.go:29] interesting pod/e2e-worker-w0-6af50f-openshift-default-8545fb68f-xcfbs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Nov 02 15:12:16.219249 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:12:16.219204    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-xcfbs" podUID="a395ec29-6257-43f1-a2f0-1ec07bf215ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Nov 02 15:12:17.029744 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:12:17.029697    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6af50f/distinct-w0-6af50f-kserve-c6d994647-hbj9z" podUID="3dca5bf3-8e08-47a0-936e-205670d59ede" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:12:17.203165 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:12:17.091199    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nkbrk" podUID="1f68d837-fff2-418b-84d7-c2d7eb21a57e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.24:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:12:17.203165 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:12:17.091360    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1cc1b2/distinct-w5-1cc1b2-kserve-5b7459bc6d-zl8xr" podUID="211ad15d-9234-4f57-b1b9-181dd2d74208" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded"
Nov 02 15:12:17.203165 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:12:17.091571    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-624254/simulator-w3-624254-kserve-56866b7699-z47sd" podUID="345c710f-b414-4e62-86c7-d51ff4e5be15" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": context deadline exceeded"
Nov 02 15:12:17.204190 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:12:17.091595    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1cc1b2/premium-w5-1cc1b2-kserve-58db46f97c-cgdqg" podUID="210f620a-8ae7-4e4c-acc3-46f75ddc6105" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded"
Nov 02 15:12:17.205157 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:12:17.096395    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5dd8e9/simulator-w1-5dd8e9-kserve-6f9f755d6f-gnw5d" podUID="66b75e64-cb24-40be-9076-4984c132e208" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:12:17.205157 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:12:17.123825    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7ecfe1/distinct-w4-7ecfe1-kserve-6c847c647d-d2jvv" podUID="f1f1ca5e-cca7-4c54-939f-06f095defcdd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:12:17.205157 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:12:17.146811    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-739bf1/simulator-w2-739bf1-kserve-7c74f85465-ltk4t" podUID="cf6a1ded-28a3-49eb-878b-b4ad77a0bb5c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:12:17.205157 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:12:17.192740    5304 patch_prober.go:29] interesting pod/e2e-worker-w0-6af50f-openshift-default-8545fb68f-xcfbs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Nov 02 15:12:17.205157 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:12:17.204882    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-xcfbs" podUID="a395ec29-6257-43f1-a2f0-1ec07bf215ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Nov 02 15:12:17.216258 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:12:17.216219    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5dd8e9/premium-w1-5dd8e9-kserve-67588456fb-j6lph" podUID="4ccc607a-14e4-43ca-8999-78100d4b799f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:12:17.275163 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:12:17.272852    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5113c2/premium-w6-5113c2-kserve-b84945cb5-pc6f8" podUID="9fb3e992-7bb0-497a-a82c-6e396ba241c1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:12:17.598591 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:12:17.598541    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-624254/distinct-w3-624254-kserve-5465dc4765-dmq67" podUID="1a5aa910-a6de-417b-8638-81e4ec5c1825" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded"
Nov 02 15:12:18.073450 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:12:18.073344    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-739bf1/distinct-2-w2-739bf1-kserve-5c58479c48-qgfkh" podUID="9746bf8a-91bd-4715-a02e-ae2e651ca54d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:12:18.124395 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:12:18.124357    5304 patch_prober.go:29] interesting pod/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-25flx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 15:12:18.157151 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:12:18.132231    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-25flx" podUID="6f80b62a-49ba-4922-8f3e-804d6bdcc46f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:12:18.203273 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:12:18.203222    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1cc1b2/distinct-w5-1cc1b2-kserve-5b7459bc6d-zl8xr" podUID="211ad15d-9234-4f57-b1b9-181dd2d74208" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded"
Nov 02 15:12:18.215302 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:12:18.210163    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6af50f/distinct-w0-6af50f-kserve-c6d994647-hbj9z" podUID="3dca5bf3-8e08-47a0-936e-205670d59ede" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded"
Nov 02 15:12:18.228296 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:12:18.226533    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5dd8e9/premium-w1-5dd8e9-kserve-67588456fb-j6lph" podUID="4ccc607a-14e4-43ca-8999-78100d4b799f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded"
Nov 02 15:12:18.330215 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:12:18.226583    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nkbrk" podUID="1f68d837-fff2-418b-84d7-c2d7eb21a57e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.24:8000/ready\": context deadline exceeded"
Nov 02 15:12:18.330215 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:12:18.226611    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1cc1b2/premium-w5-1cc1b2-kserve-58db46f97c-cgdqg" podUID="210f620a-8ae7-4e4c-acc3-46f75ddc6105" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:12:18.330215 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:12:18.227963    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-624254/simulator-w3-624254-kserve-56866b7699-z47sd" podUID="345c710f-b414-4e62-86c7-d51ff4e5be15" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:12:18.330215 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:12:18.228731    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7ecfe1/distinct-w4-7ecfe1-kserve-6c847c647d-d2jvv" podUID="f1f1ca5e-cca7-4c54-939f-06f095defcdd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded"
Nov 02 15:12:18.330215 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:12:18.228758    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5dd8e9/simulator-w1-5dd8e9-kserve-6f9f755d6f-gnw5d" podUID="66b75e64-cb24-40be-9076-4984c132e208" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": context deadline exceeded"
Nov 02 15:12:18.330215 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:12:18.229578    5304 patch_prober.go:29] interesting pod/e2e-worker-w0-6af50f-openshift-default-8545fb68f-xcfbs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Nov 02 15:12:18.330215 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:12:18.312049    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-xcfbs" podUID="a395ec29-6257-43f1-a2f0-1ec07bf215ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Nov 02 15:12:18.330215 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:12:18.304247    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-739bf1/simulator-w2-739bf1-kserve-7c74f85465-ltk4t" podUID="cf6a1ded-28a3-49eb-878b-b4ad77a0bb5c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:12:18.330215 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:12:18.286241    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5113c2/premium-w6-5113c2-kserve-b84945cb5-pc6f8" podUID="9fb3e992-7bb0-497a-a82c-6e396ba241c1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:12:18.427096 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:12:18.427056    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-1cc1b2/distinct-w5-1cc1b2-kserve-5b7459bc6d-zl8xr"
Nov 02 15:12:18.427434 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:12:18.427412    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-6af50f/distinct-w0-6af50f-kserve-c6d994647-hbj9z"
Nov 02 15:12:18.427593 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:12:18.427579    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5dd8e9/premium-w1-5dd8e9-kserve-67588456fb-j6lph"
Nov 02 15:12:18.453152 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:12:18.427722    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nkbrk"
Nov 02 15:12:18.614631 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:12:18.614547    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-1cc1b2/premium-w5-1cc1b2-kserve-58db46f97c-cgdqg"
Nov 02 15:12:18.614900 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:12:18.614881    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-624254/simulator-w3-624254-kserve-56866b7699-z47sd"
Nov 02 15:12:18.615056 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:12:18.615041    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-7ecfe1/distinct-w4-7ecfe1-kserve-6c847c647d-d2jvv"
Nov 02 15:12:18.615214 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:12:18.615201    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5dd8e9/simulator-w1-5dd8e9-kserve-6f9f755d6f-gnw5d"
Nov 02 15:12:18.615425 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:12:18.615410    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-739bf1/simulator-w2-739bf1-kserve-7c74f85465-ltk4t"
Nov 02 15:12:18.615551 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:12:18.615540    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-5113c2/premium-w6-5113c2-kserve-b84945cb5-pc6f8"
Nov 02 15:12:18.616005 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:12:18.612774    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-624254/distinct-w3-624254-kserve-5465dc4765-dmq67" podUID="1a5aa910-a6de-417b-8638-81e4ec5c1825" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:12:18.643151 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:12:18.617804    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-624254/distinct-w3-624254-kserve-5465dc4765-dmq67"
Nov 02 15:12:18.907210 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:12:18.791497    5304 patch_prober.go:29] interesting pod/e2e-worker-w0-6af50f-openshift-default-8545fb68f-mp9v6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 15:12:18.907472 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:12:18.907440    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-mp9v6" podUID="1786cd64-e217-4f69-bcb6-6bda8aace53d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:12:18.908590 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:12:18.820216    5304 patch_prober.go:29] interesting pod/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-9gswz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 15:12:18.908766 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:12:18.908742    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-9gswz" podUID="45f29144-431b-4948-abfa-a1fc5a6847ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:12:18.925608 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:12:18.925566    5304 patch_prober.go:29] interesting pod/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-k2p6m container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 15:12:19.382353 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:12:19.382306    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-k2p6m" podUID="1bd23848-eb1d-45fd-9d87-c11302b3c111" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:12:19.393249 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:12:19.307762    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1cc1b2/distinct-w5-1cc1b2-kserve-5b7459bc6d-zl8xr" podUID="211ad15d-9234-4f57-b1b9-181dd2d74208" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:12:19.393908 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:12:19.322212    5304 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-5dd8e9/simulator-w1-5dd8e9-kserve-6f9f755d6f-gnw5d" podUID="66b75e64-cb24-40be-9076-4984c132e208" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:12:19.394076 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:12:19.329185    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6af50f/distinct-w0-6af50f-kserve-c6d994647-hbj9z" podUID="3dca5bf3-8e08-47a0-936e-205670d59ede" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:12:19.394373 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:12:19.349772    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5dd8e9/simulator-w1-5dd8e9-kserve-6f9f755d6f-gnw5d" podUID="66b75e64-cb24-40be-9076-4984c132e208" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:12:19.431953 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:12:19.350258    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-739bf1/simulator-w2-739bf1-kserve-7c74f85465-ltk4t" podUID="cf6a1ded-28a3-49eb-878b-b4ad77a0bb5c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:12:19.678399 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:12:19.350289    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5113c2/premium-w6-5113c2-kserve-b84945cb5-pc6f8" podUID="9fb3e992-7bb0-497a-a82c-6e396ba241c1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:12:19.678829 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:12:19.350332    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7ecfe1/distinct-w4-7ecfe1-kserve-6c847c647d-d2jvv" podUID="f1f1ca5e-cca7-4c54-939f-06f095defcdd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:12:19.678946 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:12:19.353073    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5dd8e9/premium-w1-5dd8e9-kserve-67588456fb-j6lph" podUID="4ccc607a-14e4-43ca-8999-78100d4b799f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded"
Nov 02 15:12:19.681160 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:12:19.353103    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nkbrk" podUID="1f68d837-fff2-418b-84d7-c2d7eb21a57e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.24:8000/ready\": context deadline exceeded"
Nov 02 15:12:19.681577 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:12:19.353144    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1cc1b2/premium-w5-1cc1b2-kserve-58db46f97c-cgdqg" podUID="210f620a-8ae7-4e4c-acc3-46f75ddc6105" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded"
Nov 02 15:12:19.681876 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:12:19.353167    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-624254/simulator-w3-624254-kserve-56866b7699-z47sd" podUID="345c710f-b414-4e62-86c7-d51ff4e5be15" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": context deadline exceeded"
Nov 02 15:12:19.682173 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:12:19.128651    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-739bf1/distinct-2-w2-739bf1-kserve-5c58479c48-qgfkh" podUID="9746bf8a-91bd-4715-a02e-ae2e651ca54d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:12:19.702144 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:12:19.355490    5304 patch_prober.go:29] interesting pod/e2e-worker-w0-6af50f-openshift-default-8545fb68f-xcfbs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Nov 02 15:12:19.702144 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:12:19.694007    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-xcfbs" podUID="a395ec29-6257-43f1-a2f0-1ec07bf215ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Nov 02 15:12:19.702144 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:12:19.693539    5304 patch_prober.go:29] interesting pod/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-qqqdn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 15:12:19.702144 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:12:19.694065    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-qqqdn" podUID="59fee359-49bd-4a5f-b5da-e9e22c9328f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:12:19.702144 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:12:19.694182    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-739bf1/distinct-2-w2-739bf1-kserve-5c58479c48-qgfkh"
Nov 02 15:12:19.702144 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:12:19.631260    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-624254/distinct-w3-624254-kserve-5465dc4765-dmq67" podUID="1a5aa910-a6de-417b-8638-81e4ec5c1825" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:12:19.796435 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:12:19.796372    5304 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-624254/distinct-w3-624254-kserve-5465dc4765-dmq67" podUID="1a5aa910-a6de-417b-8638-81e4ec5c1825" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:12:19.850850 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:12:19.850812    5304 patch_prober.go:29] interesting pod/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-k9kqx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 15:12:19.851178 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:12:19.851144    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-k9kqx" podUID="7f0b208d-d18a-4a38-952c-4a1abb1b30a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:12:19.941229 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:12:19.928223    5304 patch_prober.go:29] interesting pod/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-64phg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 15:12:19.941229 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:12:19.928283    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-64phg" podUID="e60cf723-3d4a-4bcc-bc72-0c984eba466b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:12:20.188926 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:12:20.188891    5304 patch_prober.go:29] interesting pod/e2e-worker-w0-6af50f-openshift-default-8545fb68f-xcfbs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Nov 02 15:12:20.189264 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:12:20.189217    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-xcfbs" podUID="a395ec29-6257-43f1-a2f0-1ec07bf215ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Nov 02 15:12:20.438107 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:12:20.438049    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5dd8e9/simulator-w1-5dd8e9-kserve-6f9f755d6f-gnw5d" podUID="66b75e64-cb24-40be-9076-4984c132e208" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:12:20.585151 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:12:20.557343    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1cc1b2/distinct-w5-1cc1b2-kserve-5b7459bc6d-zl8xr" podUID="211ad15d-9234-4f57-b1b9-181dd2d74208" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:12:20.714207 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:12:20.566278    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6af50f/distinct-w0-6af50f-kserve-c6d994647-hbj9z" podUID="3dca5bf3-8e08-47a0-936e-205670d59ede" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:12:20.758829 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:12:20.758781    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-624254/simulator-w3-624254-kserve-56866b7699-z47sd" podUID="345c710f-b414-4e62-86c7-d51ff4e5be15" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": context deadline exceeded"
Nov 02 15:12:20.759711 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:12:20.759673    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1cc1b2/premium-w5-1cc1b2-kserve-58db46f97c-cgdqg" podUID="210f620a-8ae7-4e4c-acc3-46f75ddc6105" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:12:20.912152 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:12:20.911274    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-739bf1/simulator-w2-739bf1-kserve-7c74f85465-ltk4t" podUID="cf6a1ded-28a3-49eb-878b-b4ad77a0bb5c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:12:20.940151 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:12:20.915366    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7ecfe1/distinct-w4-7ecfe1-kserve-6c847c647d-d2jvv" podUID="f1f1ca5e-cca7-4c54-939f-06f095defcdd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:12:20.976146 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:12:20.969277    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-739bf1/distinct-2-w2-739bf1-kserve-5c58479c48-qgfkh" podUID="9746bf8a-91bd-4715-a02e-ae2e651ca54d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded"
Nov 02 15:12:20.976146 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:12:20.969740    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-624254/distinct-w3-624254-kserve-5465dc4765-dmq67" podUID="1a5aa910-a6de-417b-8638-81e4ec5c1825" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:12:21.106162 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:12:21.072552    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nkbrk" podUID="1f68d837-fff2-418b-84d7-c2d7eb21a57e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.24:8000/ready\": context deadline exceeded"
Nov 02 15:12:21.134536 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:12:21.134487    5304 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-1cc1b2/premium-w5-1cc1b2-kserve-58db46f97c-cgdqg" podUID="210f620a-8ae7-4e4c-acc3-46f75ddc6105" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/health\": context deadline exceeded"
Nov 02 15:12:21.164152 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:12:21.153705    5304 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.043s"
Nov 02 15:12:21.189809 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:12:21.181381    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5113c2/premium-w6-5113c2-kserve-b84945cb5-pc6f8" podUID="9fb3e992-7bb0-497a-a82c-6e396ba241c1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:12:21.192878 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:12:21.192823    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5dd8e9/premium-w1-5dd8e9-kserve-67588456fb-j6lph" podUID="4ccc607a-14e4-43ca-8999-78100d4b799f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded"
Nov 02 15:12:21.202411 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:12:21.201271    5304 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-6af50f/distinct-w0-6af50f-kserve-c6d994647-hbj9z" podUID="3dca5bf3-8e08-47a0-936e-205670d59ede" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/health\": context deadline exceeded"
Nov 02 15:12:21.202411 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:12:21.201433    5304 patch_prober.go:29] interesting pod/e2e-worker-w0-6af50f-openshift-default-8545fb68f-xcfbs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Nov 02 15:12:21.202411 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:12:21.201476    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-xcfbs" podUID="a395ec29-6257-43f1-a2f0-1ec07bf215ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Nov 02 15:12:21.512151 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:12:21.469337    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5dd8e9/simulator-w1-5dd8e9-kserve-6f9f755d6f-gnw5d" podUID="66b75e64-cb24-40be-9076-4984c132e208" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": context deadline exceeded"
Nov 02 15:12:21.741291 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:12:21.741261    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-624254/distinct-w3-624254-kserve-5465dc4765-dmq67"
Nov 02 15:12:21.784909 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:12:21.773245    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1cc1b2/premium-w5-1cc1b2-kserve-58db46f97c-cgdqg" podUID="210f620a-8ae7-4e4c-acc3-46f75ddc6105" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:12:21.790236 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:12:21.790193    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-624254/simulator-w3-624254-kserve-56866b7699-z47sd" podUID="345c710f-b414-4e62-86c7-d51ff4e5be15" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": context deadline exceeded"
Nov 02 15:12:21.791319 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:12:21.791182    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5dd8e9/premium-w1-5dd8e9-kserve-67588456fb-j6lph"
Nov 02 15:12:21.801203 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:12:21.793957    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6af50f/distinct-w0-6af50f-kserve-c6d994647-hbj9z" podUID="3dca5bf3-8e08-47a0-936e-205670d59ede" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:12:21.802453 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:12:21.802426    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1cc1b2/distinct-w5-1cc1b2-kserve-5b7459bc6d-zl8xr" podUID="211ad15d-9234-4f57-b1b9-181dd2d74208" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded"
Nov 02 15:12:21.911622 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:12:21.911574    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-739bf1/simulator-w2-739bf1-kserve-7c74f85465-ltk4t" podUID="cf6a1ded-28a3-49eb-878b-b4ad77a0bb5c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded"
Nov 02 15:12:21.921396 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:12:21.921341    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7ecfe1/distinct-w4-7ecfe1-kserve-6c847c647d-d2jvv" podUID="f1f1ca5e-cca7-4c54-939f-06f095defcdd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:12:21.936941 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:12:21.936916    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-739bf1/simulator-w2-739bf1-kserve-7c74f85465-ltk4t"
Nov 02 15:12:21.998150 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:12:21.969640    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-5113c2/premium-w6-5113c2-kserve-b84945cb5-pc6f8"
Nov 02 15:12:21.998150 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:12:21.970871    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-624254/simulator-w3-624254-kserve-56866b7699-z47sd"
Nov 02 15:12:22.000863 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:12:21.998657    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nkbrk"
Nov 02 15:12:22.002700 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:12:22.002675    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-1cc1b2/premium-w5-1cc1b2-kserve-58db46f97c-cgdqg"
Nov 02 15:12:22.003716 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:12:22.003696    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-6af50f/distinct-w0-6af50f-kserve-c6d994647-hbj9z"
Nov 02 15:12:22.006564 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:12:22.006544    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-7ecfe1/distinct-w4-7ecfe1-kserve-6c847c647d-d2jvv"
Nov 02 15:12:22.012243 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:12:22.012221    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-739bf1/distinct-2-w2-739bf1-kserve-5c58479c48-qgfkh"
Nov 02 15:12:22.044343 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:12:22.044242    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5dd8e9/simulator-w1-5dd8e9-kserve-6f9f755d6f-gnw5d"
Nov 02 15:12:22.053146 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:12:22.053102    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-1cc1b2/distinct-w5-1cc1b2-kserve-5b7459bc6d-zl8xr"
Nov 02 15:12:22.687322 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:12:22.687290    5304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-xcfbs"
Nov 02 15:12:22.687822 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:12:22.687735    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-xcfbs"
Nov 02 15:12:22.689571 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:12:22.689548    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-xcfbs"
Nov 02 15:12:24.239518 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:12:24.239486    5304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-0621b336-gw-openshift-default-68bdbfb8f7-2j6jv"]
Nov 02 15:12:24.244246 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:12:24.244217    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-0621b336-gw-openshift-default-68bdbfb8f7-2j6jv"
Nov 02 15:12:24.260292 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:12:24.260264    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-0621b336-gw-openshift-default-68bdbfb8f7-2j6jv"]
Nov 02 15:12:24.277169 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:12:24.277116    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39bb7834-0bb3-4dc2-a658-a7d527a7aa1f-istio-data\") pod \"e2e-migrate-0621b336-gw-openshift-default-68bdbfb8f7-2j6jv\" (UID: \"39bb7834-0bb3-4dc2-a658-a7d527a7aa1f\") " pod="openshift-ingress/e2e-migrate-0621b336-gw-openshift-default-68bdbfb8f7-2j6jv"
Nov 02 15:12:24.277322 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:12:24.277229    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rv4rz\" (UniqueName: \"kubernetes.io/projected/39bb7834-0bb3-4dc2-a658-a7d527a7aa1f-kube-api-access-rv4rz\") pod \"e2e-migrate-0621b336-gw-openshift-default-68bdbfb8f7-2j6jv\" (UID: \"39bb7834-0bb3-4dc2-a658-a7d527a7aa1f\") " pod="openshift-ingress/e2e-migrate-0621b336-gw-openshift-default-68bdbfb8f7-2j6jv"
Nov 02 15:12:24.277383 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:12:24.277371    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39bb7834-0bb3-4dc2-a658-a7d527a7aa1f-workload-socket\") pod \"e2e-migrate-0621b336-gw-openshift-default-68bdbfb8f7-2j6jv\" (UID: \"39bb7834-0bb3-4dc2-a658-a7d527a7aa1f\") " pod="openshift-ingress/e2e-migrate-0621b336-gw-openshift-default-68bdbfb8f7-2j6jv"
Nov 02 15:12:24.277435 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:12:24.277414    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39bb7834-0bb3-4dc2-a658-a7d527a7aa1f-istio-envoy\") pod \"e2e-migrate-0621b336-gw-openshift-default-68bdbfb8f7-2j6jv\" (UID: \"39bb7834-0bb3-4dc2-a658-a7d527a7aa1f\") " pod="openshift-ingress/e2e-migrate-0621b336-gw-openshift-default-68bdbfb8f7-2j6jv"
Nov 02 15:12:24.277490 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:12:24.277448    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39bb7834-0bb3-4dc2-a658-a7d527a7aa1f-workload-certs\") pod \"e2e-migrate-0621b336-gw-openshift-default-68bdbfb8f7-2j6jv\" (UID: \"39bb7834-0bb3-4dc2-a658-a7d527a7aa1f\") " pod="openshift-ingress/e2e-migrate-0621b336-gw-openshift-default-68bdbfb8f7-2j6jv"
Nov 02 15:12:24.277588 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:12:24.277570    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39bb7834-0bb3-4dc2-a658-a7d527a7aa1f-istiod-ca-cert\") pod \"e2e-migrate-0621b336-gw-openshift-default-68bdbfb8f7-2j6jv\" (UID: \"39bb7834-0bb3-4dc2-a658-a7d527a7aa1f\") " pod="openshift-ingress/e2e-migrate-0621b336-gw-openshift-default-68bdbfb8f7-2j6jv"
Nov 02 15:12:24.277651 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:12:24.277635    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39bb7834-0bb3-4dc2-a658-a7d527a7aa1f-credential-socket\") pod \"e2e-migrate-0621b336-gw-openshift-default-68bdbfb8f7-2j6jv\" (UID: \"39bb7834-0bb3-4dc2-a658-a7d527a7aa1f\") " pod="openshift-ingress/e2e-migrate-0621b336-gw-openshift-default-68bdbfb8f7-2j6jv"
Nov 02 15:12:24.277751 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:12:24.277734    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39bb7834-0bb3-4dc2-a658-a7d527a7aa1f-istio-token\") pod \"e2e-migrate-0621b336-gw-openshift-default-68bdbfb8f7-2j6jv\" (UID: \"39bb7834-0bb3-4dc2-a658-a7d527a7aa1f\") " pod="openshift-ingress/e2e-migrate-0621b336-gw-openshift-default-68bdbfb8f7-2j6jv"
Nov 02 15:12:24.277832 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:12:24.277814    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39bb7834-0bb3-4dc2-a658-a7d527a7aa1f-istio-podinfo\") pod \"e2e-migrate-0621b336-gw-openshift-default-68bdbfb8f7-2j6jv\" (UID: \"39bb7834-0bb3-4dc2-a658-a7d527a7aa1f\") " pod="openshift-ingress/e2e-migrate-0621b336-gw-openshift-default-68bdbfb8f7-2j6jv"
Nov 02 15:12:24.280675 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:12:24.280654    5304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-0621b336-gw-openshift-default-dockercfg-dflsh\""
Nov 02 15:12:24.391151 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:12:24.390375    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39bb7834-0bb3-4dc2-a658-a7d527a7aa1f-workload-socket\") pod \"e2e-migrate-0621b336-gw-openshift-default-68bdbfb8f7-2j6jv\" (UID: \"39bb7834-0bb3-4dc2-a658-a7d527a7aa1f\") " pod="openshift-ingress/e2e-migrate-0621b336-gw-openshift-default-68bdbfb8f7-2j6jv"
Nov 02 15:12:24.391151 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:12:24.390454    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39bb7834-0bb3-4dc2-a658-a7d527a7aa1f-istio-envoy\") pod \"e2e-migrate-0621b336-gw-openshift-default-68bdbfb8f7-2j6jv\" (UID: \"39bb7834-0bb3-4dc2-a658-a7d527a7aa1f\") " pod="openshift-ingress/e2e-migrate-0621b336-gw-openshift-default-68bdbfb8f7-2j6jv"
Nov 02 15:12:24.391151 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:12:24.390490    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39bb7834-0bb3-4dc2-a658-a7d527a7aa1f-workload-certs\") pod \"e2e-migrate-0621b336-gw-openshift-default-68bdbfb8f7-2j6jv\" (UID: \"39bb7834-0bb3-4dc2-a658-a7d527a7aa1f\") " pod="openshift-ingress/e2e-migrate-0621b336-gw-openshift-default-68bdbfb8f7-2j6jv"
Nov 02 15:12:24.391151 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:12:24.390552    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39bb7834-0bb3-4dc2-a658-a7d527a7aa1f-istiod-ca-cert\") pod \"e2e-migrate-0621b336-gw-openshift-default-68bdbfb8f7-2j6jv\" (UID: \"39bb7834-0bb3-4dc2-a658-a7d527a7aa1f\") " pod="openshift-ingress/e2e-migrate-0621b336-gw-openshift-default-68bdbfb8f7-2j6jv"
Nov 02 15:12:24.391151 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:12:24.390606    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39bb7834-0bb3-4dc2-a658-a7d527a7aa1f-credential-socket\") pod \"e2e-migrate-0621b336-gw-openshift-default-68bdbfb8f7-2j6jv\" (UID: \"39bb7834-0bb3-4dc2-a658-a7d527a7aa1f\") " pod="openshift-ingress/e2e-migrate-0621b336-gw-openshift-default-68bdbfb8f7-2j6jv"
Nov 02 15:12:24.391151 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:12:24.390672    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39bb7834-0bb3-4dc2-a658-a7d527a7aa1f-istio-token\") pod \"e2e-migrate-0621b336-gw-openshift-default-68bdbfb8f7-2j6jv\" (UID: \"39bb7834-0bb3-4dc2-a658-a7d527a7aa1f\") " pod="openshift-ingress/e2e-migrate-0621b336-gw-openshift-default-68bdbfb8f7-2j6jv"
Nov 02 15:12:24.391151 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:12:24.390778    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39bb7834-0bb3-4dc2-a658-a7d527a7aa1f-istio-podinfo\") pod \"e2e-migrate-0621b336-gw-openshift-default-68bdbfb8f7-2j6jv\" (UID: \"39bb7834-0bb3-4dc2-a658-a7d527a7aa1f\") " pod="openshift-ingress/e2e-migrate-0621b336-gw-openshift-default-68bdbfb8f7-2j6jv"
Nov 02 15:12:24.391151 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:12:24.390885    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39bb7834-0bb3-4dc2-a658-a7d527a7aa1f-istio-data\") pod \"e2e-migrate-0621b336-gw-openshift-default-68bdbfb8f7-2j6jv\" (UID: \"39bb7834-0bb3-4dc2-a658-a7d527a7aa1f\") " pod="openshift-ingress/e2e-migrate-0621b336-gw-openshift-default-68bdbfb8f7-2j6jv"
Nov 02 15:12:24.391151 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:12:24.390921    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rv4rz\" (UniqueName: \"kubernetes.io/projected/39bb7834-0bb3-4dc2-a658-a7d527a7aa1f-kube-api-access-rv4rz\") pod \"e2e-migrate-0621b336-gw-openshift-default-68bdbfb8f7-2j6jv\" (UID: \"39bb7834-0bb3-4dc2-a658-a7d527a7aa1f\") " pod="openshift-ingress/e2e-migrate-0621b336-gw-openshift-default-68bdbfb8f7-2j6jv"
Nov 02 15:12:24.395189 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:12:24.391717    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39bb7834-0bb3-4dc2-a658-a7d527a7aa1f-workload-socket\") pod \"e2e-migrate-0621b336-gw-openshift-default-68bdbfb8f7-2j6jv\" (UID: \"39bb7834-0bb3-4dc2-a658-a7d527a7aa1f\") " pod="openshift-ingress/e2e-migrate-0621b336-gw-openshift-default-68bdbfb8f7-2j6jv"
Nov 02 15:12:24.415472 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:12:24.411536    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39bb7834-0bb3-4dc2-a658-a7d527a7aa1f-workload-certs\") pod \"e2e-migrate-0621b336-gw-openshift-default-68bdbfb8f7-2j6jv\" (UID: \"39bb7834-0bb3-4dc2-a658-a7d527a7aa1f\") " pod="openshift-ingress/e2e-migrate-0621b336-gw-openshift-default-68bdbfb8f7-2j6jv"
Nov 02 15:12:24.415472 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:12:24.412430    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39bb7834-0bb3-4dc2-a658-a7d527a7aa1f-istiod-ca-cert\") pod \"e2e-migrate-0621b336-gw-openshift-default-68bdbfb8f7-2j6jv\" (UID: \"39bb7834-0bb3-4dc2-a658-a7d527a7aa1f\") " pod="openshift-ingress/e2e-migrate-0621b336-gw-openshift-default-68bdbfb8f7-2j6jv"
Nov 02 15:12:24.434202 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:12:24.427497    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39bb7834-0bb3-4dc2-a658-a7d527a7aa1f-credential-socket\") pod \"e2e-migrate-0621b336-gw-openshift-default-68bdbfb8f7-2j6jv\" (UID: \"39bb7834-0bb3-4dc2-a658-a7d527a7aa1f\") " pod="openshift-ingress/e2e-migrate-0621b336-gw-openshift-default-68bdbfb8f7-2j6jv"
Nov 02 15:12:24.473826 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:12:24.465551    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39bb7834-0bb3-4dc2-a658-a7d527a7aa1f-istio-data\") pod \"e2e-migrate-0621b336-gw-openshift-default-68bdbfb8f7-2j6jv\" (UID: \"39bb7834-0bb3-4dc2-a658-a7d527a7aa1f\") " pod="openshift-ingress/e2e-migrate-0621b336-gw-openshift-default-68bdbfb8f7-2j6jv"
Nov 02 15:12:24.478176 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:12:24.476108    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rv4rz\" (UniqueName: \"kubernetes.io/projected/39bb7834-0bb3-4dc2-a658-a7d527a7aa1f-kube-api-access-rv4rz\") pod \"e2e-migrate-0621b336-gw-openshift-default-68bdbfb8f7-2j6jv\" (UID: \"39bb7834-0bb3-4dc2-a658-a7d527a7aa1f\") " pod="openshift-ingress/e2e-migrate-0621b336-gw-openshift-default-68bdbfb8f7-2j6jv"
Nov 02 15:12:24.478176 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:12:24.476504    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39bb7834-0bb3-4dc2-a658-a7d527a7aa1f-istio-envoy\") pod \"e2e-migrate-0621b336-gw-openshift-default-68bdbfb8f7-2j6jv\" (UID: \"39bb7834-0bb3-4dc2-a658-a7d527a7aa1f\") " pod="openshift-ingress/e2e-migrate-0621b336-gw-openshift-default-68bdbfb8f7-2j6jv"
Nov 02 15:12:24.481205 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:12:24.478894    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39bb7834-0bb3-4dc2-a658-a7d527a7aa1f-istio-podinfo\") pod \"e2e-migrate-0621b336-gw-openshift-default-68bdbfb8f7-2j6jv\" (UID: \"39bb7834-0bb3-4dc2-a658-a7d527a7aa1f\") " pod="openshift-ingress/e2e-migrate-0621b336-gw-openshift-default-68bdbfb8f7-2j6jv"
Nov 02 15:12:24.505591 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:12:24.498526    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39bb7834-0bb3-4dc2-a658-a7d527a7aa1f-istio-token\") pod \"e2e-migrate-0621b336-gw-openshift-default-68bdbfb8f7-2j6jv\" (UID: \"39bb7834-0bb3-4dc2-a658-a7d527a7aa1f\") " pod="openshift-ingress/e2e-migrate-0621b336-gw-openshift-default-68bdbfb8f7-2j6jv"
Nov 02 15:12:24.558587 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:12:24.558550    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-0621b336-gw-openshift-default-68bdbfb8f7-2j6jv"
Nov 02 15:12:24.814270 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:12:24.814182    5304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k6fjk" containerName="extractor"
Nov 02 15:12:24.962463 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:12:24.961687    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-0621b336-gw-openshift-default-68bdbfb8f7-2j6jv"]
Nov 02 15:12:24.966270 ip-10-0-135-138 kubenswrapper[5304]: W1102 15:12:24.966218    5304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39bb7834_0bb3_4dc2_a658_a7d527a7aa1f.slice/crio-bee6ec6e928e5b2863ea8f16c6c6e3ca136031fd8bee2f38ae9d32391b86016f WatchSource:0}: Error finding container bee6ec6e928e5b2863ea8f16c6c6e3ca136031fd8bee2f38ae9d32391b86016f: Status 404 returned error can't find the container with id bee6ec6e928e5b2863ea8f16c6c6e3ca136031fd8bee2f38ae9d32391b86016f
Nov 02 15:12:24.970753 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:12:24.970715    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:12:24.970889 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:12:24.970805    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:12:24.970889 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:12:24.970853    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:12:25.883083 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:12:25.883044    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-0621b336-gw-openshift-default-68bdbfb8f7-2j6jv" event={"ID":"39bb7834-0bb3-4dc2-a658-a7d527a7aa1f","Type":"ContainerStarted","Data":"636da0f4acfe48a29e82ba5d5c225329cc80f20ab40da23d329e3aaa78483bb9"}
Nov 02 15:12:25.883531 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:12:25.883103    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-0621b336-gw-openshift-default-68bdbfb8f7-2j6jv" event={"ID":"39bb7834-0bb3-4dc2-a658-a7d527a7aa1f","Type":"ContainerStarted","Data":"bee6ec6e928e5b2863ea8f16c6c6e3ca136031fd8bee2f38ae9d32391b86016f"}
Nov 02 15:12:25.948300 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:12:25.948234    5304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-0621b336-gw-openshift-default-68bdbfb8f7-2j6jv" podStartSLOduration=1.948212687 podStartE2EDuration="1.948212687s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:12:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:12:25.944806764 +0000 UTC m=+945.938631176" watchObservedRunningTime="2026-11-02 15:12:25.948212687 +0000 UTC m=+945.942037097"
Nov 02 15:12:26.559367 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:12:26.559330    5304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-0621b336-gw-openshift-default-68bdbfb8f7-2j6jv"
Nov 02 15:12:26.566261 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:12:26.566228    5304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-0621b336-gw-openshift-default-68bdbfb8f7-2j6jv"
Nov 02 15:12:26.889424 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:12:26.889385    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-0621b336-gw-openshift-default-68bdbfb8f7-2j6jv"
Nov 02 15:12:26.890507 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:12:26.890489    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-0621b336-gw-openshift-default-68bdbfb8f7-2j6jv"
Nov 02 15:12:34.670764 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:12:34.670724    5304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-649ad0be-openshift-default-5564b54495-rrtzm"]
Nov 02 15:12:34.676191 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:12:34.676166    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-649ad0be-openshift-default-5564b54495-rrtzm"
Nov 02 15:12:34.721437 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:12:34.721402    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-649ad0be-openshift-default-5564b54495-rrtzm"]
Nov 02 15:12:34.762813 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:12:34.762779    5304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-649ad0be-openshift-default-dockercfg-vt8fc\""
Nov 02 15:12:34.805183 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:12:34.802372    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lfgdq\" (UniqueName: \"kubernetes.io/projected/0f5c1330-dd1a-40d2-8f1f-e91aeedd639f-kube-api-access-lfgdq\") pod \"e2e-ait-649ad0be-openshift-default-5564b54495-rrtzm\" (UID: \"0f5c1330-dd1a-40d2-8f1f-e91aeedd639f\") " pod="openshift-ingress/e2e-ait-649ad0be-openshift-default-5564b54495-rrtzm"
Nov 02 15:12:34.805183 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:12:34.802431    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f5c1330-dd1a-40d2-8f1f-e91aeedd639f-istio-envoy\") pod \"e2e-ait-649ad0be-openshift-default-5564b54495-rrtzm\" (UID: \"0f5c1330-dd1a-40d2-8f1f-e91aeedd639f\") " pod="openshift-ingress/e2e-ait-649ad0be-openshift-default-5564b54495-rrtzm"
Nov 02 15:12:34.805183 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:12:34.802486    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f5c1330-dd1a-40d2-8f1f-e91aeedd639f-istiod-ca-cert\") pod \"e2e-ait-649ad0be-openshift-default-5564b54495-rrtzm\" (UID: \"0f5c1330-dd1a-40d2-8f1f-e91aeedd639f\") " pod="openshift-ingress/e2e-ait-649ad0be-openshift-default-5564b54495-rrtzm"
Nov 02 15:12:34.805183 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:12:34.802534    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f5c1330-dd1a-40d2-8f1f-e91aeedd639f-workload-socket\") pod \"e2e-ait-649ad0be-openshift-default-5564b54495-rrtzm\" (UID: \"0f5c1330-dd1a-40d2-8f1f-e91aeedd639f\") " pod="openshift-ingress/e2e-ait-649ad0be-openshift-default-5564b54495-rrtzm"
Nov 02 15:12:34.805183 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:12:34.802588    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f5c1330-dd1a-40d2-8f1f-e91aeedd639f-workload-certs\") pod \"e2e-ait-649ad0be-openshift-default-5564b54495-rrtzm\" (UID: \"0f5c1330-dd1a-40d2-8f1f-e91aeedd639f\") " pod="openshift-ingress/e2e-ait-649ad0be-openshift-default-5564b54495-rrtzm"
Nov 02 15:12:34.805183 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:12:34.802615    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f5c1330-dd1a-40d2-8f1f-e91aeedd639f-istio-token\") pod \"e2e-ait-649ad0be-openshift-default-5564b54495-rrtzm\" (UID: \"0f5c1330-dd1a-40d2-8f1f-e91aeedd639f\") " pod="openshift-ingress/e2e-ait-649ad0be-openshift-default-5564b54495-rrtzm"
Nov 02 15:12:34.805183 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:12:34.802711    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f5c1330-dd1a-40d2-8f1f-e91aeedd639f-istio-podinfo\") pod \"e2e-ait-649ad0be-openshift-default-5564b54495-rrtzm\" (UID: \"0f5c1330-dd1a-40d2-8f1f-e91aeedd639f\") " pod="openshift-ingress/e2e-ait-649ad0be-openshift-default-5564b54495-rrtzm"
Nov 02 15:12:34.805183 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:12:34.802740    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f5c1330-dd1a-40d2-8f1f-e91aeedd639f-credential-socket\") pod \"e2e-ait-649ad0be-openshift-default-5564b54495-rrtzm\" (UID: \"0f5c1330-dd1a-40d2-8f1f-e91aeedd639f\") " pod="openshift-ingress/e2e-ait-649ad0be-openshift-default-5564b54495-rrtzm"
Nov 02 15:12:34.805183 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:12:34.802904    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f5c1330-dd1a-40d2-8f1f-e91aeedd639f-istio-data\") pod \"e2e-ait-649ad0be-openshift-default-5564b54495-rrtzm\" (UID: \"0f5c1330-dd1a-40d2-8f1f-e91aeedd639f\") " pod="openshift-ingress/e2e-ait-649ad0be-openshift-default-5564b54495-rrtzm"
Nov 02 15:12:34.975670 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:12:34.975553    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f5c1330-dd1a-40d2-8f1f-e91aeedd639f-istio-data\") pod \"e2e-ait-649ad0be-openshift-default-5564b54495-rrtzm\" (UID: \"0f5c1330-dd1a-40d2-8f1f-e91aeedd639f\") " pod="openshift-ingress/e2e-ait-649ad0be-openshift-default-5564b54495-rrtzm"
Nov 02 15:12:34.976087 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:12:34.976054    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lfgdq\" (UniqueName: \"kubernetes.io/projected/0f5c1330-dd1a-40d2-8f1f-e91aeedd639f-kube-api-access-lfgdq\") pod \"e2e-ait-649ad0be-openshift-default-5564b54495-rrtzm\" (UID: \"0f5c1330-dd1a-40d2-8f1f-e91aeedd639f\") " pod="openshift-ingress/e2e-ait-649ad0be-openshift-default-5564b54495-rrtzm"
Nov 02 15:12:34.976335 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:12:34.976317    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f5c1330-dd1a-40d2-8f1f-e91aeedd639f-istio-envoy\") pod \"e2e-ait-649ad0be-openshift-default-5564b54495-rrtzm\" (UID: \"0f5c1330-dd1a-40d2-8f1f-e91aeedd639f\") " pod="openshift-ingress/e2e-ait-649ad0be-openshift-default-5564b54495-rrtzm"
Nov 02 15:12:34.976560 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:12:34.976524    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f5c1330-dd1a-40d2-8f1f-e91aeedd639f-istiod-ca-cert\") pod \"e2e-ait-649ad0be-openshift-default-5564b54495-rrtzm\" (UID: \"0f5c1330-dd1a-40d2-8f1f-e91aeedd639f\") " pod="openshift-ingress/e2e-ait-649ad0be-openshift-default-5564b54495-rrtzm"
Nov 02 15:12:34.976725 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:12:34.976703    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f5c1330-dd1a-40d2-8f1f-e91aeedd639f-workload-socket\") pod \"e2e-ait-649ad0be-openshift-default-5564b54495-rrtzm\" (UID: \"0f5c1330-dd1a-40d2-8f1f-e91aeedd639f\") " pod="openshift-ingress/e2e-ait-649ad0be-openshift-default-5564b54495-rrtzm"
Nov 02 15:12:34.976867 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:12:34.976846    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f5c1330-dd1a-40d2-8f1f-e91aeedd639f-workload-certs\") pod \"e2e-ait-649ad0be-openshift-default-5564b54495-rrtzm\" (UID: \"0f5c1330-dd1a-40d2-8f1f-e91aeedd639f\") " pod="openshift-ingress/e2e-ait-649ad0be-openshift-default-5564b54495-rrtzm"
Nov 02 15:12:34.976976 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:12:34.976963    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f5c1330-dd1a-40d2-8f1f-e91aeedd639f-istio-token\") pod \"e2e-ait-649ad0be-openshift-default-5564b54495-rrtzm\" (UID: \"0f5c1330-dd1a-40d2-8f1f-e91aeedd639f\") " pod="openshift-ingress/e2e-ait-649ad0be-openshift-default-5564b54495-rrtzm"
Nov 02 15:12:34.977194 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:12:34.977176    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f5c1330-dd1a-40d2-8f1f-e91aeedd639f-istio-podinfo\") pod \"e2e-ait-649ad0be-openshift-default-5564b54495-rrtzm\" (UID: \"0f5c1330-dd1a-40d2-8f1f-e91aeedd639f\") " pod="openshift-ingress/e2e-ait-649ad0be-openshift-default-5564b54495-rrtzm"
Nov 02 15:12:34.977317 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:12:34.977303    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f5c1330-dd1a-40d2-8f1f-e91aeedd639f-credential-socket\") pod \"e2e-ait-649ad0be-openshift-default-5564b54495-rrtzm\" (UID: \"0f5c1330-dd1a-40d2-8f1f-e91aeedd639f\") " pod="openshift-ingress/e2e-ait-649ad0be-openshift-default-5564b54495-rrtzm"
Nov 02 15:12:34.977791 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:12:34.977761    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f5c1330-dd1a-40d2-8f1f-e91aeedd639f-workload-socket\") pod \"e2e-ait-649ad0be-openshift-default-5564b54495-rrtzm\" (UID: \"0f5c1330-dd1a-40d2-8f1f-e91aeedd639f\") " pod="openshift-ingress/e2e-ait-649ad0be-openshift-default-5564b54495-rrtzm"
Nov 02 15:12:34.979314 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:12:34.978607    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f5c1330-dd1a-40d2-8f1f-e91aeedd639f-istiod-ca-cert\") pod \"e2e-ait-649ad0be-openshift-default-5564b54495-rrtzm\" (UID: \"0f5c1330-dd1a-40d2-8f1f-e91aeedd639f\") " pod="openshift-ingress/e2e-ait-649ad0be-openshift-default-5564b54495-rrtzm"
Nov 02 15:12:34.979314 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:12:34.976562    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f5c1330-dd1a-40d2-8f1f-e91aeedd639f-istio-data\") pod \"e2e-ait-649ad0be-openshift-default-5564b54495-rrtzm\" (UID: \"0f5c1330-dd1a-40d2-8f1f-e91aeedd639f\") " pod="openshift-ingress/e2e-ait-649ad0be-openshift-default-5564b54495-rrtzm"
Nov 02 15:12:34.979841 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:12:34.979821    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f5c1330-dd1a-40d2-8f1f-e91aeedd639f-workload-certs\") pod \"e2e-ait-649ad0be-openshift-default-5564b54495-rrtzm\" (UID: \"0f5c1330-dd1a-40d2-8f1f-e91aeedd639f\") " pod="openshift-ingress/e2e-ait-649ad0be-openshift-default-5564b54495-rrtzm"
Nov 02 15:12:34.984099 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:12:34.981413    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f5c1330-dd1a-40d2-8f1f-e91aeedd639f-credential-socket\") pod \"e2e-ait-649ad0be-openshift-default-5564b54495-rrtzm\" (UID: \"0f5c1330-dd1a-40d2-8f1f-e91aeedd639f\") " pod="openshift-ingress/e2e-ait-649ad0be-openshift-default-5564b54495-rrtzm"
Nov 02 15:12:34.993152 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:12:34.993073    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f5c1330-dd1a-40d2-8f1f-e91aeedd639f-istio-podinfo\") pod \"e2e-ait-649ad0be-openshift-default-5564b54495-rrtzm\" (UID: \"0f5c1330-dd1a-40d2-8f1f-e91aeedd639f\") " pod="openshift-ingress/e2e-ait-649ad0be-openshift-default-5564b54495-rrtzm"
Nov 02 15:12:34.994211 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:12:34.994150    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f5c1330-dd1a-40d2-8f1f-e91aeedd639f-istio-envoy\") pod \"e2e-ait-649ad0be-openshift-default-5564b54495-rrtzm\" (UID: \"0f5c1330-dd1a-40d2-8f1f-e91aeedd639f\") " pod="openshift-ingress/e2e-ait-649ad0be-openshift-default-5564b54495-rrtzm"
Nov 02 15:12:35.008958 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:12:35.008923    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lfgdq\" (UniqueName: \"kubernetes.io/projected/0f5c1330-dd1a-40d2-8f1f-e91aeedd639f-kube-api-access-lfgdq\") pod \"e2e-ait-649ad0be-openshift-default-5564b54495-rrtzm\" (UID: \"0f5c1330-dd1a-40d2-8f1f-e91aeedd639f\") " pod="openshift-ingress/e2e-ait-649ad0be-openshift-default-5564b54495-rrtzm"
Nov 02 15:12:35.018437 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:12:35.018402    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f5c1330-dd1a-40d2-8f1f-e91aeedd639f-istio-token\") pod \"e2e-ait-649ad0be-openshift-default-5564b54495-rrtzm\" (UID: \"0f5c1330-dd1a-40d2-8f1f-e91aeedd639f\") " pod="openshift-ingress/e2e-ait-649ad0be-openshift-default-5564b54495-rrtzm"
Nov 02 15:12:35.294665 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:12:35.294573    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-649ad0be-openshift-default-5564b54495-rrtzm"
Nov 02 15:12:35.618732 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:12:35.618689    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-649ad0be-openshift-default-5564b54495-rrtzm"]
Nov 02 15:12:35.621488 ip-10-0-135-138 kubenswrapper[5304]: W1102 15:12:35.621451    5304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f5c1330_dd1a_40d2_8f1f_e91aeedd639f.slice/crio-4275003250f60d13cd2ed35204bbe484a9e15f534439055679164371690f7eae WatchSource:0}: Error finding container 4275003250f60d13cd2ed35204bbe484a9e15f534439055679164371690f7eae: Status 404 returned error can't find the container with id 4275003250f60d13cd2ed35204bbe484a9e15f534439055679164371690f7eae
Nov 02 15:12:35.624148 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:12:35.624099    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:12:35.624234 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:12:35.624192    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:12:35.624234 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:12:35.624225    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:12:36.003240 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:12:36.003154    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-649ad0be-openshift-default-5564b54495-rrtzm" event={"ID":"0f5c1330-dd1a-40d2-8f1f-e91aeedd639f","Type":"ContainerStarted","Data":"5e78e646252f977c661e9b3bec6df7ab5f6cb223413c74ee5efc67a31aaadcbf"}
Nov 02 15:12:36.003240 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:12:36.003215    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-649ad0be-openshift-default-5564b54495-rrtzm" event={"ID":"0f5c1330-dd1a-40d2-8f1f-e91aeedd639f","Type":"ContainerStarted","Data":"4275003250f60d13cd2ed35204bbe484a9e15f534439055679164371690f7eae"}
Nov 02 15:12:36.064620 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:12:36.064559    5304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-649ad0be-openshift-default-5564b54495-rrtzm" podStartSLOduration=2.064541612 podStartE2EDuration="2.064541612s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:12:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:12:36.061421334 +0000 UTC m=+956.055245749" watchObservedRunningTime="2026-11-02 15:12:36.064541612 +0000 UTC m=+956.058366021"
Nov 02 15:12:36.295079 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:12:36.294980    5304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-649ad0be-openshift-default-5564b54495-rrtzm"
Nov 02 15:12:36.311294 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:12:36.309408    5304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-649ad0be-openshift-default-5564b54495-rrtzm"
Nov 02 15:12:37.012788 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:12:37.012734    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-649ad0be-openshift-default-5564b54495-rrtzm"
Nov 02 15:12:37.013857 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:12:37.013825    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-649ad0be-openshift-default-5564b54495-rrtzm"
Nov 02 15:12:47.505501 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:12:47.505467    5304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zkvr7"]
Nov 02 15:12:47.510742 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:12:47.510723    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zkvr7"
Nov 02 15:12:47.534900 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:12:47.534871    5304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Nov 02 15:12:47.535049 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:12:47.534954    5304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Nov 02 15:12:47.554067 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:12:47.544309    5304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-rcl8t\""
Nov 02 15:12:47.554067 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:12:47.549374    5304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Nov 02 15:12:47.563182 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:12:47.563149    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zkvr7"]
Nov 02 15:12:47.673309 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:12:47.673275    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f967e0cd-2c0f-42ad-9163-2786089f3ca9-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zkvr7\" (UID: \"f967e0cd-2c0f-42ad-9163-2786089f3ca9\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zkvr7"
Nov 02 15:12:47.673516 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:12:47.673345    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f967e0cd-2c0f-42ad-9163-2786089f3ca9-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zkvr7\" (UID: \"f967e0cd-2c0f-42ad-9163-2786089f3ca9\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zkvr7"
Nov 02 15:12:47.673516 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:12:47.673461    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/f967e0cd-2c0f-42ad-9163-2786089f3ca9-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zkvr7\" (UID: \"f967e0cd-2c0f-42ad-9163-2786089f3ca9\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zkvr7"
Nov 02 15:12:47.673639 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:12:47.673552    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sh7jk\" (UniqueName: \"kubernetes.io/projected/f967e0cd-2c0f-42ad-9163-2786089f3ca9-kube-api-access-sh7jk\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zkvr7\" (UID: \"f967e0cd-2c0f-42ad-9163-2786089f3ca9\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zkvr7"
Nov 02 15:12:47.673697 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:12:47.673647    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/f967e0cd-2c0f-42ad-9163-2786089f3ca9-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zkvr7\" (UID: \"f967e0cd-2c0f-42ad-9163-2786089f3ca9\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zkvr7"
Nov 02 15:12:47.673751 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:12:47.673714    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/f967e0cd-2c0f-42ad-9163-2786089f3ca9-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zkvr7\" (UID: \"f967e0cd-2c0f-42ad-9163-2786089f3ca9\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zkvr7"
Nov 02 15:12:47.673804 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:12:47.673786    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/f967e0cd-2c0f-42ad-9163-2786089f3ca9-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zkvr7\" (UID: \"f967e0cd-2c0f-42ad-9163-2786089f3ca9\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zkvr7"
Nov 02 15:12:47.779186 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:12:47.775280    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f967e0cd-2c0f-42ad-9163-2786089f3ca9-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zkvr7\" (UID: \"f967e0cd-2c0f-42ad-9163-2786089f3ca9\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zkvr7"
Nov 02 15:12:47.779186 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:12:47.775357    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f967e0cd-2c0f-42ad-9163-2786089f3ca9-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zkvr7\" (UID: \"f967e0cd-2c0f-42ad-9163-2786089f3ca9\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zkvr7"
Nov 02 15:12:47.779186 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:12:47.775410    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/f967e0cd-2c0f-42ad-9163-2786089f3ca9-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zkvr7\" (UID: \"f967e0cd-2c0f-42ad-9163-2786089f3ca9\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zkvr7"
Nov 02 15:12:47.779186 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:12:47.775455    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sh7jk\" (UniqueName: \"kubernetes.io/projected/f967e0cd-2c0f-42ad-9163-2786089f3ca9-kube-api-access-sh7jk\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zkvr7\" (UID: \"f967e0cd-2c0f-42ad-9163-2786089f3ca9\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zkvr7"
Nov 02 15:12:47.779186 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:12:47.775523    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/f967e0cd-2c0f-42ad-9163-2786089f3ca9-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zkvr7\" (UID: \"f967e0cd-2c0f-42ad-9163-2786089f3ca9\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zkvr7"
Nov 02 15:12:47.779186 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:12:47.775578    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/f967e0cd-2c0f-42ad-9163-2786089f3ca9-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zkvr7\" (UID: \"f967e0cd-2c0f-42ad-9163-2786089f3ca9\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zkvr7"
Nov 02 15:12:47.779186 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:12:47.775613    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/f967e0cd-2c0f-42ad-9163-2786089f3ca9-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zkvr7\" (UID: \"f967e0cd-2c0f-42ad-9163-2786089f3ca9\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zkvr7"
Nov 02 15:12:47.779186 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:12:47.776403    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/f967e0cd-2c0f-42ad-9163-2786089f3ca9-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zkvr7\" (UID: \"f967e0cd-2c0f-42ad-9163-2786089f3ca9\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zkvr7"
Nov 02 15:12:47.780694 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:12:47.780667    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f967e0cd-2c0f-42ad-9163-2786089f3ca9-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zkvr7\" (UID: \"f967e0cd-2c0f-42ad-9163-2786089f3ca9\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zkvr7"
Nov 02 15:12:47.780929 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:12:47.780905    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/f967e0cd-2c0f-42ad-9163-2786089f3ca9-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zkvr7\" (UID: \"f967e0cd-2c0f-42ad-9163-2786089f3ca9\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zkvr7"
Nov 02 15:12:47.788529 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:12:47.788502    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/f967e0cd-2c0f-42ad-9163-2786089f3ca9-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zkvr7\" (UID: \"f967e0cd-2c0f-42ad-9163-2786089f3ca9\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zkvr7"
Nov 02 15:12:47.791660 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:12:47.791634    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/f967e0cd-2c0f-42ad-9163-2786089f3ca9-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zkvr7\" (UID: \"f967e0cd-2c0f-42ad-9163-2786089f3ca9\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zkvr7"
Nov 02 15:12:47.800636 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:12:47.800586    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f967e0cd-2c0f-42ad-9163-2786089f3ca9-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zkvr7\" (UID: \"f967e0cd-2c0f-42ad-9163-2786089f3ca9\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zkvr7"
Nov 02 15:12:47.805456 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:12:47.805429    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sh7jk\" (UniqueName: \"kubernetes.io/projected/f967e0cd-2c0f-42ad-9163-2786089f3ca9-kube-api-access-sh7jk\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zkvr7\" (UID: \"f967e0cd-2c0f-42ad-9163-2786089f3ca9\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zkvr7"
Nov 02 15:12:47.823183 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:12:47.823161    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zkvr7"
Nov 02 15:12:48.152508 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:12:48.152473    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zkvr7"]
Nov 02 15:12:48.198735 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:12:48.195669    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:12:48.198735 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:12:48.195780    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:12:49.188118 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:12:49.187807    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zkvr7" event={"ID":"f967e0cd-2c0f-42ad-9163-2786089f3ca9","Type":"ContainerStarted","Data":"0615d280ec875ca2e4dcc25eefcb85c4e22720556e04bfc7562aef72d3f5299c"}
Nov 02 15:12:49.188118 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:12:49.187883    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zkvr7" event={"ID":"f967e0cd-2c0f-42ad-9163-2786089f3ca9","Type":"ContainerStarted","Data":"540981a105f720d68dbd17d5c8fc1075a11a58087d6f2ccdf76dc8bc7b50d545"}
Nov 02 15:12:49.188118 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:12:49.187913    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zkvr7"
Nov 02 15:12:49.191177 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:12:49.190048    5304 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-zkvr7 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 15:12:49.191177 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:12:49.190106    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zkvr7" podUID="f967e0cd-2c0f-42ad-9163-2786089f3ca9" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 15:12:49.254170 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:12:49.254098    5304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zkvr7" podStartSLOduration=2.254082161 podStartE2EDuration="2.254082161s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:12:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:12:49.251527558 +0000 UTC m=+969.245351967" watchObservedRunningTime="2026-11-02 15:12:49.254082161 +0000 UTC m=+969.247906570"
Nov 02 15:12:50.194959 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:12:50.194927    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zkvr7"
Nov 02 15:12:54.804280 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:12:54.804236    5304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k6fjk" containerName="extractor"
Nov 02 15:13:14.721750 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:13:14.721696    5304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c2fcc8a1-cv6nw"]
Nov 02 15:13:14.725319 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:13:14.725295    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c2fcc8a1-cv6nw"
Nov 02 15:13:14.744899 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:13:14.744856    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c2fcc8a1-cv6nw"]
Nov 02 15:13:14.751333 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:13:14.751307    5304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 15:13:14.751490 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:13:14.751336    5304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-m5926\""
Nov 02 15:13:14.888004 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:13:14.887971    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/943b1698-8a48-4d62-a882-08b6b002b679-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-c2fcc8a1-cv6nw\" (UID: \"943b1698-8a48-4d62-a882-08b6b002b679\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c2fcc8a1-cv6nw"
Nov 02 15:13:14.888242 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:13:14.888215    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/943b1698-8a48-4d62-a882-08b6b002b679-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-c2fcc8a1-cv6nw\" (UID: \"943b1698-8a48-4d62-a882-08b6b002b679\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c2fcc8a1-cv6nw"
Nov 02 15:13:14.888465 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:13:14.888446    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7nldg\" (UniqueName: \"kubernetes.io/projected/943b1698-8a48-4d62-a882-08b6b002b679-kube-api-access-7nldg\") pod \"maas-api-revoke-keys-e2e-derive-c2fcc8a1-cv6nw\" (UID: \"943b1698-8a48-4d62-a882-08b6b002b679\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c2fcc8a1-cv6nw"
Nov 02 15:13:14.989504 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:13:14.989427    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/943b1698-8a48-4d62-a882-08b6b002b679-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-c2fcc8a1-cv6nw\" (UID: \"943b1698-8a48-4d62-a882-08b6b002b679\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c2fcc8a1-cv6nw"
Nov 02 15:13:14.989648 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:13:14.989601    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7nldg\" (UniqueName: \"kubernetes.io/projected/943b1698-8a48-4d62-a882-08b6b002b679-kube-api-access-7nldg\") pod \"maas-api-revoke-keys-e2e-derive-c2fcc8a1-cv6nw\" (UID: \"943b1698-8a48-4d62-a882-08b6b002b679\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c2fcc8a1-cv6nw"
Nov 02 15:13:14.989693 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:13:14.989658    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/943b1698-8a48-4d62-a882-08b6b002b679-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-c2fcc8a1-cv6nw\" (UID: \"943b1698-8a48-4d62-a882-08b6b002b679\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c2fcc8a1-cv6nw"
Nov 02 15:13:14.990022 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:13:14.989999    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/943b1698-8a48-4d62-a882-08b6b002b679-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-c2fcc8a1-cv6nw\" (UID: \"943b1698-8a48-4d62-a882-08b6b002b679\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c2fcc8a1-cv6nw"
Nov 02 15:13:14.991930 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:13:14.991906    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/943b1698-8a48-4d62-a882-08b6b002b679-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-c2fcc8a1-cv6nw\" (UID: \"943b1698-8a48-4d62-a882-08b6b002b679\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c2fcc8a1-cv6nw"
Nov 02 15:13:15.013308 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:13:15.013274    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7nldg\" (UniqueName: \"kubernetes.io/projected/943b1698-8a48-4d62-a882-08b6b002b679-kube-api-access-7nldg\") pod \"maas-api-revoke-keys-e2e-derive-c2fcc8a1-cv6nw\" (UID: \"943b1698-8a48-4d62-a882-08b6b002b679\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c2fcc8a1-cv6nw"
Nov 02 15:13:15.038879 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:13:15.038848    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c2fcc8a1-cv6nw"
Nov 02 15:13:15.345467 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:13:15.345440    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c2fcc8a1-cv6nw"]
Nov 02 15:13:15.348050 ip-10-0-135-138 kubenswrapper[5304]: W1102 15:13:15.348025    5304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod943b1698_8a48_4d62_a882_08b6b002b679.slice/crio-37ccc54212c72d98e0a82fa8f73dceb905dfd9434a5be5b287a1cd9e87624681 WatchSource:0}: Error finding container 37ccc54212c72d98e0a82fa8f73dceb905dfd9434a5be5b287a1cd9e87624681: Status 404 returned error can't find the container with id 37ccc54212c72d98e0a82fa8f73dceb905dfd9434a5be5b287a1cd9e87624681
Nov 02 15:13:15.350032 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:13:15.350016    5304 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 02 15:13:15.512302 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:13:15.512264    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c2fcc8a1-cv6nw" event={"ID":"943b1698-8a48-4d62-a882-08b6b002b679","Type":"ContainerStarted","Data":"37ccc54212c72d98e0a82fa8f73dceb905dfd9434a5be5b287a1cd9e87624681"}
Nov 02 15:13:18.540722 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:13:18.540679    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c2fcc8a1-cv6nw" event={"ID":"943b1698-8a48-4d62-a882-08b6b002b679","Type":"ContainerStarted","Data":"156ba9c562d45aac63e59d03266fcd448db031c105d41582bc99e393c277ae45"}
Nov 02 15:13:18.582571 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:13:18.582511    5304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c2fcc8a1-cv6nw" podStartSLOduration=2.2260382180000002 podStartE2EDuration="4.582494278s" totalImagesPullingTime="2.35645606s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:13:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:13:18.580309208 +0000 UTC m=+998.574133618" watchObservedRunningTime="2026-11-02 15:13:18.582494278 +0000 UTC m=+998.576318686"
Nov 02 15:13:19.548349 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:13:19.548313    5304 generic.go:356] "Generic (PLEG): container finished" podID="943b1698-8a48-4d62-a882-08b6b002b679" containerID="156ba9c562d45aac63e59d03266fcd448db031c105d41582bc99e393c277ae45" exitCode=0
Nov 02 15:13:19.548741 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:13:19.548397    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c2fcc8a1-cv6nw" event={"ID":"943b1698-8a48-4d62-a882-08b6b002b679","Type":"ContainerDied","Data":"156ba9c562d45aac63e59d03266fcd448db031c105d41582bc99e393c277ae45"}
Nov 02 15:13:20.723866 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:13:20.723836    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c2fcc8a1-cv6nw"
Nov 02 15:13:20.779272 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:13:20.779189    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/943b1698-8a48-4d62-a882-08b6b002b679-kube-api-access-7nldg\" (UniqueName: \"kubernetes.io/projected/943b1698-8a48-4d62-a882-08b6b002b679-kube-api-access-7nldg\") pod \"943b1698-8a48-4d62-a882-08b6b002b679\" (UID: \"943b1698-8a48-4d62-a882-08b6b002b679\") "
Nov 02 15:13:20.779272 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:13:20.779237    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/943b1698-8a48-4d62-a882-08b6b002b679-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/943b1698-8a48-4d62-a882-08b6b002b679-maas-api-xks-ca\") pod \"943b1698-8a48-4d62-a882-08b6b002b679\" (UID: \"943b1698-8a48-4d62-a882-08b6b002b679\") "
Nov 02 15:13:20.779560 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:13:20.779321    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/943b1698-8a48-4d62-a882-08b6b002b679-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/943b1698-8a48-4d62-a882-08b6b002b679-maas-api-openshift-ca\") pod \"943b1698-8a48-4d62-a882-08b6b002b679\" (UID: \"943b1698-8a48-4d62-a882-08b6b002b679\") "
Nov 02 15:13:20.780397 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:13:20.780363    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/943b1698-8a48-4d62-a882-08b6b002b679-maas-api-openshift-ca" pod "943b1698-8a48-4d62-a882-08b6b002b679" (UID: "943b1698-8a48-4d62-a882-08b6b002b679"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:13:20.785488 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:13:20.785444    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/943b1698-8a48-4d62-a882-08b6b002b679-kube-api-access-7nldg" pod "943b1698-8a48-4d62-a882-08b6b002b679" (UID: "943b1698-8a48-4d62-a882-08b6b002b679"). InnerVolumeSpecName "kube-api-access-7nldg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:13:20.788431 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:13:20.787177    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/943b1698-8a48-4d62-a882-08b6b002b679-maas-api-xks-ca" pod "943b1698-8a48-4d62-a882-08b6b002b679" (UID: "943b1698-8a48-4d62-a882-08b6b002b679"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:13:20.881501 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:13:20.881464    5304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7nldg\" (UniqueName: \"kubernetes.io/projected/943b1698-8a48-4d62-a882-08b6b002b679-kube-api-access-7nldg\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:13:20.881501 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:13:20.881500    5304 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/943b1698-8a48-4d62-a882-08b6b002b679-maas-api-xks-ca\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:13:20.881712 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:13:20.881511    5304 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/943b1698-8a48-4d62-a882-08b6b002b679-maas-api-openshift-ca\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:13:21.562282 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:13:21.562246    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c2fcc8a1-cv6nw"
Nov 02 15:13:21.562468 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:13:21.562278    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c2fcc8a1-cv6nw" event={"ID":"943b1698-8a48-4d62-a882-08b6b002b679","Type":"ContainerDied","Data":"37ccc54212c72d98e0a82fa8f73dceb905dfd9434a5be5b287a1cd9e87624681"}
Nov 02 15:13:21.562468 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:13:21.562348    5304 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="37ccc54212c72d98e0a82fa8f73dceb905dfd9434a5be5b287a1cd9e87624681"
Nov 02 15:13:24.800807 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:13:24.800769    5304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k6fjk" containerName="extractor"
Nov 02 15:13:29.990152 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:13:29.956957    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6af50f/distinct-w0-6af50f-kserve-c6d994647-hbj9z" podUID="3dca5bf3-8e08-47a0-936e-205670d59ede" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:13:30.002155 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:13:29.974316    5304 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-624254/distinct-w3-624254-kserve-5465dc4765-dmq67" podUID="1a5aa910-a6de-417b-8638-81e4ec5c1825" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:13:30.002155 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:13:29.999921    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-624254/distinct-w3-624254-kserve-5465dc4765-dmq67" podUID="1a5aa910-a6de-417b-8638-81e4ec5c1825" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded"
Nov 02 15:13:30.117153 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:13:30.108962    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nkbrk" podUID="1f68d837-fff2-418b-84d7-c2d7eb21a57e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.24:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:13:30.413193 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:13:30.413119    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1cc1b2/distinct-w5-1cc1b2-kserve-5b7459bc6d-zl8xr" podUID="211ad15d-9234-4f57-b1b9-181dd2d74208" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded"
Nov 02 15:13:30.439145 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:13:30.438949    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5dd8e9/simulator-w1-5dd8e9-kserve-6f9f755d6f-gnw5d" podUID="66b75e64-cb24-40be-9076-4984c132e208" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:13:31.006151 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:13:30.958681    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-739bf1/simulator-w2-739bf1-kserve-7c74f85465-ltk4t" podUID="cf6a1ded-28a3-49eb-878b-b4ad77a0bb5c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded"
Nov 02 15:13:31.011381 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:13:31.008371    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7ecfe1/distinct-w4-7ecfe1-kserve-6c847c647d-d2jvv" podUID="f1f1ca5e-cca7-4c54-939f-06f095defcdd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded"
Nov 02 15:13:31.011381 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:13:31.008503    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1cc1b2/premium-w5-1cc1b2-kserve-58db46f97c-cgdqg" podUID="210f620a-8ae7-4e4c-acc3-46f75ddc6105" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded"
Nov 02 15:13:31.011381 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:13:31.009110    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-624254/distinct-w3-624254-kserve-5465dc4765-dmq67" podUID="1a5aa910-a6de-417b-8638-81e4ec5c1825" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:13:31.011381 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:13:31.009776    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-624254/distinct-w3-624254-kserve-5465dc4765-dmq67"
Nov 02 15:13:31.011381 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:13:31.011011    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5dd8e9/premium-w1-5dd8e9-kserve-67588456fb-j6lph" podUID="4ccc607a-14e4-43ca-8999-78100d4b799f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:13:31.011381 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:13:31.011316    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6af50f/distinct-w0-6af50f-kserve-c6d994647-hbj9z" podUID="3dca5bf3-8e08-47a0-936e-205670d59ede" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:13:31.059778 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:13:31.018231    5304 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-6af50f/distinct-w0-6af50f-kserve-c6d994647-hbj9z" podUID="3dca5bf3-8e08-47a0-936e-205670d59ede" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/health\": context deadline exceeded"
Nov 02 15:13:31.059778 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:13:31.018306    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5113c2/premium-w6-5113c2-kserve-b84945cb5-pc6f8" podUID="9fb3e992-7bb0-497a-a82c-6e396ba241c1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded"
Nov 02 15:13:31.100260 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:13:31.082741    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-739bf1/distinct-2-w2-739bf1-kserve-5c58479c48-qgfkh" podUID="9746bf8a-91bd-4715-a02e-ae2e651ca54d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:13:31.100260 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:13:31.086488    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-6af50f/distinct-w0-6af50f-kserve-c6d994647-hbj9z"
Nov 02 15:13:31.100260 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:13:31.086890    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-624254/simulator-w3-624254-kserve-56866b7699-z47sd" podUID="345c710f-b414-4e62-86c7-d51ff4e5be15" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:13:31.236172 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:13:31.211368    5304 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-1cc1b2/premium-w5-1cc1b2-kserve-58db46f97c-cgdqg" podUID="210f620a-8ae7-4e4c-acc3-46f75ddc6105" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:13:31.236172 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:13:31.211484    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nkbrk" podUID="1f68d837-fff2-418b-84d7-c2d7eb21a57e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.24:8000/ready\": context deadline exceeded"
Nov 02 15:13:31.236172 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:13:31.211867    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nkbrk"
Nov 02 15:13:31.451209 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:13:31.451070    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5dd8e9/simulator-w1-5dd8e9-kserve-6f9f755d6f-gnw5d" podUID="66b75e64-cb24-40be-9076-4984c132e208" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": context deadline exceeded"
Nov 02 15:13:31.533179 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:13:31.476927    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1cc1b2/distinct-w5-1cc1b2-kserve-5b7459bc6d-zl8xr" podUID="211ad15d-9234-4f57-b1b9-181dd2d74208" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:13:31.533179 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:13:31.479919    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5dd8e9/simulator-w1-5dd8e9-kserve-6f9f755d6f-gnw5d"
Nov 02 15:13:31.533179 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:13:31.479977    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-1cc1b2/distinct-w5-1cc1b2-kserve-5b7459bc6d-zl8xr"
Nov 02 15:13:32.021149 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:13:32.020722    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-739bf1/simulator-w2-739bf1-kserve-7c74f85465-ltk4t" podUID="cf6a1ded-28a3-49eb-878b-b4ad77a0bb5c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:13:32.021625 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:13:32.021547    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-739bf1/simulator-w2-739bf1-kserve-7c74f85465-ltk4t"
Nov 02 15:13:32.069175 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:13:32.022859    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5113c2/premium-w6-5113c2-kserve-b84945cb5-pc6f8" podUID="9fb3e992-7bb0-497a-a82c-6e396ba241c1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:13:32.069175 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:13:32.023556    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-5113c2/premium-w6-5113c2-kserve-b84945cb5-pc6f8"
Nov 02 15:13:32.069175 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:13:32.029184    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6af50f/distinct-w0-6af50f-kserve-c6d994647-hbj9z" podUID="3dca5bf3-8e08-47a0-936e-205670d59ede" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded"
Nov 02 15:13:32.228572 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:13:32.070792    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5dd8e9/premium-w1-5dd8e9-kserve-67588456fb-j6lph" podUID="4ccc607a-14e4-43ca-8999-78100d4b799f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:13:32.228572 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:13:32.071089    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7ecfe1/distinct-w4-7ecfe1-kserve-6c847c647d-d2jvv" podUID="f1f1ca5e-cca7-4c54-939f-06f095defcdd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded"
Nov 02 15:13:32.228572 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:13:32.098192    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-624254/distinct-w3-624254-kserve-5465dc4765-dmq67" podUID="1a5aa910-a6de-417b-8638-81e4ec5c1825" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded"
Nov 02 15:13:32.228572 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:13:32.098689    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5dd8e9/premium-w1-5dd8e9-kserve-67588456fb-j6lph"
Nov 02 15:13:32.228572 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:13:32.098750    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-7ecfe1/distinct-w4-7ecfe1-kserve-6c847c647d-d2jvv"
Nov 02 15:13:32.228572 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:13:32.104371    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-739bf1/distinct-2-w2-739bf1-kserve-5c58479c48-qgfkh" podUID="9746bf8a-91bd-4715-a02e-ae2e651ca54d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:13:32.228572 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:13:32.104637    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-739bf1/distinct-2-w2-739bf1-kserve-5c58479c48-qgfkh"
Nov 02 15:13:32.228572 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:13:32.159204    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-624254/simulator-w3-624254-kserve-56866b7699-z47sd" podUID="345c710f-b414-4e62-86c7-d51ff4e5be15" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": context deadline exceeded"
Nov 02 15:13:32.248283 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:13:32.248238    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nkbrk" podUID="1f68d837-fff2-418b-84d7-c2d7eb21a57e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.24:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:13:32.274207 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:13:32.249035    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1cc1b2/premium-w5-1cc1b2-kserve-58db46f97c-cgdqg" podUID="210f620a-8ae7-4e4c-acc3-46f75ddc6105" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded"
Nov 02 15:13:32.287561 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:13:32.287509    5304 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-624254/simulator-w3-624254-kserve-56866b7699-z47sd" podUID="345c710f-b414-4e62-86c7-d51ff4e5be15" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/health\": context deadline exceeded"
Nov 02 15:13:32.597748 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:13:32.597699    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5dd8e9/simulator-w1-5dd8e9-kserve-6f9f755d6f-gnw5d" podUID="66b75e64-cb24-40be-9076-4984c132e208" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": context deadline exceeded"
Nov 02 15:13:32.635207 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:13:32.620743    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1cc1b2/distinct-w5-1cc1b2-kserve-5b7459bc6d-zl8xr" podUID="211ad15d-9234-4f57-b1b9-181dd2d74208" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:13:33.099801 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:13:33.099760    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-624254/distinct-w3-624254-kserve-5465dc4765-dmq67" podUID="1a5aa910-a6de-417b-8638-81e4ec5c1825" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:13:33.116150 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:13:33.102664    5304 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-7ecfe1/distinct-w4-7ecfe1-kserve-6c847c647d-d2jvv" podUID="f1f1ca5e-cca7-4c54-939f-06f095defcdd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/health\": context deadline exceeded"
Nov 02 15:13:33.117150 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:13:33.117086    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-739bf1/distinct-2-w2-739bf1-kserve-5c58479c48-qgfkh" podUID="9746bf8a-91bd-4715-a02e-ae2e651ca54d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded"
Nov 02 15:13:33.120580 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:13:33.118732    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-739bf1/simulator-w2-739bf1-kserve-7c74f85465-ltk4t" podUID="cf6a1ded-28a3-49eb-878b-b4ad77a0bb5c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded"
Nov 02 15:13:33.147528 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:13:33.118753    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5113c2/premium-w6-5113c2-kserve-b84945cb5-pc6f8" podUID="9fb3e992-7bb0-497a-a82c-6e396ba241c1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded"
Nov 02 15:13:33.147995 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:13:33.118771    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6af50f/distinct-w0-6af50f-kserve-c6d994647-hbj9z" podUID="3dca5bf3-8e08-47a0-936e-205670d59ede" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:13:33.149899 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:13:33.133712    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-624254/simulator-w3-624254-kserve-56866b7699-z47sd"
Nov 02 15:13:33.150063 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:13:33.150049    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-0621b336-gw-openshift-default-68bdbfb8f7-2j6jv"]
Nov 02 15:13:33.150326 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:13:33.150310    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-1cc1b2/premium-w5-1cc1b2-kserve-58db46f97c-cgdqg"
Nov 02 15:13:33.151344 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:13:33.149494    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5dd8e9/premium-w1-5dd8e9-kserve-67588456fb-j6lph" podUID="4ccc607a-14e4-43ca-8999-78100d4b799f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded"
Nov 02 15:13:33.155311 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:13:33.149465    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7ecfe1/distinct-w4-7ecfe1-kserve-6c847c647d-d2jvv" podUID="f1f1ca5e-cca7-4c54-939f-06f095defcdd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:13:33.176540 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:13:33.154320    5304 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-0621b336-gw-openshift-default-68bdbfb8f7-2j6jv" podUID="39bb7834-0bb3-4dc2-a658-a7d527a7aa1f" containerName="istio-proxy" containerID="cri-o://636da0f4acfe48a29e82ba5d5c225329cc80f20ab40da23d329e3aaa78483bb9" gracePeriod=30
Nov 02 15:13:33.310068 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:13:33.267243    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nkbrk" podUID="1f68d837-fff2-418b-84d7-c2d7eb21a57e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.24:8000/ready\": context deadline exceeded"
Nov 02 15:13:33.310068 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:13:33.275461    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1cc1b2/premium-w5-1cc1b2-kserve-58db46f97c-cgdqg" podUID="210f620a-8ae7-4e4c-acc3-46f75ddc6105" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded"
Nov 02 15:13:33.310068 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:13:33.282731    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-624254/simulator-w3-624254-kserve-56866b7699-z47sd" podUID="345c710f-b414-4e62-86c7-d51ff4e5be15" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": context deadline exceeded"
Nov 02 15:13:33.310068 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:13:33.282785    5304 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nkbrk" podUID="1f68d837-fff2-418b-84d7-c2d7eb21a57e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.24:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:13:33.310068 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:13:33.309510    5304 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-qm7zm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 15:13:33.310068 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:13:33.309571    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-qm7zm" podUID="4d0dd04f-b8ff-4ecd-a1fd-ec15ce4dea5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:13:33.349310 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:13:33.341717    5304 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-4cc4m container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 15:13:33.349310 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:13:33.341779    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-4cc4m" podUID="af975b55-20fa-459f-8659-a7a69f2d2248" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:13:33.629816 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:13:33.629772    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1cc1b2/distinct-w5-1cc1b2-kserve-5b7459bc6d-zl8xr" podUID="211ad15d-9234-4f57-b1b9-181dd2d74208" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:13:33.707396 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:13:33.630392    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5dd8e9/simulator-w1-5dd8e9-kserve-6f9f755d6f-gnw5d" podUID="66b75e64-cb24-40be-9076-4984c132e208" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": context deadline exceeded"
Nov 02 15:13:33.965206 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:13:33.945142    5304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-991657be-openshift-default-647b7b77d9-cx5qr"]
Nov 02 15:13:34.061858 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:13:34.061819    5304 cpu_manager.go:414] "containerMap: removing container" podUID="943b1698-8a48-4d62-a882-08b6b002b679" containerName="revoke-keys"
Nov 02 15:13:34.066707 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:13:34.066668    5304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="943b1698-8a48-4d62-a882-08b6b002b679" containerName="revoke-keys"
Nov 02 15:13:34.067215 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:13:34.067183    5304 memory_manager.go:365] "RemoveStaleState removing state" podUID="943b1698-8a48-4d62-a882-08b6b002b679" containerName="revoke-keys"
Nov 02 15:13:34.076220 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:13:34.076192    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-991657be-openshift-default-647b7b77d9-cx5qr"]
Nov 02 15:13:34.076626 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:13:34.076602    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-991657be-openshift-default-647b7b77d9-cx5qr"
Nov 02 15:13:34.106825 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:13:34.103405    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-624254/distinct-w3-624254-kserve-5465dc4765-dmq67" podUID="1a5aa910-a6de-417b-8638-81e4ec5c1825" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:13:34.125731 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:13:34.125681    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-739bf1/distinct-2-w2-739bf1-kserve-5c58479c48-qgfkh" podUID="9746bf8a-91bd-4715-a02e-ae2e651ca54d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded"
Nov 02 15:13:34.128462 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:13:34.125467    5304 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-5dd8e9/premium-w1-5dd8e9-kserve-67588456fb-j6lph" podUID="4ccc607a-14e4-43ca-8999-78100d4b799f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/health\": context deadline exceeded"
Nov 02 15:13:34.140150 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:13:34.134759    5304 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-1cc1b2/distinct-w5-1cc1b2-kserve-5b7459bc6d-zl8xr" podUID="211ad15d-9234-4f57-b1b9-181dd2d74208" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:13:34.159175 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:13:34.148334    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5113c2/premium-w6-5113c2-kserve-b84945cb5-pc6f8" podUID="9fb3e992-7bb0-497a-a82c-6e396ba241c1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded"
Nov 02 15:13:34.159175 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:13:34.148691    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-739bf1/simulator-w2-739bf1-kserve-7c74f85465-ltk4t" podUID="cf6a1ded-28a3-49eb-878b-b4ad77a0bb5c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded"
Nov 02 15:13:34.194149 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:13:34.190829    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6af50f/distinct-w0-6af50f-kserve-c6d994647-hbj9z" podUID="3dca5bf3-8e08-47a0-936e-205670d59ede" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:13:34.211171 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:13:34.207540    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7ecfe1/distinct-w4-7ecfe1-kserve-6c847c647d-d2jvv" podUID="f1f1ca5e-cca7-4c54-939f-06f095defcdd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:13:34.231215 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:13:34.229825    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5dd8e9/premium-w1-5dd8e9-kserve-67588456fb-j6lph" podUID="4ccc607a-14e4-43ca-8999-78100d4b799f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:13:34.234337 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:13:34.234309    5304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-991657be-openshift-default-dockercfg-w2txd\""
Nov 02 15:13:34.364166 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:13:34.359240    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1cc1b2/premium-w5-1cc1b2-kserve-58db46f97c-cgdqg" podUID="210f620a-8ae7-4e4c-acc3-46f75ddc6105" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:13:34.364166 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:13:34.359721    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nkbrk" podUID="1f68d837-fff2-418b-84d7-c2d7eb21a57e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.24:8000/ready\": context deadline exceeded"
Nov 02 15:13:34.364166 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:13:34.360729    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-624254/simulator-w3-624254-kserve-56866b7699-z47sd" podUID="345c710f-b414-4e62-86c7-d51ff4e5be15" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:13:34.451455 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:13:34.438685    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lx6h9\" (UniqueName: \"kubernetes.io/projected/8626b8a7-0fc5-465d-b071-7b4a7b7fde06-kube-api-access-lx6h9\") pod \"e2e-ait-991657be-openshift-default-647b7b77d9-cx5qr\" (UID: \"8626b8a7-0fc5-465d-b071-7b4a7b7fde06\") " pod="openshift-ingress/e2e-ait-991657be-openshift-default-647b7b77d9-cx5qr"
Nov 02 15:13:34.451455 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:13:34.438836    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8626b8a7-0fc5-465d-b071-7b4a7b7fde06-istio-envoy\") pod \"e2e-ait-991657be-openshift-default-647b7b77d9-cx5qr\" (UID: \"8626b8a7-0fc5-465d-b071-7b4a7b7fde06\") " pod="openshift-ingress/e2e-ait-991657be-openshift-default-647b7b77d9-cx5qr"
Nov 02 15:13:34.451455 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:13:34.438872    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8626b8a7-0fc5-465d-b071-7b4a7b7fde06-istio-data\") pod \"e2e-ait-991657be-openshift-default-647b7b77d9-cx5qr\" (UID: \"8626b8a7-0fc5-465d-b071-7b4a7b7fde06\") " pod="openshift-ingress/e2e-ait-991657be-openshift-default-647b7b77d9-cx5qr"
Nov 02 15:13:34.451455 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:13:34.438967    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8626b8a7-0fc5-465d-b071-7b4a7b7fde06-credential-socket\") pod \"e2e-ait-991657be-openshift-default-647b7b77d9-cx5qr\" (UID: \"8626b8a7-0fc5-465d-b071-7b4a7b7fde06\") " pod="openshift-ingress/e2e-ait-991657be-openshift-default-647b7b77d9-cx5qr"
Nov 02 15:13:34.451455 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:13:34.439051    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8626b8a7-0fc5-465d-b071-7b4a7b7fde06-workload-certs\") pod \"e2e-ait-991657be-openshift-default-647b7b77d9-cx5qr\" (UID: \"8626b8a7-0fc5-465d-b071-7b4a7b7fde06\") " pod="openshift-ingress/e2e-ait-991657be-openshift-default-647b7b77d9-cx5qr"
Nov 02 15:13:34.451455 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:13:34.439116    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8626b8a7-0fc5-465d-b071-7b4a7b7fde06-workload-socket\") pod \"e2e-ait-991657be-openshift-default-647b7b77d9-cx5qr\" (UID: \"8626b8a7-0fc5-465d-b071-7b4a7b7fde06\") " pod="openshift-ingress/e2e-ait-991657be-openshift-default-647b7b77d9-cx5qr"
Nov 02 15:13:34.451455 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:13:34.439165    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8626b8a7-0fc5-465d-b071-7b4a7b7fde06-istio-podinfo\") pod \"e2e-ait-991657be-openshift-default-647b7b77d9-cx5qr\" (UID: \"8626b8a7-0fc5-465d-b071-7b4a7b7fde06\") " pod="openshift-ingress/e2e-ait-991657be-openshift-default-647b7b77d9-cx5qr"
Nov 02 15:13:34.451455 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:13:34.439194    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8626b8a7-0fc5-465d-b071-7b4a7b7fde06-istio-token\") pod \"e2e-ait-991657be-openshift-default-647b7b77d9-cx5qr\" (UID: \"8626b8a7-0fc5-465d-b071-7b4a7b7fde06\") " pod="openshift-ingress/e2e-ait-991657be-openshift-default-647b7b77d9-cx5qr"
Nov 02 15:13:34.451455 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:13:34.439289    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8626b8a7-0fc5-465d-b071-7b4a7b7fde06-istiod-ca-cert\") pod \"e2e-ait-991657be-openshift-default-647b7b77d9-cx5qr\" (UID: \"8626b8a7-0fc5-465d-b071-7b4a7b7fde06\") " pod="openshift-ingress/e2e-ait-991657be-openshift-default-647b7b77d9-cx5qr"
Nov 02 15:13:34.544243 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:13:34.508332    5304 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-739bf1/distinct-2-w2-739bf1-kserve-5c58479c48-qgfkh" podUID="9746bf8a-91bd-4715-a02e-ae2e651ca54d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:13:34.571153 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:13:34.545811    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8626b8a7-0fc5-465d-b071-7b4a7b7fde06-istio-envoy\") pod \"e2e-ait-991657be-openshift-default-647b7b77d9-cx5qr\" (UID: \"8626b8a7-0fc5-465d-b071-7b4a7b7fde06\") " pod="openshift-ingress/e2e-ait-991657be-openshift-default-647b7b77d9-cx5qr"
Nov 02 15:13:34.571675 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:13:34.571644    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8626b8a7-0fc5-465d-b071-7b4a7b7fde06-istio-data\") pod \"e2e-ait-991657be-openshift-default-647b7b77d9-cx5qr\" (UID: \"8626b8a7-0fc5-465d-b071-7b4a7b7fde06\") " pod="openshift-ingress/e2e-ait-991657be-openshift-default-647b7b77d9-cx5qr"
Nov 02 15:13:34.572485 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:13:34.572307    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8626b8a7-0fc5-465d-b071-7b4a7b7fde06-istio-data\") pod \"e2e-ait-991657be-openshift-default-647b7b77d9-cx5qr\" (UID: \"8626b8a7-0fc5-465d-b071-7b4a7b7fde06\") " pod="openshift-ingress/e2e-ait-991657be-openshift-default-647b7b77d9-cx5qr"
Nov 02 15:13:34.572831 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:13:34.572809    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8626b8a7-0fc5-465d-b071-7b4a7b7fde06-credential-socket\") pod \"e2e-ait-991657be-openshift-default-647b7b77d9-cx5qr\" (UID: \"8626b8a7-0fc5-465d-b071-7b4a7b7fde06\") " pod="openshift-ingress/e2e-ait-991657be-openshift-default-647b7b77d9-cx5qr"
Nov 02 15:13:34.588373 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:13:34.573344    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8626b8a7-0fc5-465d-b071-7b4a7b7fde06-credential-socket\") pod \"e2e-ait-991657be-openshift-default-647b7b77d9-cx5qr\" (UID: \"8626b8a7-0fc5-465d-b071-7b4a7b7fde06\") " pod="openshift-ingress/e2e-ait-991657be-openshift-default-647b7b77d9-cx5qr"
Nov 02 15:13:34.589692 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:13:34.589472    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8626b8a7-0fc5-465d-b071-7b4a7b7fde06-workload-certs\") pod \"e2e-ait-991657be-openshift-default-647b7b77d9-cx5qr\" (UID: \"8626b8a7-0fc5-465d-b071-7b4a7b7fde06\") " pod="openshift-ingress/e2e-ait-991657be-openshift-default-647b7b77d9-cx5qr"
Nov 02 15:13:34.616542 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:13:34.616362    5304 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-tl8dm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.30:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 15:13:34.616542 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:13:34.616453    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-tl8dm" podUID="7a09b277-1d1f-45c3-a18a-d393acfa5882" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.30:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:13:34.616846 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:13:34.616571    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8626b8a7-0fc5-465d-b071-7b4a7b7fde06-istio-envoy\") pod \"e2e-ait-991657be-openshift-default-647b7b77d9-cx5qr\" (UID: \"8626b8a7-0fc5-465d-b071-7b4a7b7fde06\") " pod="openshift-ingress/e2e-ait-991657be-openshift-default-647b7b77d9-cx5qr"
Nov 02 15:13:34.617350 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:13:34.617312    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8626b8a7-0fc5-465d-b071-7b4a7b7fde06-workload-socket\") pod \"e2e-ait-991657be-openshift-default-647b7b77d9-cx5qr\" (UID: \"8626b8a7-0fc5-465d-b071-7b4a7b7fde06\") " pod="openshift-ingress/e2e-ait-991657be-openshift-default-647b7b77d9-cx5qr"
Nov 02 15:13:34.617581 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:13:34.617520    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8626b8a7-0fc5-465d-b071-7b4a7b7fde06-istio-podinfo\") pod \"e2e-ait-991657be-openshift-default-647b7b77d9-cx5qr\" (UID: \"8626b8a7-0fc5-465d-b071-7b4a7b7fde06\") " pod="openshift-ingress/e2e-ait-991657be-openshift-default-647b7b77d9-cx5qr"
Nov 02 15:13:34.619456 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:13:34.617560    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8626b8a7-0fc5-465d-b071-7b4a7b7fde06-istio-token\") pod \"e2e-ait-991657be-openshift-default-647b7b77d9-cx5qr\" (UID: \"8626b8a7-0fc5-465d-b071-7b4a7b7fde06\") " pod="openshift-ingress/e2e-ait-991657be-openshift-default-647b7b77d9-cx5qr"
Nov 02 15:13:34.631174 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:13:34.623493    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8626b8a7-0fc5-465d-b071-7b4a7b7fde06-istiod-ca-cert\") pod \"e2e-ait-991657be-openshift-default-647b7b77d9-cx5qr\" (UID: \"8626b8a7-0fc5-465d-b071-7b4a7b7fde06\") " pod="openshift-ingress/e2e-ait-991657be-openshift-default-647b7b77d9-cx5qr"
Nov 02 15:13:34.631556 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:13:34.618807    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8626b8a7-0fc5-465d-b071-7b4a7b7fde06-workload-socket\") pod \"e2e-ait-991657be-openshift-default-647b7b77d9-cx5qr\" (UID: \"8626b8a7-0fc5-465d-b071-7b4a7b7fde06\") " pod="openshift-ingress/e2e-ait-991657be-openshift-default-647b7b77d9-cx5qr"
Nov 02 15:13:34.631556 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:13:34.618422    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8626b8a7-0fc5-465d-b071-7b4a7b7fde06-workload-certs\") pod \"e2e-ait-991657be-openshift-default-647b7b77d9-cx5qr\" (UID: \"8626b8a7-0fc5-465d-b071-7b4a7b7fde06\") " pod="openshift-ingress/e2e-ait-991657be-openshift-default-647b7b77d9-cx5qr"
Nov 02 15:13:34.661444 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:13:34.643859    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8626b8a7-0fc5-465d-b071-7b4a7b7fde06-istio-podinfo\") pod \"e2e-ait-991657be-openshift-default-647b7b77d9-cx5qr\" (UID: \"8626b8a7-0fc5-465d-b071-7b4a7b7fde06\") " pod="openshift-ingress/e2e-ait-991657be-openshift-default-647b7b77d9-cx5qr"
Nov 02 15:13:34.661837 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:13:34.661772    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1cc1b2/distinct-w5-1cc1b2-kserve-5b7459bc6d-zl8xr" podUID="211ad15d-9234-4f57-b1b9-181dd2d74208" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:13:34.717196 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:13:34.716844    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5dd8e9/simulator-w1-5dd8e9-kserve-6f9f755d6f-gnw5d" podUID="66b75e64-cb24-40be-9076-4984c132e208" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:13:34.734914 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:13:34.721938    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8626b8a7-0fc5-465d-b071-7b4a7b7fde06-istiod-ca-cert\") pod \"e2e-ait-991657be-openshift-default-647b7b77d9-cx5qr\" (UID: \"8626b8a7-0fc5-465d-b071-7b4a7b7fde06\") " pod="openshift-ingress/e2e-ait-991657be-openshift-default-647b7b77d9-cx5qr"
Nov 02 15:13:34.734914 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:13:34.722080    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lx6h9\" (UniqueName: \"kubernetes.io/projected/8626b8a7-0fc5-465d-b071-7b4a7b7fde06-kube-api-access-lx6h9\") pod \"e2e-ait-991657be-openshift-default-647b7b77d9-cx5qr\" (UID: \"8626b8a7-0fc5-465d-b071-7b4a7b7fde06\") " pod="openshift-ingress/e2e-ait-991657be-openshift-default-647b7b77d9-cx5qr"
Nov 02 15:13:34.793153 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:13:34.786211    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c2fcc8a1-cv6nw"]
Nov 02 15:13:34.834219 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:13:34.815180    5304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c2fcc8a1-cv6nw"]
Nov 02 15:13:34.883178 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:13:34.880223    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8626b8a7-0fc5-465d-b071-7b4a7b7fde06-istio-token\") pod \"e2e-ait-991657be-openshift-default-647b7b77d9-cx5qr\" (UID: \"8626b8a7-0fc5-465d-b071-7b4a7b7fde06\") " pod="openshift-ingress/e2e-ait-991657be-openshift-default-647b7b77d9-cx5qr"
Nov 02 15:13:35.009190 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:13:35.006961    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lx6h9\" (UniqueName: \"kubernetes.io/projected/8626b8a7-0fc5-465d-b071-7b4a7b7fde06-kube-api-access-lx6h9\") pod \"e2e-ait-991657be-openshift-default-647b7b77d9-cx5qr\" (UID: \"8626b8a7-0fc5-465d-b071-7b4a7b7fde06\") " pod="openshift-ingress/e2e-ait-991657be-openshift-default-647b7b77d9-cx5qr"
Nov 02 15:13:35.063150 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:13:35.047229    5304 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-5113c2/premium-w6-5113c2-kserve-b84945cb5-pc6f8" podUID="9fb3e992-7bb0-497a-a82c-6e396ba241c1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:13:35.150847 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:13:35.140826    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-991657be-openshift-default-647b7b77d9-cx5qr"
Nov 02 15:13:35.150847 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:13:35.141243    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-624254/distinct-w3-624254-kserve-5465dc4765-dmq67" podUID="1a5aa910-a6de-417b-8638-81e4ec5c1825" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:13:35.150847 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:13:35.141467    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-739bf1/distinct-2-w2-739bf1-kserve-5c58479c48-qgfkh" podUID="9746bf8a-91bd-4715-a02e-ae2e651ca54d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded"
Nov 02 15:13:35.175167 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:13:35.164470    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-739bf1/simulator-w2-739bf1-kserve-7c74f85465-ltk4t" podUID="cf6a1ded-28a3-49eb-878b-b4ad77a0bb5c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:13:35.175167 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:13:35.164854    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5113c2/premium-w6-5113c2-kserve-b84945cb5-pc6f8" podUID="9fb3e992-7bb0-497a-a82c-6e396ba241c1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded"
Nov 02 15:13:35.199150 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:13:35.198553    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6af50f/distinct-w0-6af50f-kserve-c6d994647-hbj9z" podUID="3dca5bf3-8e08-47a0-936e-205670d59ede" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded"
Nov 02 15:13:35.220201 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:13:35.210497    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7ecfe1/distinct-w4-7ecfe1-kserve-6c847c647d-d2jvv" podUID="f1f1ca5e-cca7-4c54-939f-06f095defcdd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:13:35.239040 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:13:35.238501    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5dd8e9/premium-w1-5dd8e9-kserve-67588456fb-j6lph" podUID="4ccc607a-14e4-43ca-8999-78100d4b799f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded"
Nov 02 15:13:35.376764 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:13:35.361586    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-624254/simulator-w3-624254-kserve-56866b7699-z47sd" podUID="345c710f-b414-4e62-86c7-d51ff4e5be15" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:13:35.376764 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:13:35.363205    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1cc1b2/premium-w5-1cc1b2-kserve-58db46f97c-cgdqg" podUID="210f620a-8ae7-4e4c-acc3-46f75ddc6105" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded"
Nov 02 15:13:35.376764 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:13:35.363552    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nkbrk" podUID="1f68d837-fff2-418b-84d7-c2d7eb21a57e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.24:8000/ready\": context deadline exceeded"
Nov 02 15:13:35.842079 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:13:35.826889    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5dd8e9/simulator-w1-5dd8e9-kserve-6f9f755d6f-gnw5d" podUID="66b75e64-cb24-40be-9076-4984c132e208" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": context deadline exceeded"
Nov 02 15:13:35.842079 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:13:35.835198    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1cc1b2/distinct-w5-1cc1b2-kserve-5b7459bc6d-zl8xr" podUID="211ad15d-9234-4f57-b1b9-181dd2d74208" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:13:36.248152 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:13:36.247700    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-624254/distinct-w3-624254-kserve-5465dc4765-dmq67" podUID="1a5aa910-a6de-417b-8638-81e4ec5c1825" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded"
Nov 02 15:13:36.303314 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:13:36.249183    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5dd8e9/premium-w1-5dd8e9-kserve-67588456fb-j6lph" podUID="4ccc607a-14e4-43ca-8999-78100d4b799f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:13:36.303314 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:13:36.250046    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5113c2/premium-w6-5113c2-kserve-b84945cb5-pc6f8" podUID="9fb3e992-7bb0-497a-a82c-6e396ba241c1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded"
Nov 02 15:13:36.303314 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:13:36.250714    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-739bf1/simulator-w2-739bf1-kserve-7c74f85465-ltk4t" podUID="cf6a1ded-28a3-49eb-878b-b4ad77a0bb5c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded"
Nov 02 15:13:36.303314 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:13:36.265211    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7ecfe1/distinct-w4-7ecfe1-kserve-6c847c647d-d2jvv" podUID="f1f1ca5e-cca7-4c54-939f-06f095defcdd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded"
Nov 02 15:13:36.303314 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:13:36.291241    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6af50f/distinct-w0-6af50f-kserve-c6d994647-hbj9z" podUID="3dca5bf3-8e08-47a0-936e-205670d59ede" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:13:36.303314 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:13:36.299689    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-739bf1/distinct-2-w2-739bf1-kserve-5c58479c48-qgfkh" podUID="9746bf8a-91bd-4715-a02e-ae2e651ca54d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:13:36.403152 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:13:36.376569    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-624254/simulator-w3-624254-kserve-56866b7699-z47sd" podUID="345c710f-b414-4e62-86c7-d51ff4e5be15" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": context deadline exceeded"
Nov 02 15:13:36.403152 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:13:36.378207    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nkbrk" podUID="1f68d837-fff2-418b-84d7-c2d7eb21a57e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.24:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:13:36.403152 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:13:36.401994    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1cc1b2/premium-w5-1cc1b2-kserve-58db46f97c-cgdqg" podUID="210f620a-8ae7-4e4c-acc3-46f75ddc6105" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:13:36.891207 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:13:36.885950    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1cc1b2/distinct-w5-1cc1b2-kserve-5b7459bc6d-zl8xr" podUID="211ad15d-9234-4f57-b1b9-181dd2d74208" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:13:37.290360 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:13:37.289033    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7ecfe1/distinct-w4-7ecfe1-kserve-6c847c647d-d2jvv" podUID="f1f1ca5e-cca7-4c54-939f-06f095defcdd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:13:37.292044 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:13:37.290596    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-624254/distinct-w3-624254-kserve-5465dc4765-dmq67" podUID="1a5aa910-a6de-417b-8638-81e4ec5c1825" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded"
Nov 02 15:13:37.292044 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:13:37.290834    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-739bf1/simulator-w2-739bf1-kserve-7c74f85465-ltk4t" podUID="cf6a1ded-28a3-49eb-878b-b4ad77a0bb5c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded"
Nov 02 15:13:37.381149 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:13:37.317414    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6af50f/distinct-w0-6af50f-kserve-c6d994647-hbj9z" podUID="3dca5bf3-8e08-47a0-936e-205670d59ede" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded"
Nov 02 15:13:37.402146 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:13:37.397986    5304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="943b1698-8a48-4d62-a882-08b6b002b679" path="/var/lib/kubelet/pods/943b1698-8a48-4d62-a882-08b6b002b679/volumes"
Nov 02 15:13:37.402146 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:13:37.398478    5304 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.119s"
Nov 02 15:13:37.402146 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:13:37.398689    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-5113c2/premium-w6-5113c2-kserve-b84945cb5-pc6f8"
Nov 02 15:13:37.402146 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:13:37.398722    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-991657be-openshift-default-647b7b77d9-cx5qr"]
Nov 02 15:13:37.411620 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:13:37.404942    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-739bf1/distinct-2-w2-739bf1-kserve-5c58479c48-qgfkh" podUID="9746bf8a-91bd-4715-a02e-ae2e651ca54d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded"
Nov 02 15:13:37.419732 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:13:37.419687    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5dd8e9/simulator-w1-5dd8e9-kserve-6f9f755d6f-gnw5d" podUID="66b75e64-cb24-40be-9076-4984c132e208" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": context deadline exceeded"
Nov 02 15:13:37.423504 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:13:37.423461    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5dd8e9/premium-w1-5dd8e9-kserve-67588456fb-j6lph" podUID="4ccc607a-14e4-43ca-8999-78100d4b799f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded"
Nov 02 15:13:37.428079 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:13:37.424080    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-624254/simulator-w3-624254-kserve-56866b7699-z47sd" podUID="345c710f-b414-4e62-86c7-d51ff4e5be15" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": context deadline exceeded"
Nov 02 15:13:37.513387 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:13:37.513361    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-6af50f/distinct-w0-6af50f-kserve-c6d994647-hbj9z"
Nov 02 15:13:37.601571 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:13:37.599189    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nkbrk" podUID="1f68d837-fff2-418b-84d7-c2d7eb21a57e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.24:8000/ready\": context deadline exceeded"
Nov 02 15:13:37.685457 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:13:37.606615    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1cc1b2/premium-w5-1cc1b2-kserve-58db46f97c-cgdqg" podUID="210f620a-8ae7-4e4c-acc3-46f75ddc6105" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded"
Nov 02 15:13:37.685774 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:13:37.615644    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-7ecfe1/distinct-w4-7ecfe1-kserve-6c847c647d-d2jvv"
Nov 02 15:13:37.716103 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:13:37.715742    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-624254/simulator-w3-624254-kserve-56866b7699-z47sd"
Nov 02 15:13:37.717365 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:13:37.717298    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-624254/distinct-w3-624254-kserve-5465dc4765-dmq67"
Nov 02 15:13:37.718847 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:13:37.718722    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-739bf1/distinct-2-w2-739bf1-kserve-5c58479c48-qgfkh"
Nov 02 15:13:37.720421 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:13:37.720382    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-739bf1/simulator-w2-739bf1-kserve-7c74f85465-ltk4t"
Nov 02 15:13:37.724873 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:13:37.724852    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-1cc1b2/distinct-w5-1cc1b2-kserve-5b7459bc6d-zl8xr"
Nov 02 15:13:37.819809 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:13:37.819771    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5dd8e9/premium-w1-5dd8e9-kserve-67588456fb-j6lph"
Nov 02 15:13:37.821151 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:13:37.821104    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nkbrk"
Nov 02 15:13:37.829994 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:13:37.829959    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5dd8e9/simulator-w1-5dd8e9-kserve-6f9f755d6f-gnw5d"
Nov 02 15:13:37.907108 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:13:37.897306    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:13:37.907108 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:13:37.897400    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:13:37.907108 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:13:37.897447    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:13:37.912660 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:13:37.912626    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-991657be-openshift-default-647b7b77d9-cx5qr" event={"ID":"8626b8a7-0fc5-465d-b071-7b4a7b7fde06","Type":"ContainerStarted","Data":"16a3b1b71b765b4e64fd36ed5b062317d82e128b6dc41d45d45f20344719610f"}
Nov 02 15:13:37.919917 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:13:37.919895    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-1cc1b2/premium-w5-1cc1b2-kserve-58db46f97c-cgdqg"
Nov 02 15:13:38.923326 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:13:38.923263    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-991657be-openshift-default-647b7b77d9-cx5qr" event={"ID":"8626b8a7-0fc5-465d-b071-7b4a7b7fde06","Type":"ContainerStarted","Data":"fd35159e7c05710ef9eb630d8e491b7f21472534076a1dd42ef23a158d101e31"}
Nov 02 15:13:38.979359 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:13:38.979285    5304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-991657be-openshift-default-647b7b77d9-cx5qr" podStartSLOduration=5.979264496 podStartE2EDuration="5.979264496s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:13:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:13:38.978201132 +0000 UTC m=+1018.972025541" watchObservedRunningTime="2026-11-02 15:13:38.979264496 +0000 UTC m=+1018.973088904"
Nov 02 15:13:39.141578 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:13:39.141541    5304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-991657be-openshift-default-647b7b77d9-cx5qr"
Nov 02 15:13:39.167331 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:13:39.166456    5304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-991657be-openshift-default-647b7b77d9-cx5qr"
Nov 02 15:13:39.934370 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:13:39.934272    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-991657be-openshift-default-647b7b77d9-cx5qr"
Nov 02 15:13:39.935552 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:13:39.935523    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-991657be-openshift-default-647b7b77d9-cx5qr"
Nov 02 15:13:42.993152 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:13:42.959632    5304 patch_prober.go:29] interesting pod/e2e-migrate-0621b336-gw-openshift-default-68bdbfb8f7-2j6jv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 15:13:42.993152 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:13:42.959694    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-migrate-0621b336-gw-openshift-default-68bdbfb8f7-2j6jv" podUID="39bb7834-0bb3-4dc2-a658-a7d527a7aa1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:13:45.094469 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:13:45.094415    5304 generic.go:356] "Generic (PLEG): container finished" podID="0f5c1330-dd1a-40d2-8f1f-e91aeedd639f" containerID="5e78e646252f977c661e9b3bec6df7ab5f6cb223413c74ee5efc67a31aaadcbf" exitCode=0
Nov 02 15:13:45.095006 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:13:45.094499    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-649ad0be-openshift-default-5564b54495-rrtzm" event={"ID":"0f5c1330-dd1a-40d2-8f1f-e91aeedd639f","Type":"ContainerDied","Data":"5e78e646252f977c661e9b3bec6df7ab5f6cb223413c74ee5efc67a31aaadcbf"}
Nov 02 15:13:45.095077 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:13:45.095033    5304 scope.go:122] "RemoveContainer" containerID="5e78e646252f977c661e9b3bec6df7ab5f6cb223413c74ee5efc67a31aaadcbf"
Nov 02 15:13:45.097386 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:13:45.097340    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:13:45.097568 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:13:45.097454    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:13:45.097568 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:13:45.097519    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:13:46.125548 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:13:46.125512    5304 generic.go:356] "Generic (PLEG): container finished" podID="39bb7834-0bb3-4dc2-a658-a7d527a7aa1f" containerID="636da0f4acfe48a29e82ba5d5c225329cc80f20ab40da23d329e3aaa78483bb9" exitCode=0
Nov 02 15:13:46.130290 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:13:46.130259    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-0621b336-gw-openshift-default-68bdbfb8f7-2j6jv" event={"ID":"39bb7834-0bb3-4dc2-a658-a7d527a7aa1f","Type":"ContainerDied","Data":"636da0f4acfe48a29e82ba5d5c225329cc80f20ab40da23d329e3aaa78483bb9"}
Nov 02 15:13:46.130504 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:13:46.130483    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-649ad0be-openshift-default-5564b54495-rrtzm" event={"ID":"0f5c1330-dd1a-40d2-8f1f-e91aeedd639f","Type":"ContainerStarted","Data":"259f8a403678ea2225edbb627a49b3c692703c866ed764b2c9572645e359d8a6"}
Nov 02 15:13:46.152385 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:13:46.152352    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-0621b336-gw-openshift-default-68bdbfb8f7-2j6jv"
Nov 02 15:13:46.223835 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:13:46.223802    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/39bb7834-0bb3-4dc2-a658-a7d527a7aa1f-kube-api-access-rv4rz\" (UniqueName: \"kubernetes.io/projected/39bb7834-0bb3-4dc2-a658-a7d527a7aa1f-kube-api-access-rv4rz\") pod \"39bb7834-0bb3-4dc2-a658-a7d527a7aa1f\" (UID: \"39bb7834-0bb3-4dc2-a658-a7d527a7aa1f\") "
Nov 02 15:13:46.224013 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:13:46.223962    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/39bb7834-0bb3-4dc2-a658-a7d527a7aa1f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39bb7834-0bb3-4dc2-a658-a7d527a7aa1f-istiod-ca-cert\") pod \"39bb7834-0bb3-4dc2-a658-a7d527a7aa1f\" (UID: \"39bb7834-0bb3-4dc2-a658-a7d527a7aa1f\") "
Nov 02 15:13:46.224087 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:13:46.224072    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/39bb7834-0bb3-4dc2-a658-a7d527a7aa1f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39bb7834-0bb3-4dc2-a658-a7d527a7aa1f-istio-podinfo\") pod \"39bb7834-0bb3-4dc2-a658-a7d527a7aa1f\" (UID: \"39bb7834-0bb3-4dc2-a658-a7d527a7aa1f\") "
Nov 02 15:13:46.224165 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:13:46.224117    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39bb7834-0bb3-4dc2-a658-a7d527a7aa1f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39bb7834-0bb3-4dc2-a658-a7d527a7aa1f-workload-certs\") pod \"39bb7834-0bb3-4dc2-a658-a7d527a7aa1f\" (UID: \"39bb7834-0bb3-4dc2-a658-a7d527a7aa1f\") "
Nov 02 15:13:46.224222 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:13:46.224183    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39bb7834-0bb3-4dc2-a658-a7d527a7aa1f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39bb7834-0bb3-4dc2-a658-a7d527a7aa1f-credential-socket\") pod \"39bb7834-0bb3-4dc2-a658-a7d527a7aa1f\" (UID: \"39bb7834-0bb3-4dc2-a658-a7d527a7aa1f\") "
Nov 02 15:13:46.224280 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:13:46.224248    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39bb7834-0bb3-4dc2-a658-a7d527a7aa1f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39bb7834-0bb3-4dc2-a658-a7d527a7aa1f-workload-socket\") pod \"39bb7834-0bb3-4dc2-a658-a7d527a7aa1f\" (UID: \"39bb7834-0bb3-4dc2-a658-a7d527a7aa1f\") "
Nov 02 15:13:46.224402 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:13:46.224287    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39bb7834-0bb3-4dc2-a658-a7d527a7aa1f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39bb7834-0bb3-4dc2-a658-a7d527a7aa1f-istio-data\") pod \"39bb7834-0bb3-4dc2-a658-a7d527a7aa1f\" (UID: \"39bb7834-0bb3-4dc2-a658-a7d527a7aa1f\") "
Nov 02 15:13:46.224469 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:13:46.224405    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39bb7834-0bb3-4dc2-a658-a7d527a7aa1f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39bb7834-0bb3-4dc2-a658-a7d527a7aa1f-istio-envoy\") pod \"39bb7834-0bb3-4dc2-a658-a7d527a7aa1f\" (UID: \"39bb7834-0bb3-4dc2-a658-a7d527a7aa1f\") "
Nov 02 15:13:46.224558 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:13:46.224430    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39bb7834-0bb3-4dc2-a658-a7d527a7aa1f-istiod-ca-cert" pod "39bb7834-0bb3-4dc2-a658-a7d527a7aa1f" (UID: "39bb7834-0bb3-4dc2-a658-a7d527a7aa1f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:13:46.224625 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:13:46.224477    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/39bb7834-0bb3-4dc2-a658-a7d527a7aa1f-istio-token\" (UniqueName: \"kubernetes.io/projected/39bb7834-0bb3-4dc2-a658-a7d527a7aa1f-istio-token\") pod \"39bb7834-0bb3-4dc2-a658-a7d527a7aa1f\" (UID: \"39bb7834-0bb3-4dc2-a658-a7d527a7aa1f\") "
Nov 02 15:13:46.224625 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:13:46.224463    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39bb7834-0bb3-4dc2-a658-a7d527a7aa1f-workload-certs" pod "39bb7834-0bb3-4dc2-a658-a7d527a7aa1f" (UID: "39bb7834-0bb3-4dc2-a658-a7d527a7aa1f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:13:46.224625 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:13:46.224516    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39bb7834-0bb3-4dc2-a658-a7d527a7aa1f-credential-socket" pod "39bb7834-0bb3-4dc2-a658-a7d527a7aa1f" (UID: "39bb7834-0bb3-4dc2-a658-a7d527a7aa1f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:13:46.224625 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:13:46.224544    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39bb7834-0bb3-4dc2-a658-a7d527a7aa1f-workload-socket" pod "39bb7834-0bb3-4dc2-a658-a7d527a7aa1f" (UID: "39bb7834-0bb3-4dc2-a658-a7d527a7aa1f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:13:46.224812 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:13:46.224627    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39bb7834-0bb3-4dc2-a658-a7d527a7aa1f-istio-data" pod "39bb7834-0bb3-4dc2-a658-a7d527a7aa1f" (UID: "39bb7834-0bb3-4dc2-a658-a7d527a7aa1f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:13:46.225923 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:13:46.225825    5304 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39bb7834-0bb3-4dc2-a658-a7d527a7aa1f-istiod-ca-cert\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:13:46.225923 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:13:46.225853    5304 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39bb7834-0bb3-4dc2-a658-a7d527a7aa1f-workload-certs\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:13:46.225923 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:13:46.225869    5304 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39bb7834-0bb3-4dc2-a658-a7d527a7aa1f-credential-socket\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:13:46.225923 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:13:46.225887    5304 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39bb7834-0bb3-4dc2-a658-a7d527a7aa1f-workload-socket\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:13:46.225923 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:13:46.225901    5304 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39bb7834-0bb3-4dc2-a658-a7d527a7aa1f-istio-data\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:13:46.226532 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:13:46.226504    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39bb7834-0bb3-4dc2-a658-a7d527a7aa1f-kube-api-access-rv4rz" pod "39bb7834-0bb3-4dc2-a658-a7d527a7aa1f" (UID: "39bb7834-0bb3-4dc2-a658-a7d527a7aa1f"). InnerVolumeSpecName "kube-api-access-rv4rz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:13:46.226665 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:13:46.226651    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39bb7834-0bb3-4dc2-a658-a7d527a7aa1f-istio-token" pod "39bb7834-0bb3-4dc2-a658-a7d527a7aa1f" (UID: "39bb7834-0bb3-4dc2-a658-a7d527a7aa1f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:13:46.226789 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:13:46.226770    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39bb7834-0bb3-4dc2-a658-a7d527a7aa1f-istio-envoy" pod "39bb7834-0bb3-4dc2-a658-a7d527a7aa1f" (UID: "39bb7834-0bb3-4dc2-a658-a7d527a7aa1f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:13:46.226844 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:13:46.226812    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/39bb7834-0bb3-4dc2-a658-a7d527a7aa1f-istio-podinfo" pod "39bb7834-0bb3-4dc2-a658-a7d527a7aa1f" (UID: "39bb7834-0bb3-4dc2-a658-a7d527a7aa1f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 15:13:46.296615 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:13:46.296581    5304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-649ad0be-openshift-default-5564b54495-rrtzm"
Nov 02 15:13:46.297147 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:13:46.296982    5304 patch_prober.go:29] interesting pod/e2e-ait-649ad0be-openshift-default-5564b54495-rrtzm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Nov 02 15:13:46.297637 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:13:46.297601    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-649ad0be-openshift-default-5564b54495-rrtzm" podUID="0f5c1330-dd1a-40d2-8f1f-e91aeedd639f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Nov 02 15:13:46.376673 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:13:46.376591    5304 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39bb7834-0bb3-4dc2-a658-a7d527a7aa1f-istio-token\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:13:46.376846 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:13:46.376833    5304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rv4rz\" (UniqueName: \"kubernetes.io/projected/39bb7834-0bb3-4dc2-a658-a7d527a7aa1f-kube-api-access-rv4rz\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:13:46.376914 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:13:46.376899    5304 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39bb7834-0bb3-4dc2-a658-a7d527a7aa1f-istio-podinfo\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:13:46.376969 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:13:46.376962    5304 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39bb7834-0bb3-4dc2-a658-a7d527a7aa1f-istio-envoy\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:13:46.380861 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:13:46.380823    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-649ad0be-openshift-default-5564b54495-rrtzm"]
Nov 02 15:13:47.163541 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:13:47.163515    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-0621b336-gw-openshift-default-68bdbfb8f7-2j6jv"
Nov 02 15:13:47.164032 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:13:47.163537    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-0621b336-gw-openshift-default-68bdbfb8f7-2j6jv" event={"ID":"39bb7834-0bb3-4dc2-a658-a7d527a7aa1f","Type":"ContainerDied","Data":"bee6ec6e928e5b2863ea8f16c6c6e3ca136031fd8bee2f38ae9d32391b86016f"}
Nov 02 15:13:47.164032 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:13:47.163625    5304 scope.go:122] "RemoveContainer" containerID="636da0f4acfe48a29e82ba5d5c225329cc80f20ab40da23d329e3aaa78483bb9"
Nov 02 15:13:47.164288 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:13:47.164266    5304 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/e2e-ait-649ad0be-openshift-default-5564b54495-rrtzm" secret="" err="secret \"e2e-ait-649ad0be-openshift-default-dockercfg-vt8fc\" not found"
Nov 02 15:13:47.272949 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:13:47.272903    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-0621b336-gw-openshift-default-68bdbfb8f7-2j6jv"]
Nov 02 15:13:47.273161 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:13:47.272969    5304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-0621b336-gw-openshift-default-68bdbfb8f7-2j6jv"]
Nov 02 15:13:47.295479 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:13:47.295442    5304 patch_prober.go:29] interesting pod/e2e-ait-649ad0be-openshift-default-5564b54495-rrtzm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Nov 02 15:13:47.295626 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:13:47.295575    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-649ad0be-openshift-default-5564b54495-rrtzm" podUID="0f5c1330-dd1a-40d2-8f1f-e91aeedd639f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Nov 02 15:13:48.124766 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:13:48.124727    5304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="39bb7834-0bb3-4dc2-a658-a7d527a7aa1f" path="/var/lib/kubelet/pods/39bb7834-0bb3-4dc2-a658-a7d527a7aa1f/volumes"
Nov 02 15:13:48.180109 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:13:48.180073    5304 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-649ad0be-openshift-default-5564b54495-rrtzm" podUID="0f5c1330-dd1a-40d2-8f1f-e91aeedd639f" containerName="istio-proxy" containerID="cri-o://259f8a403678ea2225edbb627a49b3c692703c866ed764b2c9572645e359d8a6" gracePeriod=30
Nov 02 15:13:52.013386 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:13:52.013352    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-649ad0be-openshift-default-5564b54495-rrtzm"
Nov 02 15:13:54.918391 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:13:54.918348    5304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k6fjk" containerName="extractor"
Nov 02 15:13:57.074766 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:13:57.074740    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-649ad0be-openshift-default-5564b54495-rrtzm"
Nov 02 15:13:57.146785 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:13:57.146695    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f5c1330-dd1a-40d2-8f1f-e91aeedd639f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f5c1330-dd1a-40d2-8f1f-e91aeedd639f-istio-envoy\") pod \"0f5c1330-dd1a-40d2-8f1f-e91aeedd639f\" (UID: \"0f5c1330-dd1a-40d2-8f1f-e91aeedd639f\") "
Nov 02 15:13:57.146785 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:13:57.146755    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f5c1330-dd1a-40d2-8f1f-e91aeedd639f-istio-token\" (UniqueName: \"kubernetes.io/projected/0f5c1330-dd1a-40d2-8f1f-e91aeedd639f-istio-token\") pod \"0f5c1330-dd1a-40d2-8f1f-e91aeedd639f\" (UID: \"0f5c1330-dd1a-40d2-8f1f-e91aeedd639f\") "
Nov 02 15:13:57.147017 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:13:57.146825    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f5c1330-dd1a-40d2-8f1f-e91aeedd639f-kube-api-access-lfgdq\" (UniqueName: \"kubernetes.io/projected/0f5c1330-dd1a-40d2-8f1f-e91aeedd639f-kube-api-access-lfgdq\") pod \"0f5c1330-dd1a-40d2-8f1f-e91aeedd639f\" (UID: \"0f5c1330-dd1a-40d2-8f1f-e91aeedd639f\") "
Nov 02 15:13:57.147017 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:13:57.146871    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f5c1330-dd1a-40d2-8f1f-e91aeedd639f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f5c1330-dd1a-40d2-8f1f-e91aeedd639f-credential-socket\") pod \"0f5c1330-dd1a-40d2-8f1f-e91aeedd639f\" (UID: \"0f5c1330-dd1a-40d2-8f1f-e91aeedd639f\") "
Nov 02 15:13:57.147017 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:13:57.146939    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f5c1330-dd1a-40d2-8f1f-e91aeedd639f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f5c1330-dd1a-40d2-8f1f-e91aeedd639f-workload-socket\") pod \"0f5c1330-dd1a-40d2-8f1f-e91aeedd639f\" (UID: \"0f5c1330-dd1a-40d2-8f1f-e91aeedd639f\") "
Nov 02 15:13:57.147201 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:13:57.147116    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f5c1330-dd1a-40d2-8f1f-e91aeedd639f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f5c1330-dd1a-40d2-8f1f-e91aeedd639f-istio-data\") pod \"0f5c1330-dd1a-40d2-8f1f-e91aeedd639f\" (UID: \"0f5c1330-dd1a-40d2-8f1f-e91aeedd639f\") "
Nov 02 15:13:57.147264 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:13:57.147216    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f5c1330-dd1a-40d2-8f1f-e91aeedd639f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f5c1330-dd1a-40d2-8f1f-e91aeedd639f-workload-certs\") pod \"0f5c1330-dd1a-40d2-8f1f-e91aeedd639f\" (UID: \"0f5c1330-dd1a-40d2-8f1f-e91aeedd639f\") "
Nov 02 15:13:57.147317 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:13:57.147274    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0f5c1330-dd1a-40d2-8f1f-e91aeedd639f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f5c1330-dd1a-40d2-8f1f-e91aeedd639f-istiod-ca-cert\") pod \"0f5c1330-dd1a-40d2-8f1f-e91aeedd639f\" (UID: \"0f5c1330-dd1a-40d2-8f1f-e91aeedd639f\") "
Nov 02 15:13:57.148870 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:13:57.147419    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0f5c1330-dd1a-40d2-8f1f-e91aeedd639f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f5c1330-dd1a-40d2-8f1f-e91aeedd639f-istio-podinfo\") pod \"0f5c1330-dd1a-40d2-8f1f-e91aeedd639f\" (UID: \"0f5c1330-dd1a-40d2-8f1f-e91aeedd639f\") "
Nov 02 15:13:57.148870 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:13:57.147625    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f5c1330-dd1a-40d2-8f1f-e91aeedd639f-workload-socket" pod "0f5c1330-dd1a-40d2-8f1f-e91aeedd639f" (UID: "0f5c1330-dd1a-40d2-8f1f-e91aeedd639f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:13:57.148870 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:13:57.147773    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f5c1330-dd1a-40d2-8f1f-e91aeedd639f-workload-certs" pod "0f5c1330-dd1a-40d2-8f1f-e91aeedd639f" (UID: "0f5c1330-dd1a-40d2-8f1f-e91aeedd639f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:13:57.148870 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:13:57.147866    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f5c1330-dd1a-40d2-8f1f-e91aeedd639f-credential-socket" pod "0f5c1330-dd1a-40d2-8f1f-e91aeedd639f" (UID: "0f5c1330-dd1a-40d2-8f1f-e91aeedd639f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:13:57.148870 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:13:57.147960    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f5c1330-dd1a-40d2-8f1f-e91aeedd639f-istiod-ca-cert" pod "0f5c1330-dd1a-40d2-8f1f-e91aeedd639f" (UID: "0f5c1330-dd1a-40d2-8f1f-e91aeedd639f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:13:57.148870 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:13:57.148448    5304 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f5c1330-dd1a-40d2-8f1f-e91aeedd639f-workload-certs\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:13:57.148870 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:13:57.148473    5304 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f5c1330-dd1a-40d2-8f1f-e91aeedd639f-istiod-ca-cert\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:13:57.148870 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:13:57.148486    5304 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f5c1330-dd1a-40d2-8f1f-e91aeedd639f-credential-socket\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:13:57.148870 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:13:57.148501    5304 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f5c1330-dd1a-40d2-8f1f-e91aeedd639f-workload-socket\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:13:57.149483 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:13:57.149268    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f5c1330-dd1a-40d2-8f1f-e91aeedd639f-istio-data" pod "0f5c1330-dd1a-40d2-8f1f-e91aeedd639f" (UID: "0f5c1330-dd1a-40d2-8f1f-e91aeedd639f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:13:57.149544 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:13:57.149486    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f5c1330-dd1a-40d2-8f1f-e91aeedd639f-istio-token" pod "0f5c1330-dd1a-40d2-8f1f-e91aeedd639f" (UID: "0f5c1330-dd1a-40d2-8f1f-e91aeedd639f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:13:57.164072 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:13:57.151298    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f5c1330-dd1a-40d2-8f1f-e91aeedd639f-kube-api-access-lfgdq" pod "0f5c1330-dd1a-40d2-8f1f-e91aeedd639f" (UID: "0f5c1330-dd1a-40d2-8f1f-e91aeedd639f"). InnerVolumeSpecName "kube-api-access-lfgdq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:13:57.164072 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:13:57.151406    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0f5c1330-dd1a-40d2-8f1f-e91aeedd639f-istio-podinfo" pod "0f5c1330-dd1a-40d2-8f1f-e91aeedd639f" (UID: "0f5c1330-dd1a-40d2-8f1f-e91aeedd639f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 15:13:57.164072 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:13:57.151438    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f5c1330-dd1a-40d2-8f1f-e91aeedd639f-istio-envoy" pod "0f5c1330-dd1a-40d2-8f1f-e91aeedd639f" (UID: "0f5c1330-dd1a-40d2-8f1f-e91aeedd639f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:13:57.250171 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:13:57.250137    5304 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f5c1330-dd1a-40d2-8f1f-e91aeedd639f-istio-data\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:13:57.250171 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:13:57.250164    5304 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f5c1330-dd1a-40d2-8f1f-e91aeedd639f-istio-podinfo\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:13:57.250171 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:13:57.250175    5304 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f5c1330-dd1a-40d2-8f1f-e91aeedd639f-istio-envoy\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:13:57.250405 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:13:57.250189    5304 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f5c1330-dd1a-40d2-8f1f-e91aeedd639f-istio-token\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:13:57.250405 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:13:57.250198    5304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lfgdq\" (UniqueName: \"kubernetes.io/projected/0f5c1330-dd1a-40d2-8f1f-e91aeedd639f-kube-api-access-lfgdq\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:13:57.271075 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:13:57.271043    5304 generic.go:356] "Generic (PLEG): container finished" podID="0f5c1330-dd1a-40d2-8f1f-e91aeedd639f" containerID="259f8a403678ea2225edbb627a49b3c692703c866ed764b2c9572645e359d8a6" exitCode=0
Nov 02 15:13:57.271270 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:13:57.271091    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-649ad0be-openshift-default-5564b54495-rrtzm"
Nov 02 15:13:57.271270 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:13:57.271142    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-649ad0be-openshift-default-5564b54495-rrtzm" event={"ID":"0f5c1330-dd1a-40d2-8f1f-e91aeedd639f","Type":"ContainerDied","Data":"259f8a403678ea2225edbb627a49b3c692703c866ed764b2c9572645e359d8a6"}
Nov 02 15:13:57.271270 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:13:57.271203    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-649ad0be-openshift-default-5564b54495-rrtzm" event={"ID":"0f5c1330-dd1a-40d2-8f1f-e91aeedd639f","Type":"ContainerDied","Data":"4275003250f60d13cd2ed35204bbe484a9e15f534439055679164371690f7eae"}
Nov 02 15:13:57.271270 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:13:57.271230    5304 scope.go:122] "RemoveContainer" containerID="259f8a403678ea2225edbb627a49b3c692703c866ed764b2c9572645e359d8a6"
Nov 02 15:13:57.281832 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:13:57.281809    5304 scope.go:122] "RemoveContainer" containerID="5e78e646252f977c661e9b3bec6df7ab5f6cb223413c74ee5efc67a31aaadcbf"
Nov 02 15:13:57.291812 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:13:57.291793    5304 scope.go:122] "RemoveContainer" containerID="259f8a403678ea2225edbb627a49b3c692703c866ed764b2c9572645e359d8a6"
Nov 02 15:13:57.292159 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:13:57.292134    5304 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"259f8a403678ea2225edbb627a49b3c692703c866ed764b2c9572645e359d8a6\": container with ID starting with 259f8a403678ea2225edbb627a49b3c692703c866ed764b2c9572645e359d8a6 not found: ID does not exist" containerID="259f8a403678ea2225edbb627a49b3c692703c866ed764b2c9572645e359d8a6"
Nov 02 15:13:57.292250 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:13:57.292169    5304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"259f8a403678ea2225edbb627a49b3c692703c866ed764b2c9572645e359d8a6"} err="failed to get container status \"259f8a403678ea2225edbb627a49b3c692703c866ed764b2c9572645e359d8a6\": rpc error: code = NotFound desc = could not find container \"259f8a403678ea2225edbb627a49b3c692703c866ed764b2c9572645e359d8a6\": container with ID starting with 259f8a403678ea2225edbb627a49b3c692703c866ed764b2c9572645e359d8a6 not found: ID does not exist"
Nov 02 15:13:57.292250 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:13:57.292191    5304 scope.go:122] "RemoveContainer" containerID="5e78e646252f977c661e9b3bec6df7ab5f6cb223413c74ee5efc67a31aaadcbf"
Nov 02 15:13:57.292445 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:13:57.292428    5304 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e78e646252f977c661e9b3bec6df7ab5f6cb223413c74ee5efc67a31aaadcbf\": container with ID starting with 5e78e646252f977c661e9b3bec6df7ab5f6cb223413c74ee5efc67a31aaadcbf not found: ID does not exist" containerID="5e78e646252f977c661e9b3bec6df7ab5f6cb223413c74ee5efc67a31aaadcbf"
Nov 02 15:13:57.292509 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:13:57.292450    5304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e78e646252f977c661e9b3bec6df7ab5f6cb223413c74ee5efc67a31aaadcbf"} err="failed to get container status \"5e78e646252f977c661e9b3bec6df7ab5f6cb223413c74ee5efc67a31aaadcbf\": rpc error: code = NotFound desc = could not find container \"5e78e646252f977c661e9b3bec6df7ab5f6cb223413c74ee5efc67a31aaadcbf\": container with ID starting with 5e78e646252f977c661e9b3bec6df7ab5f6cb223413c74ee5efc67a31aaadcbf not found: ID does not exist"
Nov 02 15:13:57.324647 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:13:57.324613    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-649ad0be-openshift-default-5564b54495-rrtzm"]
Nov 02 15:13:57.328110 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:13:57.328082    5304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-649ad0be-openshift-default-5564b54495-rrtzm"]
Nov 02 15:13:58.120590 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:13:58.120559    5304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0f5c1330-dd1a-40d2-8f1f-e91aeedd639f" path="/var/lib/kubelet/pods/0f5c1330-dd1a-40d2-8f1f-e91aeedd639f/volumes"
Nov 02 15:14:08.229149 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:14:08.228915    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-739bf1/simulator-w2-739bf1-kserve-7c74f85465-ltk4t" podUID="cf6a1ded-28a3-49eb-878b-b4ad77a0bb5c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:14:24.802514 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:14:24.802470    5304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k6fjk" containerName="extractor"
Nov 02 15:14:40.975347 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:14:40.975302    5304 patch_prober.go:29] interesting pod/e2e-ait-991657be-openshift-default-647b7b77d9-cx5qr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 15:14:40.975753 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:14:40.975401    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-991657be-openshift-default-647b7b77d9-cx5qr" podUID="8626b8a7-0fc5-465d-b071-7b4a7b7fde06" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:14:43.706884 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:14:43.706851    5304 generic.go:356] "Generic (PLEG): container finished" podID="8626b8a7-0fc5-465d-b071-7b4a7b7fde06" containerID="fd35159e7c05710ef9eb630d8e491b7f21472534076a1dd42ef23a158d101e31" exitCode=0
Nov 02 15:14:43.707337 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:14:43.706919    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-991657be-openshift-default-647b7b77d9-cx5qr" event={"ID":"8626b8a7-0fc5-465d-b071-7b4a7b7fde06","Type":"ContainerDied","Data":"fd35159e7c05710ef9eb630d8e491b7f21472534076a1dd42ef23a158d101e31"}
Nov 02 15:14:43.707449 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:14:43.707429    5304 scope.go:122] "RemoveContainer" containerID="fd35159e7c05710ef9eb630d8e491b7f21472534076a1dd42ef23a158d101e31"
Nov 02 15:14:43.709299 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:14:43.709269    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:14:43.709389 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:14:43.709339    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:14:43.709389 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:14:43.709370    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:14:43.833177 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:14:43.826203    5304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w0-453864-openshift-default-77d4cd74c7-bp7wp"]
Nov 02 15:14:43.833177 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:14:43.827157    5304 cpu_manager.go:414] "containerMap: removing container" podUID="39bb7834-0bb3-4dc2-a658-a7d527a7aa1f" containerName="istio-proxy"
Nov 02 15:14:43.833177 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:14:43.827179    5304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="39bb7834-0bb3-4dc2-a658-a7d527a7aa1f" containerName="istio-proxy"
Nov 02 15:14:43.833177 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:14:43.827253    5304 cpu_manager.go:414] "containerMap: removing container" podUID="0f5c1330-dd1a-40d2-8f1f-e91aeedd639f" containerName="istio-proxy"
Nov 02 15:14:43.833177 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:14:43.827260    5304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f5c1330-dd1a-40d2-8f1f-e91aeedd639f" containerName="istio-proxy"
Nov 02 15:14:43.833177 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:14:43.827271    5304 cpu_manager.go:414] "containerMap: removing container" podUID="0f5c1330-dd1a-40d2-8f1f-e91aeedd639f" containerName="istio-proxy"
Nov 02 15:14:43.833177 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:14:43.827278    5304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f5c1330-dd1a-40d2-8f1f-e91aeedd639f" containerName="istio-proxy"
Nov 02 15:14:43.833177 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:14:43.827430    5304 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f5c1330-dd1a-40d2-8f1f-e91aeedd639f" containerName="istio-proxy"
Nov 02 15:14:43.833177 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:14:43.827444    5304 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f5c1330-dd1a-40d2-8f1f-e91aeedd639f" containerName="istio-proxy"
Nov 02 15:14:43.833177 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:14:43.827459    5304 memory_manager.go:365] "RemoveStaleState removing state" podUID="39bb7834-0bb3-4dc2-a658-a7d527a7aa1f" containerName="istio-proxy"
Nov 02 15:14:43.833177 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:14:43.831741    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-453864-openshift-default-77d4cd74c7-bp7wp"
Nov 02 15:14:43.862158 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:14:43.861413    5304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w0-453864-openshift-default-dockercfg-nvg5q\""
Nov 02 15:14:43.878775 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:14:43.873185    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-453864-openshift-default-77d4cd74c7-bp7wp"]
Nov 02 15:14:43.896911 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:14:43.896849    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73b5a23d-1e75-4beb-920e-8953826899af-istio-envoy\") pod \"e2e-shared-b-w0-453864-openshift-default-77d4cd74c7-bp7wp\" (UID: \"73b5a23d-1e75-4beb-920e-8953826899af\") " pod="openshift-ingress/e2e-shared-b-w0-453864-openshift-default-77d4cd74c7-bp7wp"
Nov 02 15:14:43.897099 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:14:43.896976    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73b5a23d-1e75-4beb-920e-8953826899af-istio-data\") pod \"e2e-shared-b-w0-453864-openshift-default-77d4cd74c7-bp7wp\" (UID: \"73b5a23d-1e75-4beb-920e-8953826899af\") " pod="openshift-ingress/e2e-shared-b-w0-453864-openshift-default-77d4cd74c7-bp7wp"
Nov 02 15:14:43.897099 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:14:43.897029    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jspkf\" (UniqueName: \"kubernetes.io/projected/73b5a23d-1e75-4beb-920e-8953826899af-kube-api-access-jspkf\") pod \"e2e-shared-b-w0-453864-openshift-default-77d4cd74c7-bp7wp\" (UID: \"73b5a23d-1e75-4beb-920e-8953826899af\") " pod="openshift-ingress/e2e-shared-b-w0-453864-openshift-default-77d4cd74c7-bp7wp"
Nov 02 15:14:43.897244 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:14:43.897204    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73b5a23d-1e75-4beb-920e-8953826899af-istiod-ca-cert\") pod \"e2e-shared-b-w0-453864-openshift-default-77d4cd74c7-bp7wp\" (UID: \"73b5a23d-1e75-4beb-920e-8953826899af\") " pod="openshift-ingress/e2e-shared-b-w0-453864-openshift-default-77d4cd74c7-bp7wp"
Nov 02 15:14:43.897300 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:14:43.897245    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73b5a23d-1e75-4beb-920e-8953826899af-workload-certs\") pod \"e2e-shared-b-w0-453864-openshift-default-77d4cd74c7-bp7wp\" (UID: \"73b5a23d-1e75-4beb-920e-8953826899af\") " pod="openshift-ingress/e2e-shared-b-w0-453864-openshift-default-77d4cd74c7-bp7wp"
Nov 02 15:14:43.897359 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:14:43.897336    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/73b5a23d-1e75-4beb-920e-8953826899af-istio-token\") pod \"e2e-shared-b-w0-453864-openshift-default-77d4cd74c7-bp7wp\" (UID: \"73b5a23d-1e75-4beb-920e-8953826899af\") " pod="openshift-ingress/e2e-shared-b-w0-453864-openshift-default-77d4cd74c7-bp7wp"
Nov 02 15:14:43.897515 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:14:43.897495    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73b5a23d-1e75-4beb-920e-8953826899af-credential-socket\") pod \"e2e-shared-b-w0-453864-openshift-default-77d4cd74c7-bp7wp\" (UID: \"73b5a23d-1e75-4beb-920e-8953826899af\") " pod="openshift-ingress/e2e-shared-b-w0-453864-openshift-default-77d4cd74c7-bp7wp"
Nov 02 15:14:43.897577 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:14:43.897554    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73b5a23d-1e75-4beb-920e-8953826899af-istio-podinfo\") pod \"e2e-shared-b-w0-453864-openshift-default-77d4cd74c7-bp7wp\" (UID: \"73b5a23d-1e75-4beb-920e-8953826899af\") " pod="openshift-ingress/e2e-shared-b-w0-453864-openshift-default-77d4cd74c7-bp7wp"
Nov 02 15:14:43.897837 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:14:43.897653    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73b5a23d-1e75-4beb-920e-8953826899af-workload-socket\") pod \"e2e-shared-b-w0-453864-openshift-default-77d4cd74c7-bp7wp\" (UID: \"73b5a23d-1e75-4beb-920e-8953826899af\") " pod="openshift-ingress/e2e-shared-b-w0-453864-openshift-default-77d4cd74c7-bp7wp"
Nov 02 15:14:44.007194 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:14:44.003236    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73b5a23d-1e75-4beb-920e-8953826899af-credential-socket\") pod \"e2e-shared-b-w0-453864-openshift-default-77d4cd74c7-bp7wp\" (UID: \"73b5a23d-1e75-4beb-920e-8953826899af\") " pod="openshift-ingress/e2e-shared-b-w0-453864-openshift-default-77d4cd74c7-bp7wp"
Nov 02 15:14:44.007194 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:14:44.003299    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73b5a23d-1e75-4beb-920e-8953826899af-istio-podinfo\") pod \"e2e-shared-b-w0-453864-openshift-default-77d4cd74c7-bp7wp\" (UID: \"73b5a23d-1e75-4beb-920e-8953826899af\") " pod="openshift-ingress/e2e-shared-b-w0-453864-openshift-default-77d4cd74c7-bp7wp"
Nov 02 15:14:44.007194 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:14:44.003394    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73b5a23d-1e75-4beb-920e-8953826899af-workload-socket\") pod \"e2e-shared-b-w0-453864-openshift-default-77d4cd74c7-bp7wp\" (UID: \"73b5a23d-1e75-4beb-920e-8953826899af\") " pod="openshift-ingress/e2e-shared-b-w0-453864-openshift-default-77d4cd74c7-bp7wp"
Nov 02 15:14:44.007194 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:14:44.003501    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73b5a23d-1e75-4beb-920e-8953826899af-istio-envoy\") pod \"e2e-shared-b-w0-453864-openshift-default-77d4cd74c7-bp7wp\" (UID: \"73b5a23d-1e75-4beb-920e-8953826899af\") " pod="openshift-ingress/e2e-shared-b-w0-453864-openshift-default-77d4cd74c7-bp7wp"
Nov 02 15:14:44.007194 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:14:44.003543    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73b5a23d-1e75-4beb-920e-8953826899af-istio-data\") pod \"e2e-shared-b-w0-453864-openshift-default-77d4cd74c7-bp7wp\" (UID: \"73b5a23d-1e75-4beb-920e-8953826899af\") " pod="openshift-ingress/e2e-shared-b-w0-453864-openshift-default-77d4cd74c7-bp7wp"
Nov 02 15:14:44.007194 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:14:44.003570    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jspkf\" (UniqueName: \"kubernetes.io/projected/73b5a23d-1e75-4beb-920e-8953826899af-kube-api-access-jspkf\") pod \"e2e-shared-b-w0-453864-openshift-default-77d4cd74c7-bp7wp\" (UID: \"73b5a23d-1e75-4beb-920e-8953826899af\") " pod="openshift-ingress/e2e-shared-b-w0-453864-openshift-default-77d4cd74c7-bp7wp"
Nov 02 15:14:44.007194 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:14:44.003634    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73b5a23d-1e75-4beb-920e-8953826899af-istiod-ca-cert\") pod \"e2e-shared-b-w0-453864-openshift-default-77d4cd74c7-bp7wp\" (UID: \"73b5a23d-1e75-4beb-920e-8953826899af\") " pod="openshift-ingress/e2e-shared-b-w0-453864-openshift-default-77d4cd74c7-bp7wp"
Nov 02 15:14:44.007194 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:14:44.003663    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73b5a23d-1e75-4beb-920e-8953826899af-workload-certs\") pod \"e2e-shared-b-w0-453864-openshift-default-77d4cd74c7-bp7wp\" (UID: \"73b5a23d-1e75-4beb-920e-8953826899af\") " pod="openshift-ingress/e2e-shared-b-w0-453864-openshift-default-77d4cd74c7-bp7wp"
Nov 02 15:14:44.007194 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:14:44.003747    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/73b5a23d-1e75-4beb-920e-8953826899af-istio-token\") pod \"e2e-shared-b-w0-453864-openshift-default-77d4cd74c7-bp7wp\" (UID: \"73b5a23d-1e75-4beb-920e-8953826899af\") " pod="openshift-ingress/e2e-shared-b-w0-453864-openshift-default-77d4cd74c7-bp7wp"
Nov 02 15:14:44.007194 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:14:44.004711    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73b5a23d-1e75-4beb-920e-8953826899af-credential-socket\") pod \"e2e-shared-b-w0-453864-openshift-default-77d4cd74c7-bp7wp\" (UID: \"73b5a23d-1e75-4beb-920e-8953826899af\") " pod="openshift-ingress/e2e-shared-b-w0-453864-openshift-default-77d4cd74c7-bp7wp"
Nov 02 15:14:44.020150 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:14:44.014564    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73b5a23d-1e75-4beb-920e-8953826899af-istio-data\") pod \"e2e-shared-b-w0-453864-openshift-default-77d4cd74c7-bp7wp\" (UID: \"73b5a23d-1e75-4beb-920e-8953826899af\") " pod="openshift-ingress/e2e-shared-b-w0-453864-openshift-default-77d4cd74c7-bp7wp"
Nov 02 15:14:44.020150 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:14:44.014853    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73b5a23d-1e75-4beb-920e-8953826899af-workload-socket\") pod \"e2e-shared-b-w0-453864-openshift-default-77d4cd74c7-bp7wp\" (UID: \"73b5a23d-1e75-4beb-920e-8953826899af\") " pod="openshift-ingress/e2e-shared-b-w0-453864-openshift-default-77d4cd74c7-bp7wp"
Nov 02 15:14:44.040157 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:14:44.040065    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73b5a23d-1e75-4beb-920e-8953826899af-istiod-ca-cert\") pod \"e2e-shared-b-w0-453864-openshift-default-77d4cd74c7-bp7wp\" (UID: \"73b5a23d-1e75-4beb-920e-8953826899af\") " pod="openshift-ingress/e2e-shared-b-w0-453864-openshift-default-77d4cd74c7-bp7wp"
Nov 02 15:14:44.066152 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:14:44.040485    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73b5a23d-1e75-4beb-920e-8953826899af-workload-certs\") pod \"e2e-shared-b-w0-453864-openshift-default-77d4cd74c7-bp7wp\" (UID: \"73b5a23d-1e75-4beb-920e-8953826899af\") " pod="openshift-ingress/e2e-shared-b-w0-453864-openshift-default-77d4cd74c7-bp7wp"
Nov 02 15:14:44.074154 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:14:44.069760    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/73b5a23d-1e75-4beb-920e-8953826899af-istio-token\") pod \"e2e-shared-b-w0-453864-openshift-default-77d4cd74c7-bp7wp\" (UID: \"73b5a23d-1e75-4beb-920e-8953826899af\") " pod="openshift-ingress/e2e-shared-b-w0-453864-openshift-default-77d4cd74c7-bp7wp"
Nov 02 15:14:44.074154 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:14:44.070413    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73b5a23d-1e75-4beb-920e-8953826899af-istio-envoy\") pod \"e2e-shared-b-w0-453864-openshift-default-77d4cd74c7-bp7wp\" (UID: \"73b5a23d-1e75-4beb-920e-8953826899af\") " pod="openshift-ingress/e2e-shared-b-w0-453864-openshift-default-77d4cd74c7-bp7wp"
Nov 02 15:14:44.074154 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:14:44.074085    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73b5a23d-1e75-4beb-920e-8953826899af-istio-podinfo\") pod \"e2e-shared-b-w0-453864-openshift-default-77d4cd74c7-bp7wp\" (UID: \"73b5a23d-1e75-4beb-920e-8953826899af\") " pod="openshift-ingress/e2e-shared-b-w0-453864-openshift-default-77d4cd74c7-bp7wp"
Nov 02 15:14:44.078148 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:14:44.076948    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jspkf\" (UniqueName: \"kubernetes.io/projected/73b5a23d-1e75-4beb-920e-8953826899af-kube-api-access-jspkf\") pod \"e2e-shared-b-w0-453864-openshift-default-77d4cd74c7-bp7wp\" (UID: \"73b5a23d-1e75-4beb-920e-8953826899af\") " pod="openshift-ingress/e2e-shared-b-w0-453864-openshift-default-77d4cd74c7-bp7wp"
Nov 02 15:14:44.199152 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:14:44.198159    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-453864-openshift-default-77d4cd74c7-bp7wp"
Nov 02 15:14:44.615637 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:14:44.615608    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-453864-openshift-default-77d4cd74c7-bp7wp"]
Nov 02 15:14:44.617493 ip-10-0-135-138 kubenswrapper[5304]: W1102 15:14:44.617456    5304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73b5a23d_1e75_4beb_920e_8953826899af.slice/crio-335685da465ea3eb2e51af2a0f31b60452b7db4a047d19f174fb4cd8e9e9bbe4 WatchSource:0}: Error finding container 335685da465ea3eb2e51af2a0f31b60452b7db4a047d19f174fb4cd8e9e9bbe4: Status 404 returned error can't find the container with id 335685da465ea3eb2e51af2a0f31b60452b7db4a047d19f174fb4cd8e9e9bbe4
Nov 02 15:14:44.619966 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:14:44.619926    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:14:44.620066 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:14:44.619995    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:14:44.620066 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:14:44.620023    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:14:44.725183 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:14:44.725101    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-991657be-openshift-default-647b7b77d9-cx5qr" event={"ID":"8626b8a7-0fc5-465d-b071-7b4a7b7fde06","Type":"ContainerStarted","Data":"55e15ec3c0124c5b6b479575c8b7d35af16cda178e477a46abd03f6bfa3c22af"}
Nov 02 15:14:44.727356 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:14:44.727324    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-453864-openshift-default-77d4cd74c7-bp7wp" event={"ID":"73b5a23d-1e75-4beb-920e-8953826899af","Type":"ContainerStarted","Data":"335685da465ea3eb2e51af2a0f31b60452b7db4a047d19f174fb4cd8e9e9bbe4"}
Nov 02 15:14:45.145148 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:14:45.144593    5304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-991657be-openshift-default-647b7b77d9-cx5qr"
Nov 02 15:14:45.145148 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:14:45.144729    5304 patch_prober.go:29] interesting pod/e2e-ait-991657be-openshift-default-647b7b77d9-cx5qr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Nov 02 15:14:45.145148 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:14:45.144774    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-991657be-openshift-default-647b7b77d9-cx5qr" podUID="8626b8a7-0fc5-465d-b071-7b4a7b7fde06" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Nov 02 15:14:45.778431 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:14:45.778389    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-453864-openshift-default-77d4cd74c7-bp7wp" event={"ID":"73b5a23d-1e75-4beb-920e-8953826899af","Type":"ContainerStarted","Data":"8dc3c1116b7a792b9475dfa4ec638acd8ca79da4050680fd317a05e51bd4dd81"}
Nov 02 15:14:45.857568 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:14:45.857498    5304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w0-453864-openshift-default-77d4cd74c7-bp7wp" podStartSLOduration=2.857475763 podStartE2EDuration="2.857475763s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:14:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:14:45.855728953 +0000 UTC m=+1085.849553362" watchObservedRunningTime="2026-11-02 15:14:45.857475763 +0000 UTC m=+1085.851300177"
Nov 02 15:14:46.153150 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:14:46.142045    5304 patch_prober.go:29] interesting pod/e2e-ait-991657be-openshift-default-647b7b77d9-cx5qr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Nov 02 15:14:46.153150 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:14:46.142114    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-991657be-openshift-default-647b7b77d9-cx5qr" podUID="8626b8a7-0fc5-465d-b071-7b4a7b7fde06" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Nov 02 15:14:46.199348 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:14:46.199314    5304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-453864-openshift-default-77d4cd74c7-bp7wp"
Nov 02 15:14:46.205051 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:14:46.205026    5304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-453864-openshift-default-77d4cd74c7-bp7wp"
Nov 02 15:14:46.785711 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:14:46.785681    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-453864-openshift-default-77d4cd74c7-bp7wp"
Nov 02 15:14:46.786715 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:14:46.786694    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-453864-openshift-default-77d4cd74c7-bp7wp"
Nov 02 15:14:47.152813 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:14:47.146528    5304 patch_prober.go:29] interesting pod/e2e-ait-991657be-openshift-default-647b7b77d9-cx5qr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Nov 02 15:14:47.152813 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:14:47.146598    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-991657be-openshift-default-647b7b77d9-cx5qr" podUID="8626b8a7-0fc5-465d-b071-7b4a7b7fde06" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Nov 02 15:14:48.155371 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:14:48.155296    5304 patch_prober.go:29] interesting pod/e2e-ait-991657be-openshift-default-647b7b77d9-cx5qr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Nov 02 15:14:48.155371 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:14:48.155354    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-991657be-openshift-default-647b7b77d9-cx5qr" podUID="8626b8a7-0fc5-465d-b071-7b4a7b7fde06" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Nov 02 15:14:49.156744 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:14:49.156282    5304 patch_prober.go:29] interesting pod/e2e-ait-991657be-openshift-default-647b7b77d9-cx5qr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Nov 02 15:14:49.156744 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:14:49.156340    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-991657be-openshift-default-647b7b77d9-cx5qr" podUID="8626b8a7-0fc5-465d-b071-7b4a7b7fde06" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Nov 02 15:14:50.389715 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:14:50.389680    5304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-991657be-openshift-default-647b7b77d9-cx5qr"
Nov 02 15:14:50.390561 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:14:50.390534    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-991657be-openshift-default-647b7b77d9-cx5qr"
Nov 02 15:14:50.391599 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:14:50.391571    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-991657be-openshift-default-647b7b77d9-cx5qr"
Nov 02 15:14:53.952803 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:14:53.952718    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-991657be-openshift-default-647b7b77d9-cx5qr"]
Nov 02 15:14:53.953309 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:14:53.952947    5304 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-991657be-openshift-default-647b7b77d9-cx5qr" podUID="8626b8a7-0fc5-465d-b071-7b4a7b7fde06" containerName="istio-proxy" containerID="cri-o://55e15ec3c0124c5b6b479575c8b7d35af16cda178e477a46abd03f6bfa3c22af" gracePeriod=30
Nov 02 15:14:54.799955 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:14:54.799916    5304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k6fjk" containerName="extractor"
Nov 02 15:15:00.191282 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:15:00.191229    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5113c2/premium-w6-5113c2-kserve-b84945cb5-pc6f8" podUID="9fb3e992-7bb0-497a-a82c-6e396ba241c1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:15:00.205149 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:15:00.200701    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7ecfe1/distinct-w4-7ecfe1-kserve-6c847c647d-d2jvv" podUID="f1f1ca5e-cca7-4c54-939f-06f095defcdd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:15:00.246708 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:15:00.245737    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5dd8e9/premium-w1-5dd8e9-kserve-67588456fb-j6lph" podUID="4ccc607a-14e4-43ca-8999-78100d4b799f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:15:00.278257 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:15:00.278207    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nkbrk" podUID="1f68d837-fff2-418b-84d7-c2d7eb21a57e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.24:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:15:00.298944 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:15:00.297816    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1cc1b2/premium-w5-1cc1b2-kserve-58db46f97c-cgdqg" podUID="210f620a-8ae7-4e4c-acc3-46f75ddc6105" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded"
Nov 02 15:15:00.301723 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:15:00.299859    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-624254/simulator-w3-624254-kserve-56866b7699-z47sd" podUID="345c710f-b414-4e62-86c7-d51ff4e5be15" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:15:00.602148 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:15:00.600171    5304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893875-4lhwz"]
Nov 02 15:15:00.653738 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:15:00.653696    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5dd8e9/simulator-w1-5dd8e9-kserve-6f9f755d6f-gnw5d" podUID="66b75e64-cb24-40be-9076-4984c132e208" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:15:00.692149 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:15:00.654998    5304 generic.go:356] "Generic (PLEG): container finished" podID="8626b8a7-0fc5-465d-b071-7b4a7b7fde06" containerID="55e15ec3c0124c5b6b479575c8b7d35af16cda178e477a46abd03f6bfa3c22af" exitCode=0
Nov 02 15:15:00.732749 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:15:00.732701    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1cc1b2/distinct-w5-1cc1b2-kserve-5b7459bc6d-zl8xr" podUID="211ad15d-9234-4f57-b1b9-181dd2d74208" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded"
Nov 02 15:15:00.750783 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:15:00.750735    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-991657be-openshift-default-647b7b77d9-cx5qr" event={"ID":"8626b8a7-0fc5-465d-b071-7b4a7b7fde06","Type":"ContainerDied","Data":"55e15ec3c0124c5b6b479575c8b7d35af16cda178e477a46abd03f6bfa3c22af"}
Nov 02 15:15:00.751253 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:15:00.751228    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-991657be-openshift-default-647b7b77d9-cx5qr" event={"ID":"8626b8a7-0fc5-465d-b071-7b4a7b7fde06","Type":"ContainerDied","Data":"16a3b1b71b765b4e64fd36ed5b062317d82e128b6dc41d45d45f20344719610f"}
Nov 02 15:15:00.751443 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:15:00.751429    5304 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="16a3b1b71b765b4e64fd36ed5b062317d82e128b6dc41d45d45f20344719610f"
Nov 02 15:15:00.751555 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:15:00.751388    5304 scope.go:122] "RemoveContainer" containerID="fd35159e7c05710ef9eb630d8e491b7f21472534076a1dd42ef23a158d101e31"
Nov 02 15:15:00.751789 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:15:00.751157    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893875-4lhwz"
Nov 02 15:15:00.781983 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:15:00.780957    5304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-m5926\""
Nov 02 15:15:00.864324 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:15:00.839083    5304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5113c2-29893875-v927s"]
Nov 02 15:15:00.864324 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:15:00.850286    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-991657be-openshift-default-647b7b77d9-cx5qr"
Nov 02 15:15:00.864324 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:15:00.859381    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5113c2-29893875-v927s"
Nov 02 15:15:00.883540 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:15:00.840239    5304 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-6af50f/distinct-w0-6af50f-kserve-c6d994647-hbj9z" podUID="3dca5bf3-8e08-47a0-936e-205670d59ede" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/health\": context deadline exceeded"
Nov 02 15:15:00.953152 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:15:00.952852    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8626b8a7-0fc5-465d-b071-7b4a7b7fde06-kube-api-access-lx6h9\" (UniqueName: \"kubernetes.io/projected/8626b8a7-0fc5-465d-b071-7b4a7b7fde06-kube-api-access-lx6h9\") pod \"8626b8a7-0fc5-465d-b071-7b4a7b7fde06\" (UID: \"8626b8a7-0fc5-465d-b071-7b4a7b7fde06\") "
Nov 02 15:15:00.953152 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:15:00.952935    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8626b8a7-0fc5-465d-b071-7b4a7b7fde06-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8626b8a7-0fc5-465d-b071-7b4a7b7fde06-workload-certs\") pod \"8626b8a7-0fc5-465d-b071-7b4a7b7fde06\" (UID: \"8626b8a7-0fc5-465d-b071-7b4a7b7fde06\") "
Nov 02 15:15:00.953152 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:15:00.953018    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8626b8a7-0fc5-465d-b071-7b4a7b7fde06-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8626b8a7-0fc5-465d-b071-7b4a7b7fde06-istiod-ca-cert\") pod \"8626b8a7-0fc5-465d-b071-7b4a7b7fde06\" (UID: \"8626b8a7-0fc5-465d-b071-7b4a7b7fde06\") "
Nov 02 15:15:00.953152 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:15:00.953092    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8626b8a7-0fc5-465d-b071-7b4a7b7fde06-istio-token\" (UniqueName: \"kubernetes.io/projected/8626b8a7-0fc5-465d-b071-7b4a7b7fde06-istio-token\") pod \"8626b8a7-0fc5-465d-b071-7b4a7b7fde06\" (UID: \"8626b8a7-0fc5-465d-b071-7b4a7b7fde06\") "
Nov 02 15:15:00.953513 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:15:00.953221    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8626b8a7-0fc5-465d-b071-7b4a7b7fde06-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8626b8a7-0fc5-465d-b071-7b4a7b7fde06-credential-socket\") pod \"8626b8a7-0fc5-465d-b071-7b4a7b7fde06\" (UID: \"8626b8a7-0fc5-465d-b071-7b4a7b7fde06\") "
Nov 02 15:15:00.953513 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:15:00.953303    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8626b8a7-0fc5-465d-b071-7b4a7b7fde06-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8626b8a7-0fc5-465d-b071-7b4a7b7fde06-istio-envoy\") pod \"8626b8a7-0fc5-465d-b071-7b4a7b7fde06\" (UID: \"8626b8a7-0fc5-465d-b071-7b4a7b7fde06\") "
Nov 02 15:15:00.953513 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:15:00.953368    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8626b8a7-0fc5-465d-b071-7b4a7b7fde06-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8626b8a7-0fc5-465d-b071-7b4a7b7fde06-istio-podinfo\") pod \"8626b8a7-0fc5-465d-b071-7b4a7b7fde06\" (UID: \"8626b8a7-0fc5-465d-b071-7b4a7b7fde06\") "
Nov 02 15:15:00.953513 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:15:00.953512    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8626b8a7-0fc5-465d-b071-7b4a7b7fde06-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8626b8a7-0fc5-465d-b071-7b4a7b7fde06-istio-data\") pod \"8626b8a7-0fc5-465d-b071-7b4a7b7fde06\" (UID: \"8626b8a7-0fc5-465d-b071-7b4a7b7fde06\") "
Nov 02 15:15:00.953735 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:15:00.953549    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8626b8a7-0fc5-465d-b071-7b4a7b7fde06-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8626b8a7-0fc5-465d-b071-7b4a7b7fde06-workload-socket\") pod \"8626b8a7-0fc5-465d-b071-7b4a7b7fde06\" (UID: \"8626b8a7-0fc5-465d-b071-7b4a7b7fde06\") "
Nov 02 15:15:01.009342 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:15:00.953952    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/48831181-c891-46d9-8d3f-c1a84be4a7f5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-5113c2-29893875-v927s\" (UID: \"48831181-c891-46d9-8d3f-c1a84be4a7f5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5113c2-29893875-v927s"
Nov 02 15:15:01.009342 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:15:00.954036    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4c697\" (UniqueName: \"kubernetes.io/projected/076d977c-0b83-44bb-becb-076e5decc4e6-kube-api-access-4c697\") pod \"maas-api-key-cleanup-29893875-4lhwz\" (UID: \"076d977c-0b83-44bb-becb-076e5decc4e6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893875-4lhwz"
Nov 02 15:15:01.009342 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:15:00.954107    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kwfq2\" (UniqueName: \"kubernetes.io/projected/48831181-c891-46d9-8d3f-c1a84be4a7f5-kube-api-access-kwfq2\") pod \"maas-api-key-cleanup-e2e-worker-w6-5113c2-29893875-v927s\" (UID: \"48831181-c891-46d9-8d3f-c1a84be4a7f5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5113c2-29893875-v927s"
Nov 02 15:15:01.009342 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:15:00.954284    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/076d977c-0b83-44bb-becb-076e5decc4e6-maas-api-service-ca\") pod \"maas-api-key-cleanup-29893875-4lhwz\" (UID: \"076d977c-0b83-44bb-becb-076e5decc4e6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893875-4lhwz"
Nov 02 15:15:01.009342 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:15:00.990365    5304 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-r7jjp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 15:15:01.009342 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:15:00.990439    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-r7jjp" podUID="707b5eda-0575-4b3d-b1d8-f18005ddaa07" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:15:01.009342 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:15:00.995211    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8626b8a7-0fc5-465d-b071-7b4a7b7fde06-workload-socket" pod "8626b8a7-0fc5-465d-b071-7b4a7b7fde06" (UID: "8626b8a7-0fc5-465d-b071-7b4a7b7fde06"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:15:01.009987 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:15:01.009394    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8626b8a7-0fc5-465d-b071-7b4a7b7fde06-istio-podinfo" pod "8626b8a7-0fc5-465d-b071-7b4a7b7fde06" (UID: "8626b8a7-0fc5-465d-b071-7b4a7b7fde06"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 15:15:01.009987 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:15:01.009642    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8626b8a7-0fc5-465d-b071-7b4a7b7fde06-workload-certs" pod "8626b8a7-0fc5-465d-b071-7b4a7b7fde06" (UID: "8626b8a7-0fc5-465d-b071-7b4a7b7fde06"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:15:01.009987 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:15:01.009949    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8626b8a7-0fc5-465d-b071-7b4a7b7fde06-istiod-ca-cert" pod "8626b8a7-0fc5-465d-b071-7b4a7b7fde06" (UID: "8626b8a7-0fc5-465d-b071-7b4a7b7fde06"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:15:01.031259 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:15:01.031229    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8626b8a7-0fc5-465d-b071-7b4a7b7fde06-credential-socket" pod "8626b8a7-0fc5-465d-b071-7b4a7b7fde06" (UID: "8626b8a7-0fc5-465d-b071-7b4a7b7fde06"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:15:01.096151 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:15:01.071769    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4c697\" (UniqueName: \"kubernetes.io/projected/076d977c-0b83-44bb-becb-076e5decc4e6-kube-api-access-4c697\") pod \"maas-api-key-cleanup-29893875-4lhwz\" (UID: \"076d977c-0b83-44bb-becb-076e5decc4e6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893875-4lhwz"
Nov 02 15:15:01.096151 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:15:01.071866    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kwfq2\" (UniqueName: \"kubernetes.io/projected/48831181-c891-46d9-8d3f-c1a84be4a7f5-kube-api-access-kwfq2\") pod \"maas-api-key-cleanup-e2e-worker-w6-5113c2-29893875-v927s\" (UID: \"48831181-c891-46d9-8d3f-c1a84be4a7f5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5113c2-29893875-v927s"
Nov 02 15:15:01.096151 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:15:01.072036    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/076d977c-0b83-44bb-becb-076e5decc4e6-maas-api-service-ca\") pod \"maas-api-key-cleanup-29893875-4lhwz\" (UID: \"076d977c-0b83-44bb-becb-076e5decc4e6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893875-4lhwz"
Nov 02 15:15:01.096151 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:15:01.072321    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/48831181-c891-46d9-8d3f-c1a84be4a7f5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-5113c2-29893875-v927s\" (UID: \"48831181-c891-46d9-8d3f-c1a84be4a7f5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5113c2-29893875-v927s"
Nov 02 15:15:01.096151 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:15:01.072456    5304 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8626b8a7-0fc5-465d-b071-7b4a7b7fde06-workload-certs\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:15:01.096151 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:15:01.072474    5304 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8626b8a7-0fc5-465d-b071-7b4a7b7fde06-istiod-ca-cert\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:15:01.096151 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:15:01.072489    5304 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8626b8a7-0fc5-465d-b071-7b4a7b7fde06-credential-socket\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:15:01.096151 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:15:01.072504    5304 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8626b8a7-0fc5-465d-b071-7b4a7b7fde06-istio-podinfo\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:15:01.096151 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:15:01.072521    5304 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8626b8a7-0fc5-465d-b071-7b4a7b7fde06-workload-socket\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:15:01.096151 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:15:01.073397    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/48831181-c891-46d9-8d3f-c1a84be4a7f5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-5113c2-29893875-v927s\" (UID: \"48831181-c891-46d9-8d3f-c1a84be4a7f5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5113c2-29893875-v927s"
Nov 02 15:15:01.096151 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:15:01.087742    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/076d977c-0b83-44bb-becb-076e5decc4e6-maas-api-service-ca\") pod \"maas-api-key-cleanup-29893875-4lhwz\" (UID: \"076d977c-0b83-44bb-becb-076e5decc4e6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893875-4lhwz"
Nov 02 15:15:01.096151 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:15:01.088328    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6af50f/distinct-w0-6af50f-kserve-c6d994647-hbj9z" podUID="3dca5bf3-8e08-47a0-936e-205670d59ede" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:15:01.154190 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:15:01.145786    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8626b8a7-0fc5-465d-b071-7b4a7b7fde06-kube-api-access-lx6h9" pod "8626b8a7-0fc5-465d-b071-7b4a7b7fde06" (UID: "8626b8a7-0fc5-465d-b071-7b4a7b7fde06"). InnerVolumeSpecName "kube-api-access-lx6h9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:15:01.154373 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:15:01.154303    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8626b8a7-0fc5-465d-b071-7b4a7b7fde06-istio-token" pod "8626b8a7-0fc5-465d-b071-7b4a7b7fde06" (UID: "8626b8a7-0fc5-465d-b071-7b4a7b7fde06"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:15:01.154552 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:15:01.154527    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8626b8a7-0fc5-465d-b071-7b4a7b7fde06-istio-data" pod "8626b8a7-0fc5-465d-b071-7b4a7b7fde06" (UID: "8626b8a7-0fc5-465d-b071-7b4a7b7fde06"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:15:01.171295 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:15:01.161407    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-624254/distinct-w3-624254-kserve-5465dc4765-dmq67" podUID="1a5aa910-a6de-417b-8638-81e4ec5c1825" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:15:01.176456 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:15:01.172943    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-739bf1/simulator-w2-739bf1-kserve-7c74f85465-ltk4t" podUID="cf6a1ded-28a3-49eb-878b-b4ad77a0bb5c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded"
Nov 02 15:15:01.202176 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:15:01.182677    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8626b8a7-0fc5-465d-b071-7b4a7b7fde06-istio-envoy" pod "8626b8a7-0fc5-465d-b071-7b4a7b7fde06" (UID: "8626b8a7-0fc5-465d-b071-7b4a7b7fde06"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:15:01.222252 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:15:01.222213    5304 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-1cc1b2/premium-w5-1cc1b2-kserve-58db46f97c-cgdqg" podUID="210f620a-8ae7-4e4c-acc3-46f75ddc6105" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:15:01.222484 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:15:01.206142    5304 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8626b8a7-0fc5-465d-b071-7b4a7b7fde06-istio-data\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:15:01.222604 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:15:01.222591    5304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lx6h9\" (UniqueName: \"kubernetes.io/projected/8626b8a7-0fc5-465d-b071-7b4a7b7fde06-kube-api-access-lx6h9\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:15:01.222846 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:15:01.222825    5304 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8626b8a7-0fc5-465d-b071-7b4a7b7fde06-istio-token\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:15:01.222986 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:15:01.206813    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5113c2/premium-w6-5113c2-kserve-b84945cb5-pc6f8" podUID="9fb3e992-7bb0-497a-a82c-6e396ba241c1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded"
Nov 02 15:15:01.232205 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:15:01.223416    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-5113c2/premium-w6-5113c2-kserve-b84945cb5-pc6f8"
Nov 02 15:15:01.248157 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:15:01.228754    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7ecfe1/distinct-w4-7ecfe1-kserve-6c847c647d-d2jvv" podUID="f1f1ca5e-cca7-4c54-939f-06f095defcdd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded"
Nov 02 15:15:01.288185 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:15:01.283909    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-7ecfe1/distinct-w4-7ecfe1-kserve-6c847c647d-d2jvv"
Nov 02 15:15:01.288185 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:15:01.228800    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-739bf1/distinct-2-w2-739bf1-kserve-5c58479c48-qgfkh" podUID="9746bf8a-91bd-4715-a02e-ae2e651ca54d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:15:01.288185 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:15:01.269897    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4c697\" (UniqueName: \"kubernetes.io/projected/076d977c-0b83-44bb-becb-076e5decc4e6-kube-api-access-4c697\") pod \"maas-api-key-cleanup-29893875-4lhwz\" (UID: \"076d977c-0b83-44bb-becb-076e5decc4e6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893875-4lhwz"
Nov 02 15:15:01.288185 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:15:01.279358    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nkbrk" podUID="1f68d837-fff2-418b-84d7-c2d7eb21a57e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.24:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:15:01.288185 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:15:01.254375    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5dd8e9/premium-w1-5dd8e9-kserve-67588456fb-j6lph" podUID="4ccc607a-14e4-43ca-8999-78100d4b799f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:15:01.288185 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:15:01.285469    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nkbrk"
Nov 02 15:15:01.288185 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:15:01.285529    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5dd8e9/premium-w1-5dd8e9-kserve-67588456fb-j6lph"
Nov 02 15:15:01.288185 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:15:01.286564    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893875-4lhwz"
Nov 02 15:15:01.303950 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:15:01.302557    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1cc1b2/premium-w5-1cc1b2-kserve-58db46f97c-cgdqg" podUID="210f620a-8ae7-4e4c-acc3-46f75ddc6105" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:15:01.313975 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:15:01.313933    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-1cc1b2/premium-w5-1cc1b2-kserve-58db46f97c-cgdqg"
Nov 02 15:15:01.314445 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:15:01.314395    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893875-4lhwz"]
Nov 02 15:15:01.329374 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:15:01.317612    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-7ecfe1/distinct-w4-7ecfe1-kserve-6c847c647d-d2jvv"
Nov 02 15:15:01.329374 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:15:01.324824    5304 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8626b8a7-0fc5-465d-b071-7b4a7b7fde06-istio-envoy\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:15:01.379196 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:15:01.379164    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kwfq2\" (UniqueName: \"kubernetes.io/projected/48831181-c891-46d9-8d3f-c1a84be4a7f5-kube-api-access-kwfq2\") pod \"maas-api-key-cleanup-e2e-worker-w6-5113c2-29893875-v927s\" (UID: \"48831181-c891-46d9-8d3f-c1a84be4a7f5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5113c2-29893875-v927s"
Nov 02 15:15:01.400215 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:15:01.385952    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-5113c2/premium-w6-5113c2-kserve-b84945cb5-pc6f8"
Nov 02 15:15:01.446470 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:15:01.446438    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5113c2-29893875-v927s"
Nov 02 15:15:01.481185 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:15:01.476231    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5113c2-29893875-v927s"]
Nov 02 15:15:01.656538 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:15:01.656461    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-1cc1b2/premium-w5-1cc1b2-kserve-58db46f97c-cgdqg"
Nov 02 15:15:01.659022 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:15:01.658945    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5dd8e9/premium-w1-5dd8e9-kserve-67588456fb-j6lph"
Nov 02 15:15:01.736518 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:15:01.724062    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nkbrk"
Nov 02 15:15:02.106540 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:15:02.092186    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893875-4lhwz"]
Nov 02 15:15:02.249673 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:15:02.249640    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-991657be-openshift-default-647b7b77d9-cx5qr"
Nov 02 15:15:02.393176 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:15:02.391297    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5113c2-29893875-v927s"]
Nov 02 15:15:02.543709 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:15:02.543678    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-991657be-openshift-default-647b7b77d9-cx5qr"]
Nov 02 15:15:02.549656 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:15:02.549619    5304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-991657be-openshift-default-647b7b77d9-cx5qr"]
Nov 02 15:15:03.199206 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:15:03.198217    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5113c2-29893875-v927s" event={"ID":"48831181-c891-46d9-8d3f-c1a84be4a7f5","Type":"ContainerStarted","Data":"e82ec026dce7af96fa6f11e6ad2025008cca0e17adc1401b5abe85ece18981bd"}
Nov 02 15:15:03.199206 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:15:03.198287    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5113c2-29893875-v927s" event={"ID":"48831181-c891-46d9-8d3f-c1a84be4a7f5","Type":"ContainerStarted","Data":"367c4d6c30fbb64b3836bdcf88849b9970416acd6a7fc933695cc9d6d8409066"}
Nov 02 15:15:03.205154 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:15:03.201496    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893875-4lhwz" event={"ID":"076d977c-0b83-44bb-becb-076e5decc4e6","Type":"ContainerStarted","Data":"bd6b9173ce83ff9ec883b115f1287932aeb9fb46720f57368c8d4a76c7a84b78"}
Nov 02 15:15:03.205154 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:15:03.201556    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893875-4lhwz" event={"ID":"076d977c-0b83-44bb-becb-076e5decc4e6","Type":"ContainerStarted","Data":"685e3060bbd474a1e8317a399ee106f5089a2cce9079bc7c71c4aa5c4bb6eb2c"}
Nov 02 15:15:03.245151 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:15:03.245028    5304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5113c2-29893875-v927s" podStartSLOduration=3.24501011 podStartE2EDuration="3.24501011s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:15:03.240311563 +0000 UTC m=+1103.234135964" watchObservedRunningTime="2026-11-02 15:15:03.24501011 +0000 UTC m=+1103.238834518"
Nov 02 15:15:03.287577 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:15:03.286596    5304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893875-4lhwz" podStartSLOduration=3.286574844 podStartE2EDuration="3.286574844s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:15:03.285055471 +0000 UTC m=+1103.278879881" watchObservedRunningTime="2026-11-02 15:15:03.286574844 +0000 UTC m=+1103.280399248"
Nov 02 15:15:04.122387 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:15:04.122343    5304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8626b8a7-0fc5-465d-b071-7b4a7b7fde06" path="/var/lib/kubelet/pods/8626b8a7-0fc5-465d-b071-7b4a7b7fde06/volumes"
Nov 02 15:15:04.234854 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:15:04.234822    5304 generic.go:356] "Generic (PLEG): container finished" podID="076d977c-0b83-44bb-becb-076e5decc4e6" containerID="bd6b9173ce83ff9ec883b115f1287932aeb9fb46720f57368c8d4a76c7a84b78" exitCode=60
Nov 02 15:15:04.235038 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:15:04.234913    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893875-4lhwz" event={"ID":"076d977c-0b83-44bb-becb-076e5decc4e6","Type":"ContainerDied","Data":"bd6b9173ce83ff9ec883b115f1287932aeb9fb46720f57368c8d4a76c7a84b78"}
Nov 02 15:15:04.235318 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:15:04.235300    5304 scope.go:122] "RemoveContainer" containerID="bd6b9173ce83ff9ec883b115f1287932aeb9fb46720f57368c8d4a76c7a84b78"
Nov 02 15:15:04.237171 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:15:04.237071    5304 generic.go:356] "Generic (PLEG): container finished" podID="48831181-c891-46d9-8d3f-c1a84be4a7f5" containerID="e82ec026dce7af96fa6f11e6ad2025008cca0e17adc1401b5abe85ece18981bd" exitCode=60
Nov 02 15:15:04.237251 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:15:04.237221    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5113c2-29893875-v927s" event={"ID":"48831181-c891-46d9-8d3f-c1a84be4a7f5","Type":"ContainerDied","Data":"e82ec026dce7af96fa6f11e6ad2025008cca0e17adc1401b5abe85ece18981bd"}
Nov 02 15:15:04.237517 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:15:04.237495    5304 scope.go:122] "RemoveContainer" containerID="e82ec026dce7af96fa6f11e6ad2025008cca0e17adc1401b5abe85ece18981bd"
Nov 02 15:15:05.254148 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:15:05.251031    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5113c2-29893875-v927s" event={"ID":"48831181-c891-46d9-8d3f-c1a84be4a7f5","Type":"ContainerStarted","Data":"f4bf7f5f66e5a0f8a439917967674da1018f4e32ecde9dfd9f96521b9e458807"}
Nov 02 15:15:05.254660 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:15:05.254453    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893875-4lhwz" event={"ID":"076d977c-0b83-44bb-becb-076e5decc4e6","Type":"ContainerStarted","Data":"99e21d14129b2e6dcdef9ee19e59a3309cd9f9f8567549424d8d1a29a879e1aa"}
Nov 02 15:15:06.262971 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:15:06.262936    5304 generic.go:356] "Generic (PLEG): container finished" podID="076d977c-0b83-44bb-becb-076e5decc4e6" containerID="99e21d14129b2e6dcdef9ee19e59a3309cd9f9f8567549424d8d1a29a879e1aa" exitCode=60
Nov 02 15:15:06.263465 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:15:06.263014    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893875-4lhwz" event={"ID":"076d977c-0b83-44bb-becb-076e5decc4e6","Type":"ContainerDied","Data":"99e21d14129b2e6dcdef9ee19e59a3309cd9f9f8567549424d8d1a29a879e1aa"}
Nov 02 15:15:06.263465 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:15:06.263078    5304 scope.go:122] "RemoveContainer" containerID="bd6b9173ce83ff9ec883b115f1287932aeb9fb46720f57368c8d4a76c7a84b78"
Nov 02 15:15:06.263465 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:15:06.263432    5304 scope.go:122] "RemoveContainer" containerID="99e21d14129b2e6dcdef9ee19e59a3309cd9f9f8567549424d8d1a29a879e1aa"
Nov 02 15:15:06.263917 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:15:06.263887    5304 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29893875-4lhwz_odh-ai-gateway-infra(076d977c-0b83-44bb-becb-076e5decc4e6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893875-4lhwz" podUID="076d977c-0b83-44bb-becb-076e5decc4e6"
Nov 02 15:15:06.265465 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:15:06.265444    5304 generic.go:356] "Generic (PLEG): container finished" podID="48831181-c891-46d9-8d3f-c1a84be4a7f5" containerID="f4bf7f5f66e5a0f8a439917967674da1018f4e32ecde9dfd9f96521b9e458807" exitCode=60
Nov 02 15:15:06.265577 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:15:06.265505    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5113c2-29893875-v927s" event={"ID":"48831181-c891-46d9-8d3f-c1a84be4a7f5","Type":"ContainerDied","Data":"f4bf7f5f66e5a0f8a439917967674da1018f4e32ecde9dfd9f96521b9e458807"}
Nov 02 15:15:06.265866 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:15:06.265835    5304 scope.go:122] "RemoveContainer" containerID="f4bf7f5f66e5a0f8a439917967674da1018f4e32ecde9dfd9f96521b9e458807"
Nov 02 15:15:06.266204 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:15:06.266185    5304 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-5113c2-29893875-v927s_odh-ai-gateway-infra(48831181-c891-46d9-8d3f-c1a84be4a7f5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5113c2-29893875-v927s" podUID="48831181-c891-46d9-8d3f-c1a84be4a7f5"
Nov 02 15:15:06.277395 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:15:06.277363    5304 scope.go:122] "RemoveContainer" containerID="e82ec026dce7af96fa6f11e6ad2025008cca0e17adc1401b5abe85ece18981bd"
Nov 02 15:15:07.274141 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:15:07.274108    5304 scope.go:122] "RemoveContainer" containerID="99e21d14129b2e6dcdef9ee19e59a3309cd9f9f8567549424d8d1a29a879e1aa"
Nov 02 15:15:07.274597 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:15:07.274534    5304 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29893875-4lhwz_odh-ai-gateway-infra(076d977c-0b83-44bb-becb-076e5decc4e6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893875-4lhwz" podUID="076d977c-0b83-44bb-becb-076e5decc4e6"
Nov 02 15:15:07.275760 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:15:07.275740    5304 scope.go:122] "RemoveContainer" containerID="f4bf7f5f66e5a0f8a439917967674da1018f4e32ecde9dfd9f96521b9e458807"
Nov 02 15:15:07.276053 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:15:07.276036    5304 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-5113c2-29893875-v927s_odh-ai-gateway-infra(48831181-c891-46d9-8d3f-c1a84be4a7f5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5113c2-29893875-v927s" podUID="48831181-c891-46d9-8d3f-c1a84be4a7f5"
Nov 02 15:15:19.112172 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:15:19.112115    5304 scope.go:122] "RemoveContainer" containerID="f4bf7f5f66e5a0f8a439917967674da1018f4e32ecde9dfd9f96521b9e458807"
Nov 02 15:15:19.112653 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:15:19.112610    5304 scope.go:122] "RemoveContainer" containerID="99e21d14129b2e6dcdef9ee19e59a3309cd9f9f8567549424d8d1a29a879e1aa"
Nov 02 15:15:20.382667 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:15:20.382631    5304 generic.go:356] "Generic (PLEG): container finished" podID="076d977c-0b83-44bb-becb-076e5decc4e6" containerID="e4935d939a6818f0235e8b2904e4258a3640a14b04bb72afae5267fa54b5dbbb" exitCode=60
Nov 02 15:15:20.383189 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:15:20.382703    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893875-4lhwz" event={"ID":"076d977c-0b83-44bb-becb-076e5decc4e6","Type":"ContainerDied","Data":"e4935d939a6818f0235e8b2904e4258a3640a14b04bb72afae5267fa54b5dbbb"}
Nov 02 15:15:20.383189 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:15:20.382761    5304 scope.go:122] "RemoveContainer" containerID="99e21d14129b2e6dcdef9ee19e59a3309cd9f9f8567549424d8d1a29a879e1aa"
Nov 02 15:15:20.383309 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:15:20.383282    5304 scope.go:122] "RemoveContainer" containerID="e4935d939a6818f0235e8b2904e4258a3640a14b04bb72afae5267fa54b5dbbb"
Nov 02 15:15:20.383992 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:15:20.383674    5304 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29893875-4lhwz_odh-ai-gateway-infra(076d977c-0b83-44bb-becb-076e5decc4e6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893875-4lhwz" podUID="076d977c-0b83-44bb-becb-076e5decc4e6"
Nov 02 15:15:20.384832 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:15:20.384810    5304 generic.go:356] "Generic (PLEG): container finished" podID="48831181-c891-46d9-8d3f-c1a84be4a7f5" containerID="0615d21da02c14a01d2e7624c6e28559656e84971afc0e1f6119c89625d2bc22" exitCode=60
Nov 02 15:15:20.384943 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:15:20.384886    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5113c2-29893875-v927s" event={"ID":"48831181-c891-46d9-8d3f-c1a84be4a7f5","Type":"ContainerDied","Data":"0615d21da02c14a01d2e7624c6e28559656e84971afc0e1f6119c89625d2bc22"}
Nov 02 15:15:20.385454 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:15:20.385361    5304 scope.go:122] "RemoveContainer" containerID="0615d21da02c14a01d2e7624c6e28559656e84971afc0e1f6119c89625d2bc22"
Nov 02 15:15:20.385731 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:15:20.385707    5304 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-5113c2-29893875-v927s_odh-ai-gateway-infra(48831181-c891-46d9-8d3f-c1a84be4a7f5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5113c2-29893875-v927s" podUID="48831181-c891-46d9-8d3f-c1a84be4a7f5"
Nov 02 15:15:20.396515 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:15:20.396493    5304 scope.go:122] "RemoveContainer" containerID="f4bf7f5f66e5a0f8a439917967674da1018f4e32ecde9dfd9f96521b9e458807"
Nov 02 15:15:21.439864 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:15:21.439834    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893875-4lhwz"]
Nov 02 15:15:21.478165 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:15:21.478118    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5113c2-29893875-v927s"]
Nov 02 15:15:21.646750 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:15:21.646714    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893875-4lhwz"
Nov 02 15:15:21.672958 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:15:21.672915    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/076d977c-0b83-44bb-becb-076e5decc4e6-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/076d977c-0b83-44bb-becb-076e5decc4e6-maas-api-service-ca\") pod \"076d977c-0b83-44bb-becb-076e5decc4e6\" (UID: \"076d977c-0b83-44bb-becb-076e5decc4e6\") "
Nov 02 15:15:21.673173 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:15:21.673013    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/076d977c-0b83-44bb-becb-076e5decc4e6-kube-api-access-4c697\" (UniqueName: \"kubernetes.io/projected/076d977c-0b83-44bb-becb-076e5decc4e6-kube-api-access-4c697\") pod \"076d977c-0b83-44bb-becb-076e5decc4e6\" (UID: \"076d977c-0b83-44bb-becb-076e5decc4e6\") "
Nov 02 15:15:21.673665 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:15:21.673640    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/076d977c-0b83-44bb-becb-076e5decc4e6-maas-api-service-ca" pod "076d977c-0b83-44bb-becb-076e5decc4e6" (UID: "076d977c-0b83-44bb-becb-076e5decc4e6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:15:21.673772 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:15:21.673754    5304 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/076d977c-0b83-44bb-becb-076e5decc4e6-maas-api-service-ca\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:15:21.676099 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:15:21.676072    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/076d977c-0b83-44bb-becb-076e5decc4e6-kube-api-access-4c697" pod "076d977c-0b83-44bb-becb-076e5decc4e6" (UID: "076d977c-0b83-44bb-becb-076e5decc4e6"). InnerVolumeSpecName "kube-api-access-4c697". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:15:21.707977 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:15:21.707903    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5113c2-29893875-v927s"
Nov 02 15:15:21.774666 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:15:21.774632    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48831181-c891-46d9-8d3f-c1a84be4a7f5-kube-api-access-kwfq2\" (UniqueName: \"kubernetes.io/projected/48831181-c891-46d9-8d3f-c1a84be4a7f5-kube-api-access-kwfq2\") pod \"48831181-c891-46d9-8d3f-c1a84be4a7f5\" (UID: \"48831181-c891-46d9-8d3f-c1a84be4a7f5\") "
Nov 02 15:15:21.774838 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:15:21.774774    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/48831181-c891-46d9-8d3f-c1a84be4a7f5-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/48831181-c891-46d9-8d3f-c1a84be4a7f5-maas-api-service-ca\") pod \"48831181-c891-46d9-8d3f-c1a84be4a7f5\" (UID: \"48831181-c891-46d9-8d3f-c1a84be4a7f5\") "
Nov 02 15:15:21.775508 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:15:21.775479    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48831181-c891-46d9-8d3f-c1a84be4a7f5-maas-api-service-ca" pod "48831181-c891-46d9-8d3f-c1a84be4a7f5" (UID: "48831181-c891-46d9-8d3f-c1a84be4a7f5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:15:21.775870 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:15:21.775844    5304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4c697\" (UniqueName: \"kubernetes.io/projected/076d977c-0b83-44bb-becb-076e5decc4e6-kube-api-access-4c697\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:15:21.775969 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:15:21.775876    5304 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/48831181-c891-46d9-8d3f-c1a84be4a7f5-maas-api-service-ca\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:15:21.777609 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:15:21.777582    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48831181-c891-46d9-8d3f-c1a84be4a7f5-kube-api-access-kwfq2" pod "48831181-c891-46d9-8d3f-c1a84be4a7f5" (UID: "48831181-c891-46d9-8d3f-c1a84be4a7f5"). InnerVolumeSpecName "kube-api-access-kwfq2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:15:21.877071 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:15:21.877033    5304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kwfq2\" (UniqueName: \"kubernetes.io/projected/48831181-c891-46d9-8d3f-c1a84be4a7f5-kube-api-access-kwfq2\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:15:22.405467 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:15:22.405373    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5113c2-29893875-v927s" event={"ID":"48831181-c891-46d9-8d3f-c1a84be4a7f5","Type":"ContainerDied","Data":"367c4d6c30fbb64b3836bdcf88849b9970416acd6a7fc933695cc9d6d8409066"}
Nov 02 15:15:22.405467 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:15:22.405441    5304 scope.go:122] "RemoveContainer" containerID="0615d21da02c14a01d2e7624c6e28559656e84971afc0e1f6119c89625d2bc22"
Nov 02 15:15:22.405761 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:15:22.405584    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5113c2-29893875-v927s"
Nov 02 15:15:22.408618 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:15:22.408589    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893875-4lhwz" event={"ID":"076d977c-0b83-44bb-becb-076e5decc4e6","Type":"ContainerDied","Data":"685e3060bbd474a1e8317a399ee106f5089a2cce9079bc7c71c4aa5c4bb6eb2c"}
Nov 02 15:15:22.408759 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:15:22.408690    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893875-4lhwz"
Nov 02 15:15:22.421255 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:15:22.421234    5304 scope.go:122] "RemoveContainer" containerID="e4935d939a6818f0235e8b2904e4258a3640a14b04bb72afae5267fa54b5dbbb"
Nov 02 15:15:22.475794 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:15:22.475177    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5113c2-29893875-v927s"]
Nov 02 15:15:22.481047 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:15:22.480967    5304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5113c2-29893875-v927s"]
Nov 02 15:15:22.509798 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:15:22.509761    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893875-4lhwz"]
Nov 02 15:15:22.513293 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:15:22.513253    5304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893875-4lhwz"]
Nov 02 15:15:24.129175 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:15:24.129116    5304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="076d977c-0b83-44bb-becb-076e5decc4e6" path="/var/lib/kubelet/pods/076d977c-0b83-44bb-becb-076e5decc4e6/volumes"
Nov 02 15:15:24.129794 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:15:24.129773    5304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="48831181-c891-46d9-8d3f-c1a84be4a7f5" path="/var/lib/kubelet/pods/48831181-c891-46d9-8d3f-c1a84be4a7f5/volumes"
Nov 02 15:15:24.800583 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:15:24.800540    5304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k6fjk" containerName="extractor"
Nov 02 15:15:40.563676 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:15:40.563635    5304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w1-fda646-openshift-default-67f4df6b79-vb9vm"]
Nov 02 15:15:40.564999 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:15:40.564971    5304 cpu_manager.go:414] "containerMap: removing container" podUID="8626b8a7-0fc5-465d-b071-7b4a7b7fde06" containerName="istio-proxy"
Nov 02 15:15:40.565208 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:15:40.565190    5304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8626b8a7-0fc5-465d-b071-7b4a7b7fde06" containerName="istio-proxy"
Nov 02 15:15:40.565300 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:15:40.565268    5304 cpu_manager.go:414] "containerMap: removing container" podUID="076d977c-0b83-44bb-becb-076e5decc4e6" containerName="cleanup"
Nov 02 15:15:40.565300 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:15:40.565282    5304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="076d977c-0b83-44bb-becb-076e5decc4e6" containerName="cleanup"
Nov 02 15:15:40.565300 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:15:40.565293    5304 cpu_manager.go:414] "containerMap: removing container" podUID="48831181-c891-46d9-8d3f-c1a84be4a7f5" containerName="cleanup"
Nov 02 15:15:40.565462 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:15:40.565301    5304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48831181-c891-46d9-8d3f-c1a84be4a7f5" containerName="cleanup"
Nov 02 15:15:40.565462 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:15:40.565311    5304 cpu_manager.go:414] "containerMap: removing container" podUID="076d977c-0b83-44bb-becb-076e5decc4e6" containerName="cleanup"
Nov 02 15:15:40.565462 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:15:40.565318    5304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="076d977c-0b83-44bb-becb-076e5decc4e6" containerName="cleanup"
Nov 02 15:15:40.565462 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:15:40.565362    5304 cpu_manager.go:414] "containerMap: removing container" podUID="48831181-c891-46d9-8d3f-c1a84be4a7f5" containerName="cleanup"
Nov 02 15:15:40.565462 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:15:40.565369    5304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48831181-c891-46d9-8d3f-c1a84be4a7f5" containerName="cleanup"
Nov 02 15:15:40.565462 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:15:40.565397    5304 cpu_manager.go:414] "containerMap: removing container" podUID="48831181-c891-46d9-8d3f-c1a84be4a7f5" containerName="cleanup"
Nov 02 15:15:40.565462 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:15:40.565406    5304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48831181-c891-46d9-8d3f-c1a84be4a7f5" containerName="cleanup"
Nov 02 15:15:40.565776 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:15:40.565588    5304 memory_manager.go:365] "RemoveStaleState removing state" podUID="8626b8a7-0fc5-465d-b071-7b4a7b7fde06" containerName="istio-proxy"
Nov 02 15:15:40.565776 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:15:40.565606    5304 memory_manager.go:365] "RemoveStaleState removing state" podUID="076d977c-0b83-44bb-becb-076e5decc4e6" containerName="cleanup"
Nov 02 15:15:40.565776 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:15:40.565615    5304 memory_manager.go:365] "RemoveStaleState removing state" podUID="48831181-c891-46d9-8d3f-c1a84be4a7f5" containerName="cleanup"
Nov 02 15:15:40.565776 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:15:40.565623    5304 memory_manager.go:365] "RemoveStaleState removing state" podUID="076d977c-0b83-44bb-becb-076e5decc4e6" containerName="cleanup"
Nov 02 15:15:40.565776 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:15:40.565637    5304 memory_manager.go:365] "RemoveStaleState removing state" podUID="48831181-c891-46d9-8d3f-c1a84be4a7f5" containerName="cleanup"
Nov 02 15:15:40.572771 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:15:40.572745    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-fda646-openshift-default-67f4df6b79-vb9vm"
Nov 02 15:15:40.590916 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:15:40.590775    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-fda646-openshift-default-67f4df6b79-vb9vm"]
Nov 02 15:15:40.621386 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:15:40.598970    5304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w1-fda646-openshift-default-dockercfg-kdct4\""
Nov 02 15:15:40.621386 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:15:40.615453    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nd5qh\" (UniqueName: \"kubernetes.io/projected/64093f7f-6b31-43af-afe2-ca759bd8ce74-kube-api-access-nd5qh\") pod \"e2e-shared-a-w1-fda646-openshift-default-67f4df6b79-vb9vm\" (UID: \"64093f7f-6b31-43af-afe2-ca759bd8ce74\") " pod="openshift-ingress/e2e-shared-a-w1-fda646-openshift-default-67f4df6b79-vb9vm"
Nov 02 15:15:40.621386 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:15:40.615521    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64093f7f-6b31-43af-afe2-ca759bd8ce74-istio-envoy\") pod \"e2e-shared-a-w1-fda646-openshift-default-67f4df6b79-vb9vm\" (UID: \"64093f7f-6b31-43af-afe2-ca759bd8ce74\") " pod="openshift-ingress/e2e-shared-a-w1-fda646-openshift-default-67f4df6b79-vb9vm"
Nov 02 15:15:40.621386 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:15:40.615662    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64093f7f-6b31-43af-afe2-ca759bd8ce74-istio-podinfo\") pod \"e2e-shared-a-w1-fda646-openshift-default-67f4df6b79-vb9vm\" (UID: \"64093f7f-6b31-43af-afe2-ca759bd8ce74\") " pod="openshift-ingress/e2e-shared-a-w1-fda646-openshift-default-67f4df6b79-vb9vm"
Nov 02 15:15:40.621386 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:15:40.615705    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64093f7f-6b31-43af-afe2-ca759bd8ce74-workload-certs\") pod \"e2e-shared-a-w1-fda646-openshift-default-67f4df6b79-vb9vm\" (UID: \"64093f7f-6b31-43af-afe2-ca759bd8ce74\") " pod="openshift-ingress/e2e-shared-a-w1-fda646-openshift-default-67f4df6b79-vb9vm"
Nov 02 15:15:40.621386 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:15:40.615750    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64093f7f-6b31-43af-afe2-ca759bd8ce74-workload-socket\") pod \"e2e-shared-a-w1-fda646-openshift-default-67f4df6b79-vb9vm\" (UID: \"64093f7f-6b31-43af-afe2-ca759bd8ce74\") " pod="openshift-ingress/e2e-shared-a-w1-fda646-openshift-default-67f4df6b79-vb9vm"
Nov 02 15:15:40.621386 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:15:40.615775    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64093f7f-6b31-43af-afe2-ca759bd8ce74-istiod-ca-cert\") pod \"e2e-shared-a-w1-fda646-openshift-default-67f4df6b79-vb9vm\" (UID: \"64093f7f-6b31-43af-afe2-ca759bd8ce74\") " pod="openshift-ingress/e2e-shared-a-w1-fda646-openshift-default-67f4df6b79-vb9vm"
Nov 02 15:15:40.621386 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:15:40.615904    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/64093f7f-6b31-43af-afe2-ca759bd8ce74-istio-token\") pod \"e2e-shared-a-w1-fda646-openshift-default-67f4df6b79-vb9vm\" (UID: \"64093f7f-6b31-43af-afe2-ca759bd8ce74\") " pod="openshift-ingress/e2e-shared-a-w1-fda646-openshift-default-67f4df6b79-vb9vm"
Nov 02 15:15:40.621386 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:15:40.615947    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64093f7f-6b31-43af-afe2-ca759bd8ce74-credential-socket\") pod \"e2e-shared-a-w1-fda646-openshift-default-67f4df6b79-vb9vm\" (UID: \"64093f7f-6b31-43af-afe2-ca759bd8ce74\") " pod="openshift-ingress/e2e-shared-a-w1-fda646-openshift-default-67f4df6b79-vb9vm"
Nov 02 15:15:40.621386 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:15:40.615991    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64093f7f-6b31-43af-afe2-ca759bd8ce74-istio-data\") pod \"e2e-shared-a-w1-fda646-openshift-default-67f4df6b79-vb9vm\" (UID: \"64093f7f-6b31-43af-afe2-ca759bd8ce74\") " pod="openshift-ingress/e2e-shared-a-w1-fda646-openshift-default-67f4df6b79-vb9vm"
Nov 02 15:15:40.718933 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:15:40.718852    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64093f7f-6b31-43af-afe2-ca759bd8ce74-credential-socket\") pod \"e2e-shared-a-w1-fda646-openshift-default-67f4df6b79-vb9vm\" (UID: \"64093f7f-6b31-43af-afe2-ca759bd8ce74\") " pod="openshift-ingress/e2e-shared-a-w1-fda646-openshift-default-67f4df6b79-vb9vm"
Nov 02 15:15:40.719210 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:15:40.719184    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64093f7f-6b31-43af-afe2-ca759bd8ce74-istio-data\") pod \"e2e-shared-a-w1-fda646-openshift-default-67f4df6b79-vb9vm\" (UID: \"64093f7f-6b31-43af-afe2-ca759bd8ce74\") " pod="openshift-ingress/e2e-shared-a-w1-fda646-openshift-default-67f4df6b79-vb9vm"
Nov 02 15:15:40.719430 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:15:40.719410    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nd5qh\" (UniqueName: \"kubernetes.io/projected/64093f7f-6b31-43af-afe2-ca759bd8ce74-kube-api-access-nd5qh\") pod \"e2e-shared-a-w1-fda646-openshift-default-67f4df6b79-vb9vm\" (UID: \"64093f7f-6b31-43af-afe2-ca759bd8ce74\") " pod="openshift-ingress/e2e-shared-a-w1-fda646-openshift-default-67f4df6b79-vb9vm"
Nov 02 15:15:40.719540 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:15:40.719520    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64093f7f-6b31-43af-afe2-ca759bd8ce74-istio-data\") pod \"e2e-shared-a-w1-fda646-openshift-default-67f4df6b79-vb9vm\" (UID: \"64093f7f-6b31-43af-afe2-ca759bd8ce74\") " pod="openshift-ingress/e2e-shared-a-w1-fda646-openshift-default-67f4df6b79-vb9vm"
Nov 02 15:15:40.719601 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:15:40.719286    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64093f7f-6b31-43af-afe2-ca759bd8ce74-credential-socket\") pod \"e2e-shared-a-w1-fda646-openshift-default-67f4df6b79-vb9vm\" (UID: \"64093f7f-6b31-43af-afe2-ca759bd8ce74\") " pod="openshift-ingress/e2e-shared-a-w1-fda646-openshift-default-67f4df6b79-vb9vm"
Nov 02 15:15:40.719732 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:15:40.719715    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64093f7f-6b31-43af-afe2-ca759bd8ce74-istio-envoy\") pod \"e2e-shared-a-w1-fda646-openshift-default-67f4df6b79-vb9vm\" (UID: \"64093f7f-6b31-43af-afe2-ca759bd8ce74\") " pod="openshift-ingress/e2e-shared-a-w1-fda646-openshift-default-67f4df6b79-vb9vm"
Nov 02 15:15:40.719986 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:15:40.719956    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64093f7f-6b31-43af-afe2-ca759bd8ce74-istio-podinfo\") pod \"e2e-shared-a-w1-fda646-openshift-default-67f4df6b79-vb9vm\" (UID: \"64093f7f-6b31-43af-afe2-ca759bd8ce74\") " pod="openshift-ingress/e2e-shared-a-w1-fda646-openshift-default-67f4df6b79-vb9vm"
Nov 02 15:15:40.720159 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:15:40.720140    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64093f7f-6b31-43af-afe2-ca759bd8ce74-workload-certs\") pod \"e2e-shared-a-w1-fda646-openshift-default-67f4df6b79-vb9vm\" (UID: \"64093f7f-6b31-43af-afe2-ca759bd8ce74\") " pod="openshift-ingress/e2e-shared-a-w1-fda646-openshift-default-67f4df6b79-vb9vm"
Nov 02 15:15:40.720359 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:15:40.720340    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64093f7f-6b31-43af-afe2-ca759bd8ce74-workload-socket\") pod \"e2e-shared-a-w1-fda646-openshift-default-67f4df6b79-vb9vm\" (UID: \"64093f7f-6b31-43af-afe2-ca759bd8ce74\") " pod="openshift-ingress/e2e-shared-a-w1-fda646-openshift-default-67f4df6b79-vb9vm"
Nov 02 15:15:40.720505 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:15:40.720486    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64093f7f-6b31-43af-afe2-ca759bd8ce74-istiod-ca-cert\") pod \"e2e-shared-a-w1-fda646-openshift-default-67f4df6b79-vb9vm\" (UID: \"64093f7f-6b31-43af-afe2-ca759bd8ce74\") " pod="openshift-ingress/e2e-shared-a-w1-fda646-openshift-default-67f4df6b79-vb9vm"
Nov 02 15:15:40.720885 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:15:40.720866    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64093f7f-6b31-43af-afe2-ca759bd8ce74-workload-socket\") pod \"e2e-shared-a-w1-fda646-openshift-default-67f4df6b79-vb9vm\" (UID: \"64093f7f-6b31-43af-afe2-ca759bd8ce74\") " pod="openshift-ingress/e2e-shared-a-w1-fda646-openshift-default-67f4df6b79-vb9vm"
Nov 02 15:15:40.721001 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:15:40.720927    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/64093f7f-6b31-43af-afe2-ca759bd8ce74-istio-token\") pod \"e2e-shared-a-w1-fda646-openshift-default-67f4df6b79-vb9vm\" (UID: \"64093f7f-6b31-43af-afe2-ca759bd8ce74\") " pod="openshift-ingress/e2e-shared-a-w1-fda646-openshift-default-67f4df6b79-vb9vm"
Nov 02 15:15:40.721113 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:15:40.720643    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64093f7f-6b31-43af-afe2-ca759bd8ce74-workload-certs\") pod \"e2e-shared-a-w1-fda646-openshift-default-67f4df6b79-vb9vm\" (UID: \"64093f7f-6b31-43af-afe2-ca759bd8ce74\") " pod="openshift-ingress/e2e-shared-a-w1-fda646-openshift-default-67f4df6b79-vb9vm"
Nov 02 15:15:40.721728 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:15:40.721709    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64093f7f-6b31-43af-afe2-ca759bd8ce74-istiod-ca-cert\") pod \"e2e-shared-a-w1-fda646-openshift-default-67f4df6b79-vb9vm\" (UID: \"64093f7f-6b31-43af-afe2-ca759bd8ce74\") " pod="openshift-ingress/e2e-shared-a-w1-fda646-openshift-default-67f4df6b79-vb9vm"
Nov 02 15:15:40.728767 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:15:40.728733    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64093f7f-6b31-43af-afe2-ca759bd8ce74-istio-podinfo\") pod \"e2e-shared-a-w1-fda646-openshift-default-67f4df6b79-vb9vm\" (UID: \"64093f7f-6b31-43af-afe2-ca759bd8ce74\") " pod="openshift-ingress/e2e-shared-a-w1-fda646-openshift-default-67f4df6b79-vb9vm"
Nov 02 15:15:40.774170 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:15:40.770550    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64093f7f-6b31-43af-afe2-ca759bd8ce74-istio-envoy\") pod \"e2e-shared-a-w1-fda646-openshift-default-67f4df6b79-vb9vm\" (UID: \"64093f7f-6b31-43af-afe2-ca759bd8ce74\") " pod="openshift-ingress/e2e-shared-a-w1-fda646-openshift-default-67f4df6b79-vb9vm"
Nov 02 15:15:40.791849 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:15:40.791813    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/64093f7f-6b31-43af-afe2-ca759bd8ce74-istio-token\") pod \"e2e-shared-a-w1-fda646-openshift-default-67f4df6b79-vb9vm\" (UID: \"64093f7f-6b31-43af-afe2-ca759bd8ce74\") " pod="openshift-ingress/e2e-shared-a-w1-fda646-openshift-default-67f4df6b79-vb9vm"
Nov 02 15:15:40.797835 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:15:40.797811    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nd5qh\" (UniqueName: \"kubernetes.io/projected/64093f7f-6b31-43af-afe2-ca759bd8ce74-kube-api-access-nd5qh\") pod \"e2e-shared-a-w1-fda646-openshift-default-67f4df6b79-vb9vm\" (UID: \"64093f7f-6b31-43af-afe2-ca759bd8ce74\") " pod="openshift-ingress/e2e-shared-a-w1-fda646-openshift-default-67f4df6b79-vb9vm"
Nov 02 15:15:40.912544 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:15:40.912514    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-fda646-openshift-default-67f4df6b79-vb9vm"
Nov 02 15:15:41.260170 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:15:41.256627    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-fda646-openshift-default-67f4df6b79-vb9vm"]
Nov 02 15:15:41.268740 ip-10-0-135-138 kubenswrapper[5304]: W1102 15:15:41.268700    5304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64093f7f_6b31_43af_afe2_ca759bd8ce74.slice/crio-a76e4a98f6dd7f89fcd42c116f53f4741e3f141ff89c682997d46edfb61354d4 WatchSource:0}: Error finding container a76e4a98f6dd7f89fcd42c116f53f4741e3f141ff89c682997d46edfb61354d4: Status 404 returned error can't find the container with id a76e4a98f6dd7f89fcd42c116f53f4741e3f141ff89c682997d46edfb61354d4
Nov 02 15:15:41.271785 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:15:41.271746    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:15:41.271865 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:15:41.271821    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:15:41.271865 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:15:41.271850    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:15:41.686962 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:15:41.686920    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-fda646-openshift-default-67f4df6b79-vb9vm" event={"ID":"64093f7f-6b31-43af-afe2-ca759bd8ce74","Type":"ContainerStarted","Data":"4a779a8180978752aebbfb26cbe6a646dab9dd301d41e3d8ec16c66a6267f975"}
Nov 02 15:15:41.687373 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:15:41.686987    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-fda646-openshift-default-67f4df6b79-vb9vm" event={"ID":"64093f7f-6b31-43af-afe2-ca759bd8ce74","Type":"ContainerStarted","Data":"a76e4a98f6dd7f89fcd42c116f53f4741e3f141ff89c682997d46edfb61354d4"}
Nov 02 15:15:41.756851 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:15:41.756792    5304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w1-fda646-openshift-default-67f4df6b79-vb9vm" podStartSLOduration=1.7567723 podStartE2EDuration="1.7567723s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:15:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:15:41.744019082 +0000 UTC m=+1141.737843502" watchObservedRunningTime="2026-11-02 15:15:41.7567723 +0000 UTC m=+1141.750596712"
Nov 02 15:15:41.913470 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:15:41.913433    5304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-fda646-openshift-default-67f4df6b79-vb9vm"
Nov 02 15:15:42.914584 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:15:42.914545    5304 patch_prober.go:29] interesting pod/e2e-shared-a-w1-fda646-openshift-default-67f4df6b79-vb9vm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body=
Nov 02 15:15:42.915027 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:15:42.914612    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-fda646-openshift-default-67f4df6b79-vb9vm" podUID="64093f7f-6b31-43af-afe2-ca759bd8ce74" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused"
Nov 02 15:15:43.913352 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:15:43.913311    5304 patch_prober.go:29] interesting pod/e2e-shared-a-w1-fda646-openshift-default-67f4df6b79-vb9vm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body=
Nov 02 15:15:43.913562 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:15:43.913371    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-fda646-openshift-default-67f4df6b79-vb9vm" podUID="64093f7f-6b31-43af-afe2-ca759bd8ce74" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused"
Nov 02 15:15:44.912856 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:15:44.912823    5304 patch_prober.go:29] interesting pod/e2e-shared-a-w1-fda646-openshift-default-67f4df6b79-vb9vm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body=
Nov 02 15:15:44.913314 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:15:44.912905    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-fda646-openshift-default-67f4df6b79-vb9vm" podUID="64093f7f-6b31-43af-afe2-ca759bd8ce74" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused"
Nov 02 15:15:45.913832 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:15:45.913789    5304 patch_prober.go:29] interesting pod/e2e-shared-a-w1-fda646-openshift-default-67f4df6b79-vb9vm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body=
Nov 02 15:15:45.914357 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:15:45.913854    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-fda646-openshift-default-67f4df6b79-vb9vm" podUID="64093f7f-6b31-43af-afe2-ca759bd8ce74" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused"
Nov 02 15:15:46.524560 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:15:46.524518    5304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-d7mbd"]
Nov 02 15:15:46.525182 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:15:46.525164    5304 cpu_manager.go:414] "containerMap: removing container" podUID="076d977c-0b83-44bb-becb-076e5decc4e6" containerName="cleanup"
Nov 02 15:15:46.525182 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:15:46.525181    5304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="076d977c-0b83-44bb-becb-076e5decc4e6" containerName="cleanup"
Nov 02 15:15:46.525320 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:15:46.525224    5304 cpu_manager.go:414] "containerMap: removing container" podUID="8626b8a7-0fc5-465d-b071-7b4a7b7fde06" containerName="istio-proxy"
Nov 02 15:15:46.525320 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:15:46.525229    5304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8626b8a7-0fc5-465d-b071-7b4a7b7fde06" containerName="istio-proxy"
Nov 02 15:15:46.525384 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:15:46.525347    5304 memory_manager.go:365] "RemoveStaleState removing state" podUID="48831181-c891-46d9-8d3f-c1a84be4a7f5" containerName="cleanup"
Nov 02 15:15:46.525384 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:15:46.525361    5304 memory_manager.go:365] "RemoveStaleState removing state" podUID="8626b8a7-0fc5-465d-b071-7b4a7b7fde06" containerName="istio-proxy"
Nov 02 15:15:46.525384 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:15:46.525369    5304 memory_manager.go:365] "RemoveStaleState removing state" podUID="076d977c-0b83-44bb-becb-076e5decc4e6" containerName="cleanup"
Nov 02 15:15:46.566768 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:15:46.566733    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-d7mbd"]
Nov 02 15:15:46.567243 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:15:46.567179    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-d7mbd"
Nov 02 15:15:46.650245 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:15:46.649504    5304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-3d7df16e-openshift-default-dockercfg-sn9kd\""
Nov 02 15:15:46.705245 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:15:46.700419    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/294f0d9e-6f89-477c-b882-61af0d1bcb5d-istio-data\") pod \"e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-d7mbd\" (UID: \"294f0d9e-6f89-477c-b882-61af0d1bcb5d\") " pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-d7mbd"
Nov 02 15:15:46.705245 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:15:46.700470    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/294f0d9e-6f89-477c-b882-61af0d1bcb5d-istiod-ca-cert\") pod \"e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-d7mbd\" (UID: \"294f0d9e-6f89-477c-b882-61af0d1bcb5d\") " pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-d7mbd"
Nov 02 15:15:46.705245 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:15:46.700579    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/294f0d9e-6f89-477c-b882-61af0d1bcb5d-istio-podinfo\") pod \"e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-d7mbd\" (UID: \"294f0d9e-6f89-477c-b882-61af0d1bcb5d\") " pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-d7mbd"
Nov 02 15:15:46.705245 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:15:46.700671    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/294f0d9e-6f89-477c-b882-61af0d1bcb5d-istio-envoy\") pod \"e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-d7mbd\" (UID: \"294f0d9e-6f89-477c-b882-61af0d1bcb5d\") " pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-d7mbd"
Nov 02 15:15:46.705245 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:15:46.700732    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/294f0d9e-6f89-477c-b882-61af0d1bcb5d-workload-socket\") pod \"e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-d7mbd\" (UID: \"294f0d9e-6f89-477c-b882-61af0d1bcb5d\") " pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-d7mbd"
Nov 02 15:15:46.705245 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:15:46.700809    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6jt89\" (UniqueName: \"kubernetes.io/projected/294f0d9e-6f89-477c-b882-61af0d1bcb5d-kube-api-access-6jt89\") pod \"e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-d7mbd\" (UID: \"294f0d9e-6f89-477c-b882-61af0d1bcb5d\") " pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-d7mbd"
Nov 02 15:15:46.705245 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:15:46.700888    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/294f0d9e-6f89-477c-b882-61af0d1bcb5d-workload-certs\") pod \"e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-d7mbd\" (UID: \"294f0d9e-6f89-477c-b882-61af0d1bcb5d\") " pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-d7mbd"
Nov 02 15:15:46.705245 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:15:46.700949    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/294f0d9e-6f89-477c-b882-61af0d1bcb5d-istio-token\") pod \"e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-d7mbd\" (UID: \"294f0d9e-6f89-477c-b882-61af0d1bcb5d\") " pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-d7mbd"
Nov 02 15:15:46.705245 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:15:46.701194    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/294f0d9e-6f89-477c-b882-61af0d1bcb5d-credential-socket\") pod \"e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-d7mbd\" (UID: \"294f0d9e-6f89-477c-b882-61af0d1bcb5d\") " pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-d7mbd"
Nov 02 15:15:46.816405 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:15:46.815363    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/294f0d9e-6f89-477c-b882-61af0d1bcb5d-credential-socket\") pod \"e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-d7mbd\" (UID: \"294f0d9e-6f89-477c-b882-61af0d1bcb5d\") " pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-d7mbd"
Nov 02 15:15:46.816405 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:15:46.815459    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/294f0d9e-6f89-477c-b882-61af0d1bcb5d-istio-data\") pod \"e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-d7mbd\" (UID: \"294f0d9e-6f89-477c-b882-61af0d1bcb5d\") " pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-d7mbd"
Nov 02 15:15:46.816405 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:15:46.815486    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/294f0d9e-6f89-477c-b882-61af0d1bcb5d-istiod-ca-cert\") pod \"e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-d7mbd\" (UID: \"294f0d9e-6f89-477c-b882-61af0d1bcb5d\") " pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-d7mbd"
Nov 02 15:15:46.816405 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:15:46.815583    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/294f0d9e-6f89-477c-b882-61af0d1bcb5d-istio-podinfo\") pod \"e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-d7mbd\" (UID: \"294f0d9e-6f89-477c-b882-61af0d1bcb5d\") " pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-d7mbd"
Nov 02 15:15:46.816405 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:15:46.815668    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/294f0d9e-6f89-477c-b882-61af0d1bcb5d-istio-envoy\") pod \"e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-d7mbd\" (UID: \"294f0d9e-6f89-477c-b882-61af0d1bcb5d\") " pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-d7mbd"
Nov 02 15:15:46.816405 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:15:46.815731    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/294f0d9e-6f89-477c-b882-61af0d1bcb5d-workload-socket\") pod \"e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-d7mbd\" (UID: \"294f0d9e-6f89-477c-b882-61af0d1bcb5d\") " pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-d7mbd"
Nov 02 15:15:46.816405 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:15:46.815809    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6jt89\" (UniqueName: \"kubernetes.io/projected/294f0d9e-6f89-477c-b882-61af0d1bcb5d-kube-api-access-6jt89\") pod \"e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-d7mbd\" (UID: \"294f0d9e-6f89-477c-b882-61af0d1bcb5d\") " pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-d7mbd"
Nov 02 15:15:46.816405 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:15:46.815870    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/294f0d9e-6f89-477c-b882-61af0d1bcb5d-workload-certs\") pod \"e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-d7mbd\" (UID: \"294f0d9e-6f89-477c-b882-61af0d1bcb5d\") " pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-d7mbd"
Nov 02 15:15:46.816405 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:15:46.815898    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/294f0d9e-6f89-477c-b882-61af0d1bcb5d-istio-token\") pod \"e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-d7mbd\" (UID: \"294f0d9e-6f89-477c-b882-61af0d1bcb5d\") " pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-d7mbd"
Nov 02 15:15:46.825538 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:15:46.817242    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/294f0d9e-6f89-477c-b882-61af0d1bcb5d-credential-socket\") pod \"e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-d7mbd\" (UID: \"294f0d9e-6f89-477c-b882-61af0d1bcb5d\") " pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-d7mbd"
Nov 02 15:15:46.825538 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:15:46.817521    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/294f0d9e-6f89-477c-b882-61af0d1bcb5d-istio-data\") pod \"e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-d7mbd\" (UID: \"294f0d9e-6f89-477c-b882-61af0d1bcb5d\") " pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-d7mbd"
Nov 02 15:15:46.856961 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:15:46.856433    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/294f0d9e-6f89-477c-b882-61af0d1bcb5d-workload-socket\") pod \"e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-d7mbd\" (UID: \"294f0d9e-6f89-477c-b882-61af0d1bcb5d\") " pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-d7mbd"
Nov 02 15:15:46.936160 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:15:46.890480    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/294f0d9e-6f89-477c-b882-61af0d1bcb5d-workload-certs\") pod \"e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-d7mbd\" (UID: \"294f0d9e-6f89-477c-b882-61af0d1bcb5d\") " pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-d7mbd"
Nov 02 15:15:46.936160 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:15:46.932618    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/294f0d9e-6f89-477c-b882-61af0d1bcb5d-istio-envoy\") pod \"e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-d7mbd\" (UID: \"294f0d9e-6f89-477c-b882-61af0d1bcb5d\") " pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-d7mbd"
Nov 02 15:15:46.970722 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:15:46.942001    5304 patch_prober.go:29] interesting pod/e2e-shared-a-w1-fda646-openshift-default-67f4df6b79-vb9vm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body=
Nov 02 15:15:46.970722 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:15:46.942093    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-fda646-openshift-default-67f4df6b79-vb9vm" podUID="64093f7f-6b31-43af-afe2-ca759bd8ce74" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused"
Nov 02 15:15:46.970722 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:15:46.946819    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6jt89\" (UniqueName: \"kubernetes.io/projected/294f0d9e-6f89-477c-b882-61af0d1bcb5d-kube-api-access-6jt89\") pod \"e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-d7mbd\" (UID: \"294f0d9e-6f89-477c-b882-61af0d1bcb5d\") " pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-d7mbd"
Nov 02 15:15:47.049893 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:15:47.024935    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/294f0d9e-6f89-477c-b882-61af0d1bcb5d-istio-token\") pod \"e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-d7mbd\" (UID: \"294f0d9e-6f89-477c-b882-61af0d1bcb5d\") " pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-d7mbd"
Nov 02 15:15:47.049893 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:15:46.818118    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/294f0d9e-6f89-477c-b882-61af0d1bcb5d-istiod-ca-cert\") pod \"e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-d7mbd\" (UID: \"294f0d9e-6f89-477c-b882-61af0d1bcb5d\") " pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-d7mbd"
Nov 02 15:15:47.049893 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:15:47.028082    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/294f0d9e-6f89-477c-b882-61af0d1bcb5d-istio-podinfo\") pod \"e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-d7mbd\" (UID: \"294f0d9e-6f89-477c-b882-61af0d1bcb5d\") " pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-d7mbd"
Nov 02 15:15:47.199980 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:15:47.199948    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-d7mbd"
Nov 02 15:15:47.411979 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:15:47.411933    5304 patch_prober.go:29] interesting pod/e2e-shared-b-w0-453864-openshift-default-77d4cd74c7-bp7wp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": read tcp 10.133.0.2:55140->10.133.0.78:15021: read: connection reset by peer" start-of-body=
Nov 02 15:15:47.412238 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:15:47.412011    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w0-453864-openshift-default-77d4cd74c7-bp7wp" podUID="73b5a23d-1e75-4beb-920e-8953826899af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": read tcp 10.133.0.2:55140->10.133.0.78:15021: read: connection reset by peer"
Nov 02 15:15:47.603090 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:15:47.603061    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-d7mbd"]
Nov 02 15:15:47.605214 ip-10-0-135-138 kubenswrapper[5304]: W1102 15:15:47.605184    5304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod294f0d9e_6f89_477c_b882_61af0d1bcb5d.slice/crio-67034ee64af4db69ab6d234d96275769bfd004d2d9df1e37e684d8c5e2ca4ead WatchSource:0}: Error finding container 67034ee64af4db69ab6d234d96275769bfd004d2d9df1e37e684d8c5e2ca4ead: Status 404 returned error can't find the container with id 67034ee64af4db69ab6d234d96275769bfd004d2d9df1e37e684d8c5e2ca4ead
Nov 02 15:15:47.607754 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:15:47.607719    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:15:47.607820 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:15:47.607787    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:15:47.607856 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:15:47.607817    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:15:48.038745 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:15:48.038714    5304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-fda646-openshift-default-67f4df6b79-vb9vm"
Nov 02 15:15:48.039503 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:15:48.039478    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-fda646-openshift-default-67f4df6b79-vb9vm"
Nov 02 15:15:48.040714 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:15:48.040685    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-fda646-openshift-default-67f4df6b79-vb9vm"
Nov 02 15:15:48.052991 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:15:48.052944    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-d7mbd" event={"ID":"294f0d9e-6f89-477c-b882-61af0d1bcb5d","Type":"ContainerStarted","Data":"65ae4b6158338164d9f26b447de78665d60d5d9499452284efb07562830609b7"}
Nov 02 15:15:48.053198 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:15:48.053017    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-d7mbd" event={"ID":"294f0d9e-6f89-477c-b882-61af0d1bcb5d","Type":"ContainerStarted","Data":"67034ee64af4db69ab6d234d96275769bfd004d2d9df1e37e684d8c5e2ca4ead"}
Nov 02 15:15:48.055443 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:15:48.055414    5304 generic.go:356] "Generic (PLEG): container finished" podID="73b5a23d-1e75-4beb-920e-8953826899af" containerID="8dc3c1116b7a792b9475dfa4ec638acd8ca79da4050680fd317a05e51bd4dd81" exitCode=0
Nov 02 15:15:48.055561 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:15:48.055480    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-453864-openshift-default-77d4cd74c7-bp7wp" event={"ID":"73b5a23d-1e75-4beb-920e-8953826899af","Type":"ContainerDied","Data":"8dc3c1116b7a792b9475dfa4ec638acd8ca79da4050680fd317a05e51bd4dd81"}
Nov 02 15:15:48.055949 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:15:48.055934    5304 scope.go:122] "RemoveContainer" containerID="8dc3c1116b7a792b9475dfa4ec638acd8ca79da4050680fd317a05e51bd4dd81"
Nov 02 15:15:48.057819 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:15:48.057791    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:15:48.057919 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:15:48.057861    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:15:48.057919 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:15:48.057900    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:15:48.131954 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:15:48.131906    5304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-d7mbd" podStartSLOduration=2.131891574 podStartE2EDuration="2.131891574s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:15:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:15:48.130761597 +0000 UTC m=+1148.124586007" watchObservedRunningTime="2026-11-02 15:15:48.131891574 +0000 UTC m=+1148.125715982"
Nov 02 15:15:48.200363 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:15:48.200324    5304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-d7mbd"
Nov 02 15:15:48.216853 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:15:48.216824    5304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-d7mbd"
Nov 02 15:15:49.064470 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:15:49.064438    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-453864-openshift-default-77d4cd74c7-bp7wp" event={"ID":"73b5a23d-1e75-4beb-920e-8953826899af","Type":"ContainerStarted","Data":"b2f58ba4fa7f872702bd7b3dffcdec3fee1050b47c7dcc372632cd4561c2f665"}
Nov 02 15:15:49.065070 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:15:49.065051    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-d7mbd"
Nov 02 15:15:49.065880 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:15:49.065861    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-d7mbd"
Nov 02 15:15:49.199060 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:15:49.199030    5304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-453864-openshift-default-77d4cd74c7-bp7wp"
Nov 02 15:15:49.199444 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:15:49.199352    5304 patch_prober.go:29] interesting pod/e2e-shared-b-w0-453864-openshift-default-77d4cd74c7-bp7wp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Nov 02 15:15:49.199620 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:15:49.199599    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-453864-openshift-default-77d4cd74c7-bp7wp" podUID="73b5a23d-1e75-4beb-920e-8953826899af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Nov 02 15:15:50.198920 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:15:50.198889    5304 patch_prober.go:29] interesting pod/e2e-shared-b-w0-453864-openshift-default-77d4cd74c7-bp7wp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Nov 02 15:15:50.199322 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:15:50.198949    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-453864-openshift-default-77d4cd74c7-bp7wp" podUID="73b5a23d-1e75-4beb-920e-8953826899af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Nov 02 15:15:51.199037 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:15:51.199005    5304 patch_prober.go:29] interesting pod/e2e-shared-b-w0-453864-openshift-default-77d4cd74c7-bp7wp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Nov 02 15:15:51.199442 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:15:51.199065    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-453864-openshift-default-77d4cd74c7-bp7wp" podUID="73b5a23d-1e75-4beb-920e-8953826899af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Nov 02 15:15:52.198993 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:15:52.198960    5304 patch_prober.go:29] interesting pod/e2e-shared-b-w0-453864-openshift-default-77d4cd74c7-bp7wp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Nov 02 15:15:52.199191 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:15:52.199025    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-453864-openshift-default-77d4cd74c7-bp7wp" podUID="73b5a23d-1e75-4beb-920e-8953826899af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Nov 02 15:15:53.199398 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:15:53.199363    5304 patch_prober.go:29] interesting pod/e2e-shared-b-w0-453864-openshift-default-77d4cd74c7-bp7wp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Nov 02 15:15:53.199801 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:15:53.199447    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-453864-openshift-default-77d4cd74c7-bp7wp" podUID="73b5a23d-1e75-4beb-920e-8953826899af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Nov 02 15:15:54.199744 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:15:54.199314    5304 patch_prober.go:29] interesting pod/e2e-shared-b-w0-453864-openshift-default-77d4cd74c7-bp7wp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Nov 02 15:15:54.199744 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:15:54.199376    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-453864-openshift-default-77d4cd74c7-bp7wp" podUID="73b5a23d-1e75-4beb-920e-8953826899af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Nov 02 15:15:54.978721 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:15:54.978656    5304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k6fjk" containerName="extractor"
Nov 02 15:15:55.231039 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:15:55.230956    5304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-453864-openshift-default-77d4cd74c7-bp7wp"
Nov 02 15:15:55.231881 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:15:55.231857    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-453864-openshift-default-77d4cd74c7-bp7wp"
Nov 02 15:15:55.232601 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:15:55.232573    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-453864-openshift-default-77d4cd74c7-bp7wp"
Nov 02 15:16:00.532738 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:16:00.532698    5304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-w9hxk"]
Nov 02 15:16:00.536869 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:16:00.536841    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-w9hxk"
Nov 02 15:16:00.558816 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:16:00.558781    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-w9hxk"]
Nov 02 15:16:00.564583 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:16:00.564548    5304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w5-246137-openshift-default-dockercfg-hmh98\""
Nov 02 15:16:00.671465 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:16:00.671430    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c41d5c79-8a9b-4ad5-995c-818fd3faf115-istio-data\") pod \"e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-w9hxk\" (UID: \"c41d5c79-8a9b-4ad5-995c-818fd3faf115\") " pod="openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-w9hxk"
Nov 02 15:16:00.671663 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:16:00.671481    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c41d5c79-8a9b-4ad5-995c-818fd3faf115-credential-socket\") pod \"e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-w9hxk\" (UID: \"c41d5c79-8a9b-4ad5-995c-818fd3faf115\") " pod="openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-w9hxk"
Nov 02 15:16:00.671663 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:16:00.671568    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c41d5c79-8a9b-4ad5-995c-818fd3faf115-istio-envoy\") pod \"e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-w9hxk\" (UID: \"c41d5c79-8a9b-4ad5-995c-818fd3faf115\") " pod="openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-w9hxk"
Nov 02 15:16:00.671663 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:16:00.671630    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c41d5c79-8a9b-4ad5-995c-818fd3faf115-workload-socket\") pod \"e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-w9hxk\" (UID: \"c41d5c79-8a9b-4ad5-995c-818fd3faf115\") " pod="openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-w9hxk"
Nov 02 15:16:00.671812 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:16:00.671746    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q5fjx\" (UniqueName: \"kubernetes.io/projected/c41d5c79-8a9b-4ad5-995c-818fd3faf115-kube-api-access-q5fjx\") pod \"e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-w9hxk\" (UID: \"c41d5c79-8a9b-4ad5-995c-818fd3faf115\") " pod="openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-w9hxk"
Nov 02 15:16:00.671868 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:16:00.671818    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c41d5c79-8a9b-4ad5-995c-818fd3faf115-istio-token\") pod \"e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-w9hxk\" (UID: \"c41d5c79-8a9b-4ad5-995c-818fd3faf115\") " pod="openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-w9hxk"
Nov 02 15:16:00.671925 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:16:00.671884    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c41d5c79-8a9b-4ad5-995c-818fd3faf115-istiod-ca-cert\") pod \"e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-w9hxk\" (UID: \"c41d5c79-8a9b-4ad5-995c-818fd3faf115\") " pod="openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-w9hxk"
Nov 02 15:16:00.672020 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:16:00.672003    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c41d5c79-8a9b-4ad5-995c-818fd3faf115-workload-certs\") pod \"e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-w9hxk\" (UID: \"c41d5c79-8a9b-4ad5-995c-818fd3faf115\") " pod="openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-w9hxk"
Nov 02 15:16:00.672067 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:16:00.672028    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c41d5c79-8a9b-4ad5-995c-818fd3faf115-istio-podinfo\") pod \"e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-w9hxk\" (UID: \"c41d5c79-8a9b-4ad5-995c-818fd3faf115\") " pod="openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-w9hxk"
Nov 02 15:16:00.795208 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:16:00.795051    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c41d5c79-8a9b-4ad5-995c-818fd3faf115-istio-data\") pod \"e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-w9hxk\" (UID: \"c41d5c79-8a9b-4ad5-995c-818fd3faf115\") " pod="openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-w9hxk"
Nov 02 15:16:00.795208 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:16:00.795139    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c41d5c79-8a9b-4ad5-995c-818fd3faf115-credential-socket\") pod \"e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-w9hxk\" (UID: \"c41d5c79-8a9b-4ad5-995c-818fd3faf115\") " pod="openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-w9hxk"
Nov 02 15:16:00.795208 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:16:00.795190    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c41d5c79-8a9b-4ad5-995c-818fd3faf115-istio-envoy\") pod \"e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-w9hxk\" (UID: \"c41d5c79-8a9b-4ad5-995c-818fd3faf115\") " pod="openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-w9hxk"
Nov 02 15:16:00.795508 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:16:00.795267    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c41d5c79-8a9b-4ad5-995c-818fd3faf115-workload-socket\") pod \"e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-w9hxk\" (UID: \"c41d5c79-8a9b-4ad5-995c-818fd3faf115\") " pod="openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-w9hxk"
Nov 02 15:16:00.795508 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:16:00.795343    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5fjx\" (UniqueName: \"kubernetes.io/projected/c41d5c79-8a9b-4ad5-995c-818fd3faf115-kube-api-access-q5fjx\") pod \"e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-w9hxk\" (UID: \"c41d5c79-8a9b-4ad5-995c-818fd3faf115\") " pod="openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-w9hxk"
Nov 02 15:16:00.795508 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:16:00.795372    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c41d5c79-8a9b-4ad5-995c-818fd3faf115-istio-token\") pod \"e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-w9hxk\" (UID: \"c41d5c79-8a9b-4ad5-995c-818fd3faf115\") " pod="openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-w9hxk"
Nov 02 15:16:00.795508 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:16:00.795434    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c41d5c79-8a9b-4ad5-995c-818fd3faf115-istiod-ca-cert\") pod \"e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-w9hxk\" (UID: \"c41d5c79-8a9b-4ad5-995c-818fd3faf115\") " pod="openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-w9hxk"
Nov 02 15:16:00.795725 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:16:00.795517    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c41d5c79-8a9b-4ad5-995c-818fd3faf115-workload-certs\") pod \"e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-w9hxk\" (UID: \"c41d5c79-8a9b-4ad5-995c-818fd3faf115\") " pod="openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-w9hxk"
Nov 02 15:16:00.795725 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:16:00.795537    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c41d5c79-8a9b-4ad5-995c-818fd3faf115-istio-data\") pod \"e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-w9hxk\" (UID: \"c41d5c79-8a9b-4ad5-995c-818fd3faf115\") " pod="openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-w9hxk"
Nov 02 15:16:00.795725 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:16:00.795544    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c41d5c79-8a9b-4ad5-995c-818fd3faf115-istio-podinfo\") pod \"e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-w9hxk\" (UID: \"c41d5c79-8a9b-4ad5-995c-818fd3faf115\") " pod="openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-w9hxk"
Nov 02 15:16:00.803254 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:16:00.796221    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c41d5c79-8a9b-4ad5-995c-818fd3faf115-credential-socket\") pod \"e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-w9hxk\" (UID: \"c41d5c79-8a9b-4ad5-995c-818fd3faf115\") " pod="openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-w9hxk"
Nov 02 15:16:00.803254 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:16:00.796809    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c41d5c79-8a9b-4ad5-995c-818fd3faf115-workload-socket\") pod \"e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-w9hxk\" (UID: \"c41d5c79-8a9b-4ad5-995c-818fd3faf115\") " pod="openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-w9hxk"
Nov 02 15:16:00.803254 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:16:00.797475    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c41d5c79-8a9b-4ad5-995c-818fd3faf115-istiod-ca-cert\") pod \"e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-w9hxk\" (UID: \"c41d5c79-8a9b-4ad5-995c-818fd3faf115\") " pod="openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-w9hxk"
Nov 02 15:16:00.803897 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:16:00.803869    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c41d5c79-8a9b-4ad5-995c-818fd3faf115-workload-certs\") pod \"e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-w9hxk\" (UID: \"c41d5c79-8a9b-4ad5-995c-818fd3faf115\") " pod="openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-w9hxk"
Nov 02 15:16:00.899149 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:16:00.874767    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c41d5c79-8a9b-4ad5-995c-818fd3faf115-istio-envoy\") pod \"e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-w9hxk\" (UID: \"c41d5c79-8a9b-4ad5-995c-818fd3faf115\") " pod="openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-w9hxk"
Nov 02 15:16:00.899669 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:16:00.899604    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q5fjx\" (UniqueName: \"kubernetes.io/projected/c41d5c79-8a9b-4ad5-995c-818fd3faf115-kube-api-access-q5fjx\") pod \"e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-w9hxk\" (UID: \"c41d5c79-8a9b-4ad5-995c-818fd3faf115\") " pod="openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-w9hxk"
Nov 02 15:16:00.941045 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:16:00.940925    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c41d5c79-8a9b-4ad5-995c-818fd3faf115-istio-podinfo\") pod \"e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-w9hxk\" (UID: \"c41d5c79-8a9b-4ad5-995c-818fd3faf115\") " pod="openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-w9hxk"
Nov 02 15:16:00.969778 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:16:00.969710    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c41d5c79-8a9b-4ad5-995c-818fd3faf115-istio-token\") pod \"e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-w9hxk\" (UID: \"c41d5c79-8a9b-4ad5-995c-818fd3faf115\") " pod="openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-w9hxk"
Nov 02 15:16:01.158048 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:16:01.158011    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-w9hxk"
Nov 02 15:16:01.519442 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:16:01.519405    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-w9hxk"]
Nov 02 15:16:01.521338 ip-10-0-135-138 kubenswrapper[5304]: W1102 15:16:01.521310    5304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc41d5c79_8a9b_4ad5_995c_818fd3faf115.slice/crio-9b7f4a9f7b3ddeffaa0f8005230c2c54bddebe1fe582fa089c0dff8f69eb8d91 WatchSource:0}: Error finding container 9b7f4a9f7b3ddeffaa0f8005230c2c54bddebe1fe582fa089c0dff8f69eb8d91: Status 404 returned error can't find the container with id 9b7f4a9f7b3ddeffaa0f8005230c2c54bddebe1fe582fa089c0dff8f69eb8d91
Nov 02 15:16:01.524072 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:16:01.524036    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:16:01.524172 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:16:01.524104    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:16:01.524172 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:16:01.524152    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:16:02.163931 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:16:02.163891    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-w9hxk" event={"ID":"c41d5c79-8a9b-4ad5-995c-818fd3faf115","Type":"ContainerStarted","Data":"5726645d5a40c50de38f15f9278fddfc748e28e031f3a911b0055733b1c724f2"}
Nov 02 15:16:02.164340 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:16:02.163954    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-w9hxk" event={"ID":"c41d5c79-8a9b-4ad5-995c-818fd3faf115","Type":"ContainerStarted","Data":"9b7f4a9f7b3ddeffaa0f8005230c2c54bddebe1fe582fa089c0dff8f69eb8d91"}
Nov 02 15:16:02.238880 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:16:02.238825    5304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-w9hxk" podStartSLOduration=2.238811793 podStartE2EDuration="2.238811793s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:16:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:16:02.226151781 +0000 UTC m=+1162.219976190" watchObservedRunningTime="2026-11-02 15:16:02.238811793 +0000 UTC m=+1162.232636202"
Nov 02 15:16:03.159162 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:16:03.159092    5304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-w9hxk"
Nov 02 15:16:03.164987 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:16:03.164958    5304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-w9hxk"
Nov 02 15:16:03.171013 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:16:03.170986    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-w9hxk"
Nov 02 15:16:03.171864 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:16:03.171843    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-w9hxk"
Nov 02 15:16:24.800996 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:16:24.800913    5304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k6fjk" containerName="extractor"
Nov 02 15:16:43.168433 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:16:43.168393    5304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-ad276f-openshift-default-7cb6d54d77-rps78"]
Nov 02 15:16:43.173222 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:16:43.173203    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-ad276f-openshift-default-7cb6d54d77-rps78"
Nov 02 15:16:43.191612 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:16:43.191589    5304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-ad276f-openshift-default-dockercfg-s8d7n\""
Nov 02 15:16:43.205997 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:16:43.205963    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-ad276f-openshift-default-7cb6d54d77-rps78"]
Nov 02 15:16:43.209566 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:16:43.209540    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/350b279b-88a8-4318-b49c-d5001077f702-istiod-ca-cert\") pod \"e2e-api-a-ad276f-openshift-default-7cb6d54d77-rps78\" (UID: \"350b279b-88a8-4318-b49c-d5001077f702\") " pod="openshift-ingress/e2e-api-a-ad276f-openshift-default-7cb6d54d77-rps78"
Nov 02 15:16:43.209767 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:16:43.209743    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/350b279b-88a8-4318-b49c-d5001077f702-istio-podinfo\") pod \"e2e-api-a-ad276f-openshift-default-7cb6d54d77-rps78\" (UID: \"350b279b-88a8-4318-b49c-d5001077f702\") " pod="openshift-ingress/e2e-api-a-ad276f-openshift-default-7cb6d54d77-rps78"
Nov 02 15:16:43.210002 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:16:43.209933    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/350b279b-88a8-4318-b49c-d5001077f702-credential-socket\") pod \"e2e-api-a-ad276f-openshift-default-7cb6d54d77-rps78\" (UID: \"350b279b-88a8-4318-b49c-d5001077f702\") " pod="openshift-ingress/e2e-api-a-ad276f-openshift-default-7cb6d54d77-rps78"
Nov 02 15:16:43.210192 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:16:43.210031    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/350b279b-88a8-4318-b49c-d5001077f702-istio-envoy\") pod \"e2e-api-a-ad276f-openshift-default-7cb6d54d77-rps78\" (UID: \"350b279b-88a8-4318-b49c-d5001077f702\") " pod="openshift-ingress/e2e-api-a-ad276f-openshift-default-7cb6d54d77-rps78"
Nov 02 15:16:43.210192 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:16:43.210073    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/350b279b-88a8-4318-b49c-d5001077f702-istio-token\") pod \"e2e-api-a-ad276f-openshift-default-7cb6d54d77-rps78\" (UID: \"350b279b-88a8-4318-b49c-d5001077f702\") " pod="openshift-ingress/e2e-api-a-ad276f-openshift-default-7cb6d54d77-rps78"
Nov 02 15:16:43.210314 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:16:43.210279    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/350b279b-88a8-4318-b49c-d5001077f702-workload-socket\") pod \"e2e-api-a-ad276f-openshift-default-7cb6d54d77-rps78\" (UID: \"350b279b-88a8-4318-b49c-d5001077f702\") " pod="openshift-ingress/e2e-api-a-ad276f-openshift-default-7cb6d54d77-rps78"
Nov 02 15:16:43.210351 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:16:43.210337    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/350b279b-88a8-4318-b49c-d5001077f702-istio-data\") pod \"e2e-api-a-ad276f-openshift-default-7cb6d54d77-rps78\" (UID: \"350b279b-88a8-4318-b49c-d5001077f702\") " pod="openshift-ingress/e2e-api-a-ad276f-openshift-default-7cb6d54d77-rps78"
Nov 02 15:16:43.210402 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:16:43.210381    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qdp4n\" (UniqueName: \"kubernetes.io/projected/350b279b-88a8-4318-b49c-d5001077f702-kube-api-access-qdp4n\") pod \"e2e-api-a-ad276f-openshift-default-7cb6d54d77-rps78\" (UID: \"350b279b-88a8-4318-b49c-d5001077f702\") " pod="openshift-ingress/e2e-api-a-ad276f-openshift-default-7cb6d54d77-rps78"
Nov 02 15:16:43.210470 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:16:43.210417    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/350b279b-88a8-4318-b49c-d5001077f702-workload-certs\") pod \"e2e-api-a-ad276f-openshift-default-7cb6d54d77-rps78\" (UID: \"350b279b-88a8-4318-b49c-d5001077f702\") " pod="openshift-ingress/e2e-api-a-ad276f-openshift-default-7cb6d54d77-rps78"
Nov 02 15:16:43.318153 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:16:43.314541    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/350b279b-88a8-4318-b49c-d5001077f702-credential-socket\") pod \"e2e-api-a-ad276f-openshift-default-7cb6d54d77-rps78\" (UID: \"350b279b-88a8-4318-b49c-d5001077f702\") " pod="openshift-ingress/e2e-api-a-ad276f-openshift-default-7cb6d54d77-rps78"
Nov 02 15:16:43.318153 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:16:43.314651    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/350b279b-88a8-4318-b49c-d5001077f702-istio-envoy\") pod \"e2e-api-a-ad276f-openshift-default-7cb6d54d77-rps78\" (UID: \"350b279b-88a8-4318-b49c-d5001077f702\") " pod="openshift-ingress/e2e-api-a-ad276f-openshift-default-7cb6d54d77-rps78"
Nov 02 15:16:43.318153 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:16:43.314687    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/350b279b-88a8-4318-b49c-d5001077f702-istio-token\") pod \"e2e-api-a-ad276f-openshift-default-7cb6d54d77-rps78\" (UID: \"350b279b-88a8-4318-b49c-d5001077f702\") " pod="openshift-ingress/e2e-api-a-ad276f-openshift-default-7cb6d54d77-rps78"
Nov 02 15:16:43.318153 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:16:43.314746    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/350b279b-88a8-4318-b49c-d5001077f702-workload-socket\") pod \"e2e-api-a-ad276f-openshift-default-7cb6d54d77-rps78\" (UID: \"350b279b-88a8-4318-b49c-d5001077f702\") " pod="openshift-ingress/e2e-api-a-ad276f-openshift-default-7cb6d54d77-rps78"
Nov 02 15:16:43.318153 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:16:43.314796    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/350b279b-88a8-4318-b49c-d5001077f702-istio-data\") pod \"e2e-api-a-ad276f-openshift-default-7cb6d54d77-rps78\" (UID: \"350b279b-88a8-4318-b49c-d5001077f702\") " pod="openshift-ingress/e2e-api-a-ad276f-openshift-default-7cb6d54d77-rps78"
Nov 02 15:16:43.318153 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:16:43.314838    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qdp4n\" (UniqueName: \"kubernetes.io/projected/350b279b-88a8-4318-b49c-d5001077f702-kube-api-access-qdp4n\") pod \"e2e-api-a-ad276f-openshift-default-7cb6d54d77-rps78\" (UID: \"350b279b-88a8-4318-b49c-d5001077f702\") " pod="openshift-ingress/e2e-api-a-ad276f-openshift-default-7cb6d54d77-rps78"
Nov 02 15:16:43.318153 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:16:43.314868    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/350b279b-88a8-4318-b49c-d5001077f702-workload-certs\") pod \"e2e-api-a-ad276f-openshift-default-7cb6d54d77-rps78\" (UID: \"350b279b-88a8-4318-b49c-d5001077f702\") " pod="openshift-ingress/e2e-api-a-ad276f-openshift-default-7cb6d54d77-rps78"
Nov 02 15:16:43.318153 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:16:43.315005    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/350b279b-88a8-4318-b49c-d5001077f702-istiod-ca-cert\") pod \"e2e-api-a-ad276f-openshift-default-7cb6d54d77-rps78\" (UID: \"350b279b-88a8-4318-b49c-d5001077f702\") " pod="openshift-ingress/e2e-api-a-ad276f-openshift-default-7cb6d54d77-rps78"
Nov 02 15:16:43.318153 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:16:43.315150    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/350b279b-88a8-4318-b49c-d5001077f702-istio-podinfo\") pod \"e2e-api-a-ad276f-openshift-default-7cb6d54d77-rps78\" (UID: \"350b279b-88a8-4318-b49c-d5001077f702\") " pod="openshift-ingress/e2e-api-a-ad276f-openshift-default-7cb6d54d77-rps78"
Nov 02 15:16:43.328708 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:16:43.327998    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/350b279b-88a8-4318-b49c-d5001077f702-istio-data\") pod \"e2e-api-a-ad276f-openshift-default-7cb6d54d77-rps78\" (UID: \"350b279b-88a8-4318-b49c-d5001077f702\") " pod="openshift-ingress/e2e-api-a-ad276f-openshift-default-7cb6d54d77-rps78"
Nov 02 15:16:43.353150 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:16:43.328897    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/350b279b-88a8-4318-b49c-d5001077f702-istiod-ca-cert\") pod \"e2e-api-a-ad276f-openshift-default-7cb6d54d77-rps78\" (UID: \"350b279b-88a8-4318-b49c-d5001077f702\") " pod="openshift-ingress/e2e-api-a-ad276f-openshift-default-7cb6d54d77-rps78"
Nov 02 15:16:43.353150 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:16:43.329184    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/350b279b-88a8-4318-b49c-d5001077f702-credential-socket\") pod \"e2e-api-a-ad276f-openshift-default-7cb6d54d77-rps78\" (UID: \"350b279b-88a8-4318-b49c-d5001077f702\") " pod="openshift-ingress/e2e-api-a-ad276f-openshift-default-7cb6d54d77-rps78"
Nov 02 15:16:43.353150 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:16:43.332067    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/350b279b-88a8-4318-b49c-d5001077f702-workload-certs\") pod \"e2e-api-a-ad276f-openshift-default-7cb6d54d77-rps78\" (UID: \"350b279b-88a8-4318-b49c-d5001077f702\") " pod="openshift-ingress/e2e-api-a-ad276f-openshift-default-7cb6d54d77-rps78"
Nov 02 15:16:43.353150 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:16:43.332319    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/350b279b-88a8-4318-b49c-d5001077f702-workload-socket\") pod \"e2e-api-a-ad276f-openshift-default-7cb6d54d77-rps78\" (UID: \"350b279b-88a8-4318-b49c-d5001077f702\") " pod="openshift-ingress/e2e-api-a-ad276f-openshift-default-7cb6d54d77-rps78"
Nov 02 15:16:43.353150 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:16:43.351500    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/350b279b-88a8-4318-b49c-d5001077f702-istio-envoy\") pod \"e2e-api-a-ad276f-openshift-default-7cb6d54d77-rps78\" (UID: \"350b279b-88a8-4318-b49c-d5001077f702\") " pod="openshift-ingress/e2e-api-a-ad276f-openshift-default-7cb6d54d77-rps78"
Nov 02 15:16:43.364435 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:16:43.363822    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/350b279b-88a8-4318-b49c-d5001077f702-istio-podinfo\") pod \"e2e-api-a-ad276f-openshift-default-7cb6d54d77-rps78\" (UID: \"350b279b-88a8-4318-b49c-d5001077f702\") " pod="openshift-ingress/e2e-api-a-ad276f-openshift-default-7cb6d54d77-rps78"
Nov 02 15:16:43.368852 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:16:43.366765    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/350b279b-88a8-4318-b49c-d5001077f702-istio-token\") pod \"e2e-api-a-ad276f-openshift-default-7cb6d54d77-rps78\" (UID: \"350b279b-88a8-4318-b49c-d5001077f702\") " pod="openshift-ingress/e2e-api-a-ad276f-openshift-default-7cb6d54d77-rps78"
Nov 02 15:16:43.368852 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:16:43.366909    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qdp4n\" (UniqueName: \"kubernetes.io/projected/350b279b-88a8-4318-b49c-d5001077f702-kube-api-access-qdp4n\") pod \"e2e-api-a-ad276f-openshift-default-7cb6d54d77-rps78\" (UID: \"350b279b-88a8-4318-b49c-d5001077f702\") " pod="openshift-ingress/e2e-api-a-ad276f-openshift-default-7cb6d54d77-rps78"
Nov 02 15:16:43.491201 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:16:43.490079    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-ad276f-openshift-default-7cb6d54d77-rps78"
Nov 02 15:16:43.926588 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:16:43.926560    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-ad276f-openshift-default-7cb6d54d77-rps78"]
Nov 02 15:16:43.928173 ip-10-0-135-138 kubenswrapper[5304]: W1102 15:16:43.928143    5304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod350b279b_88a8_4318_b49c_d5001077f702.slice/crio-ae449b9da32762514648fbff42087c42dd7799f09a52a50b005cffa6749ce176 WatchSource:0}: Error finding container ae449b9da32762514648fbff42087c42dd7799f09a52a50b005cffa6749ce176: Status 404 returned error can't find the container with id ae449b9da32762514648fbff42087c42dd7799f09a52a50b005cffa6749ce176
Nov 02 15:16:43.931042 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:16:43.931008    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:16:43.931143 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:16:43.931100    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:16:43.931190 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:16:43.931153    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:16:44.576953 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:16:44.576909    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-ad276f-openshift-default-7cb6d54d77-rps78" event={"ID":"350b279b-88a8-4318-b49c-d5001077f702","Type":"ContainerStarted","Data":"68e340f6a7203903e085d2bee7707621c57f9f1d6b1e7b75aa965438a9764cba"}
Nov 02 15:16:44.577393 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:16:44.577009    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-ad276f-openshift-default-7cb6d54d77-rps78" event={"ID":"350b279b-88a8-4318-b49c-d5001077f702","Type":"ContainerStarted","Data":"ae449b9da32762514648fbff42087c42dd7799f09a52a50b005cffa6749ce176"}
Nov 02 15:16:44.639264 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:16:44.639208    5304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-ad276f-openshift-default-7cb6d54d77-rps78" podStartSLOduration=1.639193457 podStartE2EDuration="1.639193457s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:16:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:16:44.635933992 +0000 UTC m=+1204.629758400" watchObservedRunningTime="2026-11-02 15:16:44.639193457 +0000 UTC m=+1204.633017865"
Nov 02 15:16:45.491203 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:16:45.491169    5304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-ad276f-openshift-default-7cb6d54d77-rps78"
Nov 02 15:16:45.497162 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:16:45.497119    5304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-ad276f-openshift-default-7cb6d54d77-rps78"
Nov 02 15:16:45.584264 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:16:45.584229    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-ad276f-openshift-default-7cb6d54d77-rps78"
Nov 02 15:16:45.585230 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:16:45.585211    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-ad276f-openshift-default-7cb6d54d77-rps78"
Nov 02 15:16:48.839338 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:16:48.839299    5304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-ad276f-64f66d7895-vxlcf"]
Nov 02 15:16:48.843799 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:16:48.843782    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ad276f-64f66d7895-vxlcf"
Nov 02 15:16:48.866656 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:16:48.866623    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-ad276f-64f66d7895-vxlcf"]
Nov 02 15:16:48.873399 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:16:48.871371    5304 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-ad276f\""
Nov 02 15:16:48.873399 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:16:48.871452    5304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-ad276f-dockercfg-vf9m8\""
Nov 02 15:16:48.916488 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:16:48.916456    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2gsp4\" (UniqueName: \"kubernetes.io/projected/4f4d49b5-6c2a-4869-a30a-155e3afe755f-kube-api-access-2gsp4\") pod \"payload-pre-processing-e2e-api-a-ad276f-64f66d7895-vxlcf\" (UID: \"4f4d49b5-6c2a-4869-a30a-155e3afe755f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-ad276f-64f66d7895-vxlcf"
Nov 02 15:16:48.916684 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:16:48.916662    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f4d49b5-6c2a-4869-a30a-155e3afe755f-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-ad276f-64f66d7895-vxlcf\" (UID: \"4f4d49b5-6c2a-4869-a30a-155e3afe755f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-ad276f-64f66d7895-vxlcf"
Nov 02 15:16:49.034456 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:16:49.034412    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f4d49b5-6c2a-4869-a30a-155e3afe755f-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-ad276f-64f66d7895-vxlcf\" (UID: \"4f4d49b5-6c2a-4869-a30a-155e3afe755f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-ad276f-64f66d7895-vxlcf"
Nov 02 15:16:49.034819 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:16:49.034751    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2gsp4\" (UniqueName: \"kubernetes.io/projected/4f4d49b5-6c2a-4869-a30a-155e3afe755f-kube-api-access-2gsp4\") pod \"payload-pre-processing-e2e-api-a-ad276f-64f66d7895-vxlcf\" (UID: \"4f4d49b5-6c2a-4869-a30a-155e3afe755f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-ad276f-64f66d7895-vxlcf"
Nov 02 15:16:49.048152 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:16:49.044523    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f4d49b5-6c2a-4869-a30a-155e3afe755f-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-ad276f-64f66d7895-vxlcf\" (UID: \"4f4d49b5-6c2a-4869-a30a-155e3afe755f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-ad276f-64f66d7895-vxlcf"
Nov 02 15:16:49.057296 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:16:49.057250    5304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-ad276f-76945ff5-9gg4w"]
Nov 02 15:16:49.086288 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:16:49.086058    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2gsp4\" (UniqueName: \"kubernetes.io/projected/4f4d49b5-6c2a-4869-a30a-155e3afe755f-kube-api-access-2gsp4\") pod \"payload-pre-processing-e2e-api-a-ad276f-64f66d7895-vxlcf\" (UID: \"4f4d49b5-6c2a-4869-a30a-155e3afe755f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-ad276f-64f66d7895-vxlcf"
Nov 02 15:16:49.137196 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:16:49.134751    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-ad276f-76945ff5-9gg4w"
Nov 02 15:16:49.162720 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:16:49.162686    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ad276f-64f66d7895-vxlcf"
Nov 02 15:16:49.217169 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:16:49.211471    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-ad276f-76945ff5-9gg4w"]
Nov 02 15:16:49.263973 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:16:49.263924    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca337066-9ffb-4c9e-beee-fe339577a0b7-plugins-config-volume\") pod \"payload-processing-e2e-api-a-ad276f-76945ff5-9gg4w\" (UID: \"ca337066-9ffb-4c9e-beee-fe339577a0b7\") " pod="openshift-ingress/payload-processing-e2e-api-a-ad276f-76945ff5-9gg4w"
Nov 02 15:16:49.264466 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:16:49.264401    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l9ct2\" (UniqueName: \"kubernetes.io/projected/ca337066-9ffb-4c9e-beee-fe339577a0b7-kube-api-access-l9ct2\") pod \"payload-processing-e2e-api-a-ad276f-76945ff5-9gg4w\" (UID: \"ca337066-9ffb-4c9e-beee-fe339577a0b7\") " pod="openshift-ingress/payload-processing-e2e-api-a-ad276f-76945ff5-9gg4w"
Nov 02 15:16:49.366675 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:16:49.366636    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca337066-9ffb-4c9e-beee-fe339577a0b7-plugins-config-volume\") pod \"payload-processing-e2e-api-a-ad276f-76945ff5-9gg4w\" (UID: \"ca337066-9ffb-4c9e-beee-fe339577a0b7\") " pod="openshift-ingress/payload-processing-e2e-api-a-ad276f-76945ff5-9gg4w"
Nov 02 15:16:49.366868 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:16:49.366780    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l9ct2\" (UniqueName: \"kubernetes.io/projected/ca337066-9ffb-4c9e-beee-fe339577a0b7-kube-api-access-l9ct2\") pod \"payload-processing-e2e-api-a-ad276f-76945ff5-9gg4w\" (UID: \"ca337066-9ffb-4c9e-beee-fe339577a0b7\") " pod="openshift-ingress/payload-processing-e2e-api-a-ad276f-76945ff5-9gg4w"
Nov 02 15:16:49.368212 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:16:49.368113    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca337066-9ffb-4c9e-beee-fe339577a0b7-plugins-config-volume\") pod \"payload-processing-e2e-api-a-ad276f-76945ff5-9gg4w\" (UID: \"ca337066-9ffb-4c9e-beee-fe339577a0b7\") " pod="openshift-ingress/payload-processing-e2e-api-a-ad276f-76945ff5-9gg4w"
Nov 02 15:16:49.391549 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:16:49.391471    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l9ct2\" (UniqueName: \"kubernetes.io/projected/ca337066-9ffb-4c9e-beee-fe339577a0b7-kube-api-access-l9ct2\") pod \"payload-processing-e2e-api-a-ad276f-76945ff5-9gg4w\" (UID: \"ca337066-9ffb-4c9e-beee-fe339577a0b7\") " pod="openshift-ingress/payload-processing-e2e-api-a-ad276f-76945ff5-9gg4w"
Nov 02 15:16:49.591021 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:16:49.590982    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-ad276f-64f66d7895-vxlcf"]
Nov 02 15:16:49.593689 ip-10-0-135-138 kubenswrapper[5304]: W1102 15:16:49.593658    5304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f4d49b5_6c2a_4869_a30a_155e3afe755f.slice/crio-0ab37f5595190ae92d547bb683a08a1354ae28afb84fe6a1b9759a6d87b111a3 WatchSource:0}: Error finding container 0ab37f5595190ae92d547bb683a08a1354ae28afb84fe6a1b9759a6d87b111a3: Status 404 returned error can't find the container with id 0ab37f5595190ae92d547bb683a08a1354ae28afb84fe6a1b9759a6d87b111a3
Nov 02 15:16:49.604595 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:16:49.604567    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-ad276f-76945ff5-9gg4w"
Nov 02 15:16:49.619688 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:16:49.619659    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ad276f-64f66d7895-vxlcf" event={"ID":"4f4d49b5-6c2a-4869-a30a-155e3afe755f","Type":"ContainerStarted","Data":"0ab37f5595190ae92d547bb683a08a1354ae28afb84fe6a1b9759a6d87b111a3"}
Nov 02 15:16:49.927156 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:16:49.927117    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-ad276f-76945ff5-9gg4w"]
Nov 02 15:16:49.929068 ip-10-0-135-138 kubenswrapper[5304]: W1102 15:16:49.929040    5304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca337066_9ffb_4c9e_beee_fe339577a0b7.slice/crio-aeb4600865f86f39372fe139dcd026af63d6b15cd66da7686b299d2495393770 WatchSource:0}: Error finding container aeb4600865f86f39372fe139dcd026af63d6b15cd66da7686b299d2495393770: Status 404 returned error can't find the container with id aeb4600865f86f39372fe139dcd026af63d6b15cd66da7686b299d2495393770
Nov 02 15:16:50.637929 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:16:50.637878    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ad276f-64f66d7895-vxlcf" event={"ID":"4f4d49b5-6c2a-4869-a30a-155e3afe755f","Type":"ContainerStarted","Data":"4de0394a7602536646e88be3e3ec39e6c2be0b60d021c9c80d111f576d2e57e1"}
Nov 02 15:16:50.638172 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:16:50.637991    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ad276f-64f66d7895-vxlcf"
Nov 02 15:16:50.640023 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:16:50.639976    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-ad276f-76945ff5-9gg4w" event={"ID":"ca337066-9ffb-4c9e-beee-fe339577a0b7","Type":"ContainerStarted","Data":"c2a3ecee95ed737f278b090569e9855674675b0f9f3a0bb5251097ad09e029c3"}
Nov 02 15:16:50.640192 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:16:50.640049    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-ad276f-76945ff5-9gg4w" event={"ID":"ca337066-9ffb-4c9e-beee-fe339577a0b7","Type":"ContainerStarted","Data":"aeb4600865f86f39372fe139dcd026af63d6b15cd66da7686b299d2495393770"}
Nov 02 15:16:50.640192 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:16:50.640115    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-ad276f-76945ff5-9gg4w"
Nov 02 15:16:50.680441 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:16:50.680386    5304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ad276f-64f66d7895-vxlcf" podStartSLOduration=2.680373228 podStartE2EDuration="2.680373228s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:16:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:16:50.677825539 +0000 UTC m=+1210.671649947" watchObservedRunningTime="2026-11-02 15:16:50.680373228 +0000 UTC m=+1210.674197682"
Nov 02 15:16:50.722543 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:16:50.722484    5304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-ad276f-76945ff5-9gg4w" podStartSLOduration=2.722470397 podStartE2EDuration="2.722470397s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:16:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:16:50.718358941 +0000 UTC m=+1210.712183351" watchObservedRunningTime="2026-11-02 15:16:50.722470397 +0000 UTC m=+1210.716294806"
Nov 02 15:16:54.824040 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:16:54.823997    5304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k6fjk" containerName="extractor"
Nov 02 15:17:01.701201 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:01.700250    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-ad276f-76945ff5-9gg4w"
Nov 02 15:17:01.706467 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:01.706441    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ad276f-64f66d7895-vxlcf"
Nov 02 15:17:04.212360 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:04.212277    5304 patch_prober.go:29] interesting pod/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-w9hxk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 15:17:04.213255 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:04.212372    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-w9hxk" podUID="c41d5c79-8a9b-4ad5-995c-818fd3faf115" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:17:05.106274 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:05.106239    5304 patch_prober.go:29] interesting pod/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-d7mbd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 15:17:05.106484 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:05.106331    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-d7mbd" podUID="294f0d9e-6f89-477c-b882-61af0d1bcb5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:17:10.186856 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:10.186819    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-dq45c"]
Nov 02 15:17:10.196377 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:10.187828    5304 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-dq45c" podUID="dbef5be7-9697-4eb4-b8c7-cd985e58c6b6" containerName="istio-proxy" containerID="cri-o://ac7a2102277c7c786d1251baf6db451b6db5785c9ae1f24d6e55a39c762bb01f" gracePeriod=30
Nov 02 15:17:10.207820 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:10.207787    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-576tm"]
Nov 02 15:17:10.208062 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:10.208039    5304 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-576tm" podUID="fdbb90e5-42c8-4409-95e1-a44ed4abead2" containerName="istio-proxy" containerID="cri-o://9204f7292e82a2a289873d5feea1b81d704edcdf3b90e4892e8a5bc5bd64117d" gracePeriod=30
Nov 02 15:17:10.212632 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:10.212597    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-b88h7"]
Nov 02 15:17:10.212981 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:10.212932    5304 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-b88h7" podUID="3d4f7624-882c-4cb3-8811-9ca682e0c1d5" containerName="istio-proxy" containerID="cri-o://f308b7b6046e7500e8e6d2d6f77c217a3ab3773533f03a01f6b15aa9ca7b1d96" gracePeriod=30
Nov 02 15:17:10.450320 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:10.450236    5304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-0a7163-8dd8847c5-5s25z"]
Nov 02 15:17:10.472594 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:10.472564    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-0a7163-8dd8847c5-5s25z"
Nov 02 15:17:10.481401 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:10.481366    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zw8gd\" (UniqueName: \"kubernetes.io/projected/8e3715e3-1d26-4abf-ad77-6cb9cb70c349-kube-api-access-zw8gd\") pod \"payload-pre-processing-e2e-shared-b-w5-0a7163-8dd8847c5-5s25z\" (UID: \"8e3715e3-1d26-4abf-ad77-6cb9cb70c349\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-0a7163-8dd8847c5-5s25z"
Nov 02 15:17:10.481560 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:10.481406    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e3715e3-1d26-4abf-ad77-6cb9cb70c349-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w5-0a7163-8dd8847c5-5s25z\" (UID: \"8e3715e3-1d26-4abf-ad77-6cb9cb70c349\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-0a7163-8dd8847c5-5s25z"
Nov 02 15:17:10.497809 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:10.497772    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-0a7163-8dd8847c5-5s25z"]
Nov 02 15:17:10.502514 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:10.502482    5304 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w5-0a7163\""
Nov 02 15:17:10.544506 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:10.543163    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-k2p6m"]
Nov 02 15:17:10.544506 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:10.543408    5304 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-k2p6m" podUID="1bd23848-eb1d-45fd-9d87-c11302b3c111" containerName="istio-proxy" containerID="cri-o://d6bc4523c8b8264ec504666897583f330e20731aee7d610dbbf90cd79d0807b9" gracePeriod=30
Nov 02 15:17:10.557010 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:10.556971    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-cwnwk"]
Nov 02 15:17:10.557624 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:10.557531    5304 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-cwnwk" podUID="79e8599a-22fa-4df3-a35a-4d16c271cfe2" containerName="istio-proxy" containerID="cri-o://165e1e14e0dc19782087cbf0db0d14128fb6fd4ec462abe1c9b3ae9d0ea73fe3" gracePeriod=30
Nov 02 15:17:10.605565 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:10.599866    5304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w5-0a7163-dockercfg-qzljc\""
Nov 02 15:17:10.605565 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:10.602853    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zw8gd\" (UniqueName: \"kubernetes.io/projected/8e3715e3-1d26-4abf-ad77-6cb9cb70c349-kube-api-access-zw8gd\") pod \"payload-pre-processing-e2e-shared-b-w5-0a7163-8dd8847c5-5s25z\" (UID: \"8e3715e3-1d26-4abf-ad77-6cb9cb70c349\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-0a7163-8dd8847c5-5s25z"
Nov 02 15:17:10.605565 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:10.602901    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e3715e3-1d26-4abf-ad77-6cb9cb70c349-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w5-0a7163-8dd8847c5-5s25z\" (UID: \"8e3715e3-1d26-4abf-ad77-6cb9cb70c349\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-0a7163-8dd8847c5-5s25z"
Nov 02 15:17:10.701298 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:10.701215    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e3715e3-1d26-4abf-ad77-6cb9cb70c349-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w5-0a7163-8dd8847c5-5s25z\" (UID: \"8e3715e3-1d26-4abf-ad77-6cb9cb70c349\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-0a7163-8dd8847c5-5s25z"
Nov 02 15:17:10.811190 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:10.811108    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zw8gd\" (UniqueName: \"kubernetes.io/projected/8e3715e3-1d26-4abf-ad77-6cb9cb70c349-kube-api-access-zw8gd\") pod \"payload-pre-processing-e2e-shared-b-w5-0a7163-8dd8847c5-5s25z\" (UID: \"8e3715e3-1d26-4abf-ad77-6cb9cb70c349\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-0a7163-8dd8847c5-5s25z"
Nov 02 15:17:10.835364 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:10.835325    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-0a7163-8dd8847c5-5s25z"
Nov 02 15:17:12.564992 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:12.564915    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-0a7163-8dd8847c5-5s25z"]
Nov 02 15:17:12.568852 ip-10-0-135-138 kubenswrapper[5304]: W1102 15:17:12.568810    5304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e3715e3_1d26_4abf_ad77_6cb9cb70c349.slice/crio-45acd77c5ba763c2ce0f6389b936e06e0a18c23ef2de18a9c23dd5fcf9381ea6 WatchSource:0}: Error finding container 45acd77c5ba763c2ce0f6389b936e06e0a18c23ef2de18a9c23dd5fcf9381ea6: Status 404 returned error can't find the container with id 45acd77c5ba763c2ce0f6389b936e06e0a18c23ef2de18a9c23dd5fcf9381ea6
Nov 02 15:17:13.008442 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:13.008397    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-0a7163-8dd8847c5-5s25z" event={"ID":"8e3715e3-1d26-4abf-ad77-6cb9cb70c349","Type":"ContainerStarted","Data":"7dbe891d97b673b8c7dbbf6e8c33f8470fb41bf72e172021845dcf6c75f881ce"}
Nov 02 15:17:13.008644 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:13.008474    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-0a7163-8dd8847c5-5s25z" event={"ID":"8e3715e3-1d26-4abf-ad77-6cb9cb70c349","Type":"ContainerStarted","Data":"45acd77c5ba763c2ce0f6389b936e06e0a18c23ef2de18a9c23dd5fcf9381ea6"}
Nov 02 15:17:13.008644 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:13.008609    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-0a7163-8dd8847c5-5s25z"
Nov 02 15:17:13.055230 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:13.055170    5304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-0a7163-8dd8847c5-5s25z" podStartSLOduration=3.055150559 podStartE2EDuration="3.055150559s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:17:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:17:13.048849601 +0000 UTC m=+1233.042674010" watchObservedRunningTime="2026-11-02 15:17:13.055150559 +0000 UTC m=+1233.048974969"
Nov 02 15:17:15.585107 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:15.585065    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-576tm"
Nov 02 15:17:15.615234 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:15.615209    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-dq45c"
Nov 02 15:17:15.633065 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:15.633037    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fdbb90e5-42c8-4409-95e1-a44ed4abead2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fdbb90e5-42c8-4409-95e1-a44ed4abead2-istio-podinfo\") pod \"fdbb90e5-42c8-4409-95e1-a44ed4abead2\" (UID: \"fdbb90e5-42c8-4409-95e1-a44ed4abead2\") "
Nov 02 15:17:15.633298 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:15.633088    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fdbb90e5-42c8-4409-95e1-a44ed4abead2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fdbb90e5-42c8-4409-95e1-a44ed4abead2-workload-certs\") pod \"fdbb90e5-42c8-4409-95e1-a44ed4abead2\" (UID: \"fdbb90e5-42c8-4409-95e1-a44ed4abead2\") "
Nov 02 15:17:15.633298 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:15.633189    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fdbb90e5-42c8-4409-95e1-a44ed4abead2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fdbb90e5-42c8-4409-95e1-a44ed4abead2-istiod-ca-cert\") pod \"fdbb90e5-42c8-4409-95e1-a44ed4abead2\" (UID: \"fdbb90e5-42c8-4409-95e1-a44ed4abead2\") "
Nov 02 15:17:15.633298 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:15.633243    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fdbb90e5-42c8-4409-95e1-a44ed4abead2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdbb90e5-42c8-4409-95e1-a44ed4abead2-workload-socket\") pod \"fdbb90e5-42c8-4409-95e1-a44ed4abead2\" (UID: \"fdbb90e5-42c8-4409-95e1-a44ed4abead2\") "
Nov 02 15:17:15.633480 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:15.633322    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fdbb90e5-42c8-4409-95e1-a44ed4abead2-kube-api-access-pmp5b\" (UniqueName: \"kubernetes.io/projected/fdbb90e5-42c8-4409-95e1-a44ed4abead2-kube-api-access-pmp5b\") pod \"fdbb90e5-42c8-4409-95e1-a44ed4abead2\" (UID: \"fdbb90e5-42c8-4409-95e1-a44ed4abead2\") "
Nov 02 15:17:15.633480 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:15.633401    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fdbb90e5-42c8-4409-95e1-a44ed4abead2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fdbb90e5-42c8-4409-95e1-a44ed4abead2-istio-data\") pod \"fdbb90e5-42c8-4409-95e1-a44ed4abead2\" (UID: \"fdbb90e5-42c8-4409-95e1-a44ed4abead2\") "
Nov 02 15:17:15.633560 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:15.633487    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fdbb90e5-42c8-4409-95e1-a44ed4abead2-istio-token\" (UniqueName: \"kubernetes.io/projected/fdbb90e5-42c8-4409-95e1-a44ed4abead2-istio-token\") pod \"fdbb90e5-42c8-4409-95e1-a44ed4abead2\" (UID: \"fdbb90e5-42c8-4409-95e1-a44ed4abead2\") "
Nov 02 15:17:15.633560 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:15.633516    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fdbb90e5-42c8-4409-95e1-a44ed4abead2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdbb90e5-42c8-4409-95e1-a44ed4abead2-credential-socket\") pod \"fdbb90e5-42c8-4409-95e1-a44ed4abead2\" (UID: \"fdbb90e5-42c8-4409-95e1-a44ed4abead2\") "
Nov 02 15:17:15.633931 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:15.633659    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fdbb90e5-42c8-4409-95e1-a44ed4abead2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fdbb90e5-42c8-4409-95e1-a44ed4abead2-istio-envoy\") pod \"fdbb90e5-42c8-4409-95e1-a44ed4abead2\" (UID: \"fdbb90e5-42c8-4409-95e1-a44ed4abead2\") "
Nov 02 15:17:15.634094 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:15.634060    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fdbb90e5-42c8-4409-95e1-a44ed4abead2-istiod-ca-cert" pod "fdbb90e5-42c8-4409-95e1-a44ed4abead2" (UID: "fdbb90e5-42c8-4409-95e1-a44ed4abead2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:17:15.634532 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:15.634511    5304 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fdbb90e5-42c8-4409-95e1-a44ed4abead2-istiod-ca-cert\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:17:15.634804 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:15.634781    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdbb90e5-42c8-4409-95e1-a44ed4abead2-istio-data" pod "fdbb90e5-42c8-4409-95e1-a44ed4abead2" (UID: "fdbb90e5-42c8-4409-95e1-a44ed4abead2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:17:15.635031 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:15.635014    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdbb90e5-42c8-4409-95e1-a44ed4abead2-credential-socket" pod "fdbb90e5-42c8-4409-95e1-a44ed4abead2" (UID: "fdbb90e5-42c8-4409-95e1-a44ed4abead2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:17:15.646589 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:15.646446    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdbb90e5-42c8-4409-95e1-a44ed4abead2-workload-socket" pod "fdbb90e5-42c8-4409-95e1-a44ed4abead2" (UID: "fdbb90e5-42c8-4409-95e1-a44ed4abead2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:17:15.652389 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:15.652357    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdbb90e5-42c8-4409-95e1-a44ed4abead2-workload-certs" pod "fdbb90e5-42c8-4409-95e1-a44ed4abead2" (UID: "fdbb90e5-42c8-4409-95e1-a44ed4abead2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:17:15.662026 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:15.661997    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdbb90e5-42c8-4409-95e1-a44ed4abead2-istio-envoy" pod "fdbb90e5-42c8-4409-95e1-a44ed4abead2" (UID: "fdbb90e5-42c8-4409-95e1-a44ed4abead2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:17:15.680045 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:15.662297    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fdbb90e5-42c8-4409-95e1-a44ed4abead2-istio-podinfo" pod "fdbb90e5-42c8-4409-95e1-a44ed4abead2" (UID: "fdbb90e5-42c8-4409-95e1-a44ed4abead2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 15:17:15.682114 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:15.681986    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fdbb90e5-42c8-4409-95e1-a44ed4abead2-kube-api-access-pmp5b" pod "fdbb90e5-42c8-4409-95e1-a44ed4abead2" (UID: "fdbb90e5-42c8-4409-95e1-a44ed4abead2"). InnerVolumeSpecName "kube-api-access-pmp5b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:17:15.682683 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:15.682652    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fdbb90e5-42c8-4409-95e1-a44ed4abead2-istio-token" pod "fdbb90e5-42c8-4409-95e1-a44ed4abead2" (UID: "fdbb90e5-42c8-4409-95e1-a44ed4abead2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:17:15.743995 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:15.736250    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dbef5be7-9697-4eb4-b8c7-cd985e58c6b6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dbef5be7-9697-4eb4-b8c7-cd985e58c6b6-workload-certs\") pod \"dbef5be7-9697-4eb4-b8c7-cd985e58c6b6\" (UID: \"dbef5be7-9697-4eb4-b8c7-cd985e58c6b6\") "
Nov 02 15:17:15.743995 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:15.736349    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dbef5be7-9697-4eb4-b8c7-cd985e58c6b6-kube-api-access-ls2fk\" (UniqueName: \"kubernetes.io/projected/dbef5be7-9697-4eb4-b8c7-cd985e58c6b6-kube-api-access-ls2fk\") pod \"dbef5be7-9697-4eb4-b8c7-cd985e58c6b6\" (UID: \"dbef5be7-9697-4eb4-b8c7-cd985e58c6b6\") "
Nov 02 15:17:15.743995 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:15.736644    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dbef5be7-9697-4eb4-b8c7-cd985e58c6b6-istio-token\" (UniqueName: \"kubernetes.io/projected/dbef5be7-9697-4eb4-b8c7-cd985e58c6b6-istio-token\") pod \"dbef5be7-9697-4eb4-b8c7-cd985e58c6b6\" (UID: \"dbef5be7-9697-4eb4-b8c7-cd985e58c6b6\") "
Nov 02 15:17:15.743995 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:15.736679    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbef5be7-9697-4eb4-b8c7-cd985e58c6b6-workload-certs" pod "dbef5be7-9697-4eb4-b8c7-cd985e58c6b6" (UID: "dbef5be7-9697-4eb4-b8c7-cd985e58c6b6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:17:15.743995 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:15.736718    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dbef5be7-9697-4eb4-b8c7-cd985e58c6b6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dbef5be7-9697-4eb4-b8c7-cd985e58c6b6-istiod-ca-cert\") pod \"dbef5be7-9697-4eb4-b8c7-cd985e58c6b6\" (UID: \"dbef5be7-9697-4eb4-b8c7-cd985e58c6b6\") "
Nov 02 15:17:15.743995 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:15.736770    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dbef5be7-9697-4eb4-b8c7-cd985e58c6b6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbef5be7-9697-4eb4-b8c7-cd985e58c6b6-credential-socket\") pod \"dbef5be7-9697-4eb4-b8c7-cd985e58c6b6\" (UID: \"dbef5be7-9697-4eb4-b8c7-cd985e58c6b6\") "
Nov 02 15:17:15.743995 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:15.736842    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dbef5be7-9697-4eb4-b8c7-cd985e58c6b6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dbef5be7-9697-4eb4-b8c7-cd985e58c6b6-istio-podinfo\") pod \"dbef5be7-9697-4eb4-b8c7-cd985e58c6b6\" (UID: \"dbef5be7-9697-4eb4-b8c7-cd985e58c6b6\") "
Nov 02 15:17:15.743995 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:15.736910    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dbef5be7-9697-4eb4-b8c7-cd985e58c6b6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbef5be7-9697-4eb4-b8c7-cd985e58c6b6-workload-socket\") pod \"dbef5be7-9697-4eb4-b8c7-cd985e58c6b6\" (UID: \"dbef5be7-9697-4eb4-b8c7-cd985e58c6b6\") "
Nov 02 15:17:15.743995 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:15.736950    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dbef5be7-9697-4eb4-b8c7-cd985e58c6b6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dbef5be7-9697-4eb4-b8c7-cd985e58c6b6-istio-envoy\") pod \"dbef5be7-9697-4eb4-b8c7-cd985e58c6b6\" (UID: \"dbef5be7-9697-4eb4-b8c7-cd985e58c6b6\") "
Nov 02 15:17:15.743995 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:15.737048    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dbef5be7-9697-4eb4-b8c7-cd985e58c6b6-istiod-ca-cert" pod "dbef5be7-9697-4eb4-b8c7-cd985e58c6b6" (UID: "dbef5be7-9697-4eb4-b8c7-cd985e58c6b6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:17:15.743995 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:15.737338    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbef5be7-9697-4eb4-b8c7-cd985e58c6b6-credential-socket" pod "dbef5be7-9697-4eb4-b8c7-cd985e58c6b6" (UID: "dbef5be7-9697-4eb4-b8c7-cd985e58c6b6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:17:15.743995 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:15.737468    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dbef5be7-9697-4eb4-b8c7-cd985e58c6b6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dbef5be7-9697-4eb4-b8c7-cd985e58c6b6-istio-data\") pod \"dbef5be7-9697-4eb4-b8c7-cd985e58c6b6\" (UID: \"dbef5be7-9697-4eb4-b8c7-cd985e58c6b6\") "
Nov 02 15:17:15.743995 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:15.737842    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbef5be7-9697-4eb4-b8c7-cd985e58c6b6-istio-data" pod "dbef5be7-9697-4eb4-b8c7-cd985e58c6b6" (UID: "dbef5be7-9697-4eb4-b8c7-cd985e58c6b6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:17:15.743995 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:15.738326    5304 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dbef5be7-9697-4eb4-b8c7-cd985e58c6b6-istiod-ca-cert\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:17:15.743995 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:15.738347    5304 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbef5be7-9697-4eb4-b8c7-cd985e58c6b6-credential-socket\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:17:15.743995 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:15.738362    5304 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fdbb90e5-42c8-4409-95e1-a44ed4abead2-istio-podinfo\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:17:15.743995 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:15.738376    5304 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fdbb90e5-42c8-4409-95e1-a44ed4abead2-workload-certs\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:17:15.743995 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:15.738389    5304 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dbef5be7-9697-4eb4-b8c7-cd985e58c6b6-istio-data\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:17:15.743995 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:15.738402    5304 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdbb90e5-42c8-4409-95e1-a44ed4abead2-workload-socket\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:17:15.743995 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:15.738415    5304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pmp5b\" (UniqueName: \"kubernetes.io/projected/fdbb90e5-42c8-4409-95e1-a44ed4abead2-kube-api-access-pmp5b\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:17:15.743995 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:15.738429    5304 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fdbb90e5-42c8-4409-95e1-a44ed4abead2-istio-data\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:17:15.743995 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:15.738443    5304 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dbef5be7-9697-4eb4-b8c7-cd985e58c6b6-workload-certs\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:17:15.743995 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:15.738455    5304 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fdbb90e5-42c8-4409-95e1-a44ed4abead2-istio-token\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:17:15.743995 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:15.738467    5304 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdbb90e5-42c8-4409-95e1-a44ed4abead2-credential-socket\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:17:15.743995 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:15.738480    5304 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fdbb90e5-42c8-4409-95e1-a44ed4abead2-istio-envoy\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:17:15.743995 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:15.738533    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbef5be7-9697-4eb4-b8c7-cd985e58c6b6-workload-socket" pod "dbef5be7-9697-4eb4-b8c7-cd985e58c6b6" (UID: "dbef5be7-9697-4eb4-b8c7-cd985e58c6b6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:17:15.743995 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:15.739655    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dbef5be7-9697-4eb4-b8c7-cd985e58c6b6-istio-token" pod "dbef5be7-9697-4eb4-b8c7-cd985e58c6b6" (UID: "dbef5be7-9697-4eb4-b8c7-cd985e58c6b6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:17:15.743995 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:15.742569    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dbef5be7-9697-4eb4-b8c7-cd985e58c6b6-istio-podinfo" pod "dbef5be7-9697-4eb4-b8c7-cd985e58c6b6" (UID: "dbef5be7-9697-4eb4-b8c7-cd985e58c6b6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 15:17:15.748865 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:15.744631    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbef5be7-9697-4eb4-b8c7-cd985e58c6b6-istio-envoy" pod "dbef5be7-9697-4eb4-b8c7-cd985e58c6b6" (UID: "dbef5be7-9697-4eb4-b8c7-cd985e58c6b6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:17:15.751564 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:15.751523    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dbef5be7-9697-4eb4-b8c7-cd985e58c6b6-kube-api-access-ls2fk" pod "dbef5be7-9697-4eb4-b8c7-cd985e58c6b6" (UID: "dbef5be7-9697-4eb4-b8c7-cd985e58c6b6"). InnerVolumeSpecName "kube-api-access-ls2fk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:17:15.810087 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:15.810020    5304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-bjrnm"]
Nov 02 15:17:15.811383 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:15.811358    5304 cpu_manager.go:414] "containerMap: removing container" podUID="fdbb90e5-42c8-4409-95e1-a44ed4abead2" containerName="istio-proxy"
Nov 02 15:17:15.811530 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:15.811518    5304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fdbb90e5-42c8-4409-95e1-a44ed4abead2" containerName="istio-proxy"
Nov 02 15:17:15.811633 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:15.811624    5304 cpu_manager.go:414] "containerMap: removing container" podUID="dbef5be7-9697-4eb4-b8c7-cd985e58c6b6" containerName="istio-proxy"
Nov 02 15:17:15.811712 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:15.811702    5304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dbef5be7-9697-4eb4-b8c7-cd985e58c6b6" containerName="istio-proxy"
Nov 02 15:17:15.811966 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:15.811953    5304 memory_manager.go:365] "RemoveStaleState removing state" podUID="fdbb90e5-42c8-4409-95e1-a44ed4abead2" containerName="istio-proxy"
Nov 02 15:17:15.812065 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:15.812056    5304 memory_manager.go:365] "RemoveStaleState removing state" podUID="dbef5be7-9697-4eb4-b8c7-cd985e58c6b6" containerName="istio-proxy"
Nov 02 15:17:15.846935 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:15.836855    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-bjrnm"
Nov 02 15:17:15.855336 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:15.847705    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-bjrnm"]
Nov 02 15:17:15.911265 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:15.900939    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-889zh\" (UniqueName: \"kubernetes.io/projected/97220a55-1205-4d67-90ce-49d6cdef5c43-kube-api-access-889zh\") pod \"e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-bjrnm\" (UID: \"97220a55-1205-4d67-90ce-49d6cdef5c43\") " pod="openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-bjrnm"
Nov 02 15:17:15.911265 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:15.901096    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97220a55-1205-4d67-90ce-49d6cdef5c43-istiod-ca-cert\") pod \"e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-bjrnm\" (UID: \"97220a55-1205-4d67-90ce-49d6cdef5c43\") " pod="openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-bjrnm"
Nov 02 15:17:15.911265 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:15.901179    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97220a55-1205-4d67-90ce-49d6cdef5c43-credential-socket\") pod \"e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-bjrnm\" (UID: \"97220a55-1205-4d67-90ce-49d6cdef5c43\") " pod="openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-bjrnm"
Nov 02 15:17:15.911265 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:15.901226    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97220a55-1205-4d67-90ce-49d6cdef5c43-workload-certs\") pod \"e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-bjrnm\" (UID: \"97220a55-1205-4d67-90ce-49d6cdef5c43\") " pod="openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-bjrnm"
Nov 02 15:17:15.911265 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:15.901299    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97220a55-1205-4d67-90ce-49d6cdef5c43-istio-data\") pod \"e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-bjrnm\" (UID: \"97220a55-1205-4d67-90ce-49d6cdef5c43\") " pod="openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-bjrnm"
Nov 02 15:17:15.911265 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:15.901357    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97220a55-1205-4d67-90ce-49d6cdef5c43-istio-envoy\") pod \"e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-bjrnm\" (UID: \"97220a55-1205-4d67-90ce-49d6cdef5c43\") " pod="openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-bjrnm"
Nov 02 15:17:15.911265 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:15.901435    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97220a55-1205-4d67-90ce-49d6cdef5c43-istio-podinfo\") pod \"e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-bjrnm\" (UID: \"97220a55-1205-4d67-90ce-49d6cdef5c43\") " pod="openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-bjrnm"
Nov 02 15:17:15.911265 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:15.901526    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97220a55-1205-4d67-90ce-49d6cdef5c43-workload-socket\") pod \"e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-bjrnm\" (UID: \"97220a55-1205-4d67-90ce-49d6cdef5c43\") " pod="openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-bjrnm"
Nov 02 15:17:15.911265 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:15.901586    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97220a55-1205-4d67-90ce-49d6cdef5c43-istio-token\") pod \"e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-bjrnm\" (UID: \"97220a55-1205-4d67-90ce-49d6cdef5c43\") " pod="openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-bjrnm"
Nov 02 15:17:15.911265 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:15.901789    5304 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dbef5be7-9697-4eb4-b8c7-cd985e58c6b6-istio-podinfo\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:17:15.911265 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:15.901805    5304 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbef5be7-9697-4eb4-b8c7-cd985e58c6b6-workload-socket\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:17:15.911265 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:15.901819    5304 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dbef5be7-9697-4eb4-b8c7-cd985e58c6b6-istio-envoy\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:17:15.911265 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:15.901834    5304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ls2fk\" (UniqueName: \"kubernetes.io/projected/dbef5be7-9697-4eb4-b8c7-cd985e58c6b6-kube-api-access-ls2fk\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:17:15.911265 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:15.901848    5304 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dbef5be7-9697-4eb4-b8c7-cd985e58c6b6-istio-token\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:17:16.019569 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:16.009926    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97220a55-1205-4d67-90ce-49d6cdef5c43-istiod-ca-cert\") pod \"e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-bjrnm\" (UID: \"97220a55-1205-4d67-90ce-49d6cdef5c43\") " pod="openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-bjrnm"
Nov 02 15:17:16.019569 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:16.010004    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97220a55-1205-4d67-90ce-49d6cdef5c43-credential-socket\") pod \"e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-bjrnm\" (UID: \"97220a55-1205-4d67-90ce-49d6cdef5c43\") " pod="openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-bjrnm"
Nov 02 15:17:16.019569 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:16.010044    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97220a55-1205-4d67-90ce-49d6cdef5c43-workload-certs\") pod \"e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-bjrnm\" (UID: \"97220a55-1205-4d67-90ce-49d6cdef5c43\") " pod="openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-bjrnm"
Nov 02 15:17:16.019569 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:16.010117    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97220a55-1205-4d67-90ce-49d6cdef5c43-istio-data\") pod \"e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-bjrnm\" (UID: \"97220a55-1205-4d67-90ce-49d6cdef5c43\") " pod="openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-bjrnm"
Nov 02 15:17:16.019569 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:16.010188    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97220a55-1205-4d67-90ce-49d6cdef5c43-istio-envoy\") pod \"e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-bjrnm\" (UID: \"97220a55-1205-4d67-90ce-49d6cdef5c43\") " pod="openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-bjrnm"
Nov 02 15:17:16.019569 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:16.010265    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97220a55-1205-4d67-90ce-49d6cdef5c43-istio-podinfo\") pod \"e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-bjrnm\" (UID: \"97220a55-1205-4d67-90ce-49d6cdef5c43\") " pod="openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-bjrnm"
Nov 02 15:17:16.019569 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:16.010354    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97220a55-1205-4d67-90ce-49d6cdef5c43-workload-socket\") pod \"e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-bjrnm\" (UID: \"97220a55-1205-4d67-90ce-49d6cdef5c43\") " pod="openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-bjrnm"
Nov 02 15:17:16.019569 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:16.010414    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97220a55-1205-4d67-90ce-49d6cdef5c43-istio-token\") pod \"e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-bjrnm\" (UID: \"97220a55-1205-4d67-90ce-49d6cdef5c43\") " pod="openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-bjrnm"
Nov 02 15:17:16.019569 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:16.010600    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-889zh\" (UniqueName: \"kubernetes.io/projected/97220a55-1205-4d67-90ce-49d6cdef5c43-kube-api-access-889zh\") pod \"e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-bjrnm\" (UID: \"97220a55-1205-4d67-90ce-49d6cdef5c43\") " pod="openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-bjrnm"
Nov 02 15:17:16.019569 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:16.010767    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97220a55-1205-4d67-90ce-49d6cdef5c43-istiod-ca-cert\") pod \"e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-bjrnm\" (UID: \"97220a55-1205-4d67-90ce-49d6cdef5c43\") " pod="openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-bjrnm"
Nov 02 15:17:16.019569 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:16.011103    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97220a55-1205-4d67-90ce-49d6cdef5c43-credential-socket\") pod \"e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-bjrnm\" (UID: \"97220a55-1205-4d67-90ce-49d6cdef5c43\") " pod="openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-bjrnm"
Nov 02 15:17:16.019569 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:16.011341    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97220a55-1205-4d67-90ce-49d6cdef5c43-workload-socket\") pod \"e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-bjrnm\" (UID: \"97220a55-1205-4d67-90ce-49d6cdef5c43\") " pod="openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-bjrnm"
Nov 02 15:17:16.019569 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:16.011984    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97220a55-1205-4d67-90ce-49d6cdef5c43-workload-certs\") pod \"e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-bjrnm\" (UID: \"97220a55-1205-4d67-90ce-49d6cdef5c43\") " pod="openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-bjrnm"
Nov 02 15:17:16.019569 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:16.012217    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97220a55-1205-4d67-90ce-49d6cdef5c43-istio-data\") pod \"e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-bjrnm\" (UID: \"97220a55-1205-4d67-90ce-49d6cdef5c43\") " pod="openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-bjrnm"
Nov 02 15:17:16.019569 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:16.019476    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97220a55-1205-4d67-90ce-49d6cdef5c43-istio-envoy\") pod \"e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-bjrnm\" (UID: \"97220a55-1205-4d67-90ce-49d6cdef5c43\") " pod="openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-bjrnm"
Nov 02 15:17:16.020440 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:16.019747    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97220a55-1205-4d67-90ce-49d6cdef5c43-istio-podinfo\") pod \"e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-bjrnm\" (UID: \"97220a55-1205-4d67-90ce-49d6cdef5c43\") " pod="openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-bjrnm"
Nov 02 15:17:16.053344 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:16.053303    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-k2p6m"
Nov 02 15:17:16.055433 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:16.055385    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-889zh\" (UniqueName: \"kubernetes.io/projected/97220a55-1205-4d67-90ce-49d6cdef5c43-kube-api-access-889zh\") pod \"e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-bjrnm\" (UID: \"97220a55-1205-4d67-90ce-49d6cdef5c43\") " pod="openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-bjrnm"
Nov 02 15:17:16.055806 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:16.055756    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97220a55-1205-4d67-90ce-49d6cdef5c43-istio-token\") pod \"e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-bjrnm\" (UID: \"97220a55-1205-4d67-90ce-49d6cdef5c43\") " pod="openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-bjrnm"
Nov 02 15:17:16.066523 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:16.066488    5304 generic.go:356] "Generic (PLEG): container finished" podID="1bd23848-eb1d-45fd-9d87-c11302b3c111" containerID="d6bc4523c8b8264ec504666897583f330e20731aee7d610dbbf90cd79d0807b9" exitCode=0
Nov 02 15:17:16.066670 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:16.066557    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-k2p6m"
Nov 02 15:17:16.066670 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:16.066592    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-k2p6m" event={"ID":"1bd23848-eb1d-45fd-9d87-c11302b3c111","Type":"ContainerDied","Data":"d6bc4523c8b8264ec504666897583f330e20731aee7d610dbbf90cd79d0807b9"}
Nov 02 15:17:16.066670 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:16.066662    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-k2p6m" event={"ID":"1bd23848-eb1d-45fd-9d87-c11302b3c111","Type":"ContainerDied","Data":"a19541c6415fcf38aa1ec0e52f1368b479c5275c975d1b462a73ba7a905e650e"}
Nov 02 15:17:16.066848 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:16.066703    5304 scope.go:122] "RemoveContainer" containerID="d6bc4523c8b8264ec504666897583f330e20731aee7d610dbbf90cd79d0807b9"
Nov 02 15:17:16.072237 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:16.071785    5304 generic.go:356] "Generic (PLEG): container finished" podID="fdbb90e5-42c8-4409-95e1-a44ed4abead2" containerID="9204f7292e82a2a289873d5feea1b81d704edcdf3b90e4892e8a5bc5bd64117d" exitCode=0
Nov 02 15:17:16.072237 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:16.071950    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-576tm" event={"ID":"fdbb90e5-42c8-4409-95e1-a44ed4abead2","Type":"ContainerDied","Data":"9204f7292e82a2a289873d5feea1b81d704edcdf3b90e4892e8a5bc5bd64117d"}
Nov 02 15:17:16.072237 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:16.072002    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-576tm" event={"ID":"fdbb90e5-42c8-4409-95e1-a44ed4abead2","Type":"ContainerDied","Data":"bbf768ecf1d14b4a2bb82b2594d9d532b33f3d3b87d550d8090a74a0e0aa14f7"}
Nov 02 15:17:16.072237 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:16.072109    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-576tm"
Nov 02 15:17:16.081823 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:16.081364    5304 generic.go:356] "Generic (PLEG): container finished" podID="79e8599a-22fa-4df3-a35a-4d16c271cfe2" containerID="165e1e14e0dc19782087cbf0db0d14128fb6fd4ec462abe1c9b3ae9d0ea73fe3" exitCode=0
Nov 02 15:17:16.081823 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:16.081515    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-cwnwk" event={"ID":"79e8599a-22fa-4df3-a35a-4d16c271cfe2","Type":"ContainerDied","Data":"165e1e14e0dc19782087cbf0db0d14128fb6fd4ec462abe1c9b3ae9d0ea73fe3"}
Nov 02 15:17:16.089465 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:16.089437    5304 generic.go:356] "Generic (PLEG): container finished" podID="3d4f7624-882c-4cb3-8811-9ca682e0c1d5" containerID="f308b7b6046e7500e8e6d2d6f77c217a3ab3773533f03a01f6b15aa9ca7b1d96" exitCode=0
Nov 02 15:17:16.090279 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:16.090253    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-b88h7" event={"ID":"3d4f7624-882c-4cb3-8811-9ca682e0c1d5","Type":"ContainerDied","Data":"f308b7b6046e7500e8e6d2d6f77c217a3ab3773533f03a01f6b15aa9ca7b1d96"}
Nov 02 15:17:16.090740 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:16.090671    5304 scope.go:122] "RemoveContainer" containerID="d6bc4523c8b8264ec504666897583f330e20731aee7d610dbbf90cd79d0807b9"
Nov 02 15:17:16.091507 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:17:16.091343    5304 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d6bc4523c8b8264ec504666897583f330e20731aee7d610dbbf90cd79d0807b9\": container with ID starting with d6bc4523c8b8264ec504666897583f330e20731aee7d610dbbf90cd79d0807b9 not found: ID does not exist" containerID="d6bc4523c8b8264ec504666897583f330e20731aee7d610dbbf90cd79d0807b9"
Nov 02 15:17:16.091507 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:16.091388    5304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6bc4523c8b8264ec504666897583f330e20731aee7d610dbbf90cd79d0807b9"} err="failed to get container status \"d6bc4523c8b8264ec504666897583f330e20731aee7d610dbbf90cd79d0807b9\": rpc error: code = NotFound desc = could not find container \"d6bc4523c8b8264ec504666897583f330e20731aee7d610dbbf90cd79d0807b9\": container with ID starting with d6bc4523c8b8264ec504666897583f330e20731aee7d610dbbf90cd79d0807b9 not found: ID does not exist"
Nov 02 15:17:16.091507 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:16.091412    5304 scope.go:122] "RemoveContainer" containerID="9204f7292e82a2a289873d5feea1b81d704edcdf3b90e4892e8a5bc5bd64117d"
Nov 02 15:17:16.096062 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:16.096035    5304 generic.go:356] "Generic (PLEG): container finished" podID="dbef5be7-9697-4eb4-b8c7-cd985e58c6b6" containerID="ac7a2102277c7c786d1251baf6db451b6db5785c9ae1f24d6e55a39c762bb01f" exitCode=0
Nov 02 15:17:16.096210 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:16.096081    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-dq45c" event={"ID":"dbef5be7-9697-4eb4-b8c7-cd985e58c6b6","Type":"ContainerDied","Data":"ac7a2102277c7c786d1251baf6db451b6db5785c9ae1f24d6e55a39c762bb01f"}
Nov 02 15:17:16.096210 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:16.096110    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-dq45c"
Nov 02 15:17:16.096924 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:16.096165    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-dq45c" event={"ID":"dbef5be7-9697-4eb4-b8c7-cd985e58c6b6","Type":"ContainerDied","Data":"0ecca7a206c56feb60c4a0b885cab95071d704242f28c977ed855da4a1105712"}
Nov 02 15:17:16.116265 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:16.114651    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1bd23848-eb1d-45fd-9d87-c11302b3c111-kube-api-access-j4v6m\" (UniqueName: \"kubernetes.io/projected/1bd23848-eb1d-45fd-9d87-c11302b3c111-kube-api-access-j4v6m\") pod \"1bd23848-eb1d-45fd-9d87-c11302b3c111\" (UID: \"1bd23848-eb1d-45fd-9d87-c11302b3c111\") "
Nov 02 15:17:16.116265 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:16.114716    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1bd23848-eb1d-45fd-9d87-c11302b3c111-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bd23848-eb1d-45fd-9d87-c11302b3c111-credential-socket\") pod \"1bd23848-eb1d-45fd-9d87-c11302b3c111\" (UID: \"1bd23848-eb1d-45fd-9d87-c11302b3c111\") "
Nov 02 15:17:16.116265 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:16.114791    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1bd23848-eb1d-45fd-9d87-c11302b3c111-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1bd23848-eb1d-45fd-9d87-c11302b3c111-istio-envoy\") pod \"1bd23848-eb1d-45fd-9d87-c11302b3c111\" (UID: \"1bd23848-eb1d-45fd-9d87-c11302b3c111\") "
Nov 02 15:17:16.116265 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:16.114844    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1bd23848-eb1d-45fd-9d87-c11302b3c111-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1bd23848-eb1d-45fd-9d87-c11302b3c111-istiod-ca-cert\") pod \"1bd23848-eb1d-45fd-9d87-c11302b3c111\" (UID: \"1bd23848-eb1d-45fd-9d87-c11302b3c111\") "
Nov 02 15:17:16.116265 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:16.114900    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1bd23848-eb1d-45fd-9d87-c11302b3c111-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1bd23848-eb1d-45fd-9d87-c11302b3c111-workload-certs\") pod \"1bd23848-eb1d-45fd-9d87-c11302b3c111\" (UID: \"1bd23848-eb1d-45fd-9d87-c11302b3c111\") "
Nov 02 15:17:16.116265 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:16.114959    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1bd23848-eb1d-45fd-9d87-c11302b3c111-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bd23848-eb1d-45fd-9d87-c11302b3c111-workload-socket\") pod \"1bd23848-eb1d-45fd-9d87-c11302b3c111\" (UID: \"1bd23848-eb1d-45fd-9d87-c11302b3c111\") "
Nov 02 15:17:16.116265 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:16.115056    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1bd23848-eb1d-45fd-9d87-c11302b3c111-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1bd23848-eb1d-45fd-9d87-c11302b3c111-istio-podinfo\") pod \"1bd23848-eb1d-45fd-9d87-c11302b3c111\" (UID: \"1bd23848-eb1d-45fd-9d87-c11302b3c111\") "
Nov 02 15:17:16.116265 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:16.115138    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1bd23848-eb1d-45fd-9d87-c11302b3c111-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1bd23848-eb1d-45fd-9d87-c11302b3c111-istio-data\") pod \"1bd23848-eb1d-45fd-9d87-c11302b3c111\" (UID: \"1bd23848-eb1d-45fd-9d87-c11302b3c111\") "
Nov 02 15:17:16.116265 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:16.115213    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1bd23848-eb1d-45fd-9d87-c11302b3c111-istio-token\" (UniqueName: \"kubernetes.io/projected/1bd23848-eb1d-45fd-9d87-c11302b3c111-istio-token\") pod \"1bd23848-eb1d-45fd-9d87-c11302b3c111\" (UID: \"1bd23848-eb1d-45fd-9d87-c11302b3c111\") "
Nov 02 15:17:16.116909 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:16.116543    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1bd23848-eb1d-45fd-9d87-c11302b3c111-istiod-ca-cert" pod "1bd23848-eb1d-45fd-9d87-c11302b3c111" (UID: "1bd23848-eb1d-45fd-9d87-c11302b3c111"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:17:16.126511 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:16.126224    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1bd23848-eb1d-45fd-9d87-c11302b3c111-istio-token" pod "1bd23848-eb1d-45fd-9d87-c11302b3c111" (UID: "1bd23848-eb1d-45fd-9d87-c11302b3c111"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:17:16.130476 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:16.130408    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1bd23848-eb1d-45fd-9d87-c11302b3c111-credential-socket" pod "1bd23848-eb1d-45fd-9d87-c11302b3c111" (UID: "1bd23848-eb1d-45fd-9d87-c11302b3c111"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:17:16.130840 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:16.130813    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1bd23848-eb1d-45fd-9d87-c11302b3c111-workload-certs" pod "1bd23848-eb1d-45fd-9d87-c11302b3c111" (UID: "1bd23848-eb1d-45fd-9d87-c11302b3c111"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:17:16.131031 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:16.130996    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1bd23848-eb1d-45fd-9d87-c11302b3c111-istio-data" pod "1bd23848-eb1d-45fd-9d87-c11302b3c111" (UID: "1bd23848-eb1d-45fd-9d87-c11302b3c111"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:17:16.132641 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:16.132604    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1bd23848-eb1d-45fd-9d87-c11302b3c111-workload-socket" pod "1bd23848-eb1d-45fd-9d87-c11302b3c111" (UID: "1bd23848-eb1d-45fd-9d87-c11302b3c111"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:17:16.134682 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:16.134345    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-b88h7"
Nov 02 15:17:16.137797 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:16.137536    5304 scope.go:122] "RemoveContainer" containerID="9204f7292e82a2a289873d5feea1b81d704edcdf3b90e4892e8a5bc5bd64117d"
Nov 02 15:17:16.138067 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:17:16.137950    5304 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9204f7292e82a2a289873d5feea1b81d704edcdf3b90e4892e8a5bc5bd64117d\": container with ID starting with 9204f7292e82a2a289873d5feea1b81d704edcdf3b90e4892e8a5bc5bd64117d not found: ID does not exist" containerID="9204f7292e82a2a289873d5feea1b81d704edcdf3b90e4892e8a5bc5bd64117d"
Nov 02 15:17:16.138067 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:16.137986    5304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9204f7292e82a2a289873d5feea1b81d704edcdf3b90e4892e8a5bc5bd64117d"} err="failed to get container status \"9204f7292e82a2a289873d5feea1b81d704edcdf3b90e4892e8a5bc5bd64117d\": rpc error: code = NotFound desc = could not find container \"9204f7292e82a2a289873d5feea1b81d704edcdf3b90e4892e8a5bc5bd64117d\": container with ID starting with 9204f7292e82a2a289873d5feea1b81d704edcdf3b90e4892e8a5bc5bd64117d not found: ID does not exist"
Nov 02 15:17:16.138067 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:16.138010    5304 scope.go:122] "RemoveContainer" containerID="ac7a2102277c7c786d1251baf6db451b6db5785c9ae1f24d6e55a39c762bb01f"
Nov 02 15:17:16.138728 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:16.138694    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1bd23848-eb1d-45fd-9d87-c11302b3c111-kube-api-access-j4v6m" pod "1bd23848-eb1d-45fd-9d87-c11302b3c111" (UID: "1bd23848-eb1d-45fd-9d87-c11302b3c111"). InnerVolumeSpecName "kube-api-access-j4v6m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:17:16.140532 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:16.140505    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1bd23848-eb1d-45fd-9d87-c11302b3c111-istio-podinfo" pod "1bd23848-eb1d-45fd-9d87-c11302b3c111" (UID: "1bd23848-eb1d-45fd-9d87-c11302b3c111"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 15:17:16.154538 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:16.154509    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1bd23848-eb1d-45fd-9d87-c11302b3c111-istio-envoy" pod "1bd23848-eb1d-45fd-9d87-c11302b3c111" (UID: "1bd23848-eb1d-45fd-9d87-c11302b3c111"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:17:16.179265 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:16.179201    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-576tm"]
Nov 02 15:17:16.185387 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:16.185367    5304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-576tm"]
Nov 02 15:17:16.191510 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:16.191490    5304 scope.go:122] "RemoveContainer" containerID="ac7a2102277c7c786d1251baf6db451b6db5785c9ae1f24d6e55a39c762bb01f"
Nov 02 15:17:16.192639 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:17:16.192584    5304 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac7a2102277c7c786d1251baf6db451b6db5785c9ae1f24d6e55a39c762bb01f\": container with ID starting with ac7a2102277c7c786d1251baf6db451b6db5785c9ae1f24d6e55a39c762bb01f not found: ID does not exist" containerID="ac7a2102277c7c786d1251baf6db451b6db5785c9ae1f24d6e55a39c762bb01f"
Nov 02 15:17:16.192755 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:16.192650    5304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac7a2102277c7c786d1251baf6db451b6db5785c9ae1f24d6e55a39c762bb01f"} err="failed to get container status \"ac7a2102277c7c786d1251baf6db451b6db5785c9ae1f24d6e55a39c762bb01f\": rpc error: code = NotFound desc = could not find container \"ac7a2102277c7c786d1251baf6db451b6db5785c9ae1f24d6e55a39c762bb01f\": container with ID starting with ac7a2102277c7c786d1251baf6db451b6db5785c9ae1f24d6e55a39c762bb01f not found: ID does not exist"
Nov 02 15:17:16.212644 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:16.212611    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-cwnwk"
Nov 02 15:17:16.214759 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:16.214733    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-bjrnm"
Nov 02 15:17:16.216405 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:16.216382    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d4f7624-882c-4cb3-8811-9ca682e0c1d5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d4f7624-882c-4cb3-8811-9ca682e0c1d5-credential-socket\") pod \"3d4f7624-882c-4cb3-8811-9ca682e0c1d5\" (UID: \"3d4f7624-882c-4cb3-8811-9ca682e0c1d5\") "
Nov 02 15:17:16.216518 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:16.216433    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d4f7624-882c-4cb3-8811-9ca682e0c1d5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d4f7624-882c-4cb3-8811-9ca682e0c1d5-workload-certs\") pod \"3d4f7624-882c-4cb3-8811-9ca682e0c1d5\" (UID: \"3d4f7624-882c-4cb3-8811-9ca682e0c1d5\") "
Nov 02 15:17:16.216518 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:16.216472    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d4f7624-882c-4cb3-8811-9ca682e0c1d5-istio-token\" (UniqueName: \"kubernetes.io/projected/3d4f7624-882c-4cb3-8811-9ca682e0c1d5-istio-token\") pod \"3d4f7624-882c-4cb3-8811-9ca682e0c1d5\" (UID: \"3d4f7624-882c-4cb3-8811-9ca682e0c1d5\") "
Nov 02 15:17:16.216658 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:16.216633    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d4f7624-882c-4cb3-8811-9ca682e0c1d5-credential-socket" pod "3d4f7624-882c-4cb3-8811-9ca682e0c1d5" (UID: "3d4f7624-882c-4cb3-8811-9ca682e0c1d5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:17:16.216727 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:16.216702    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d4f7624-882c-4cb3-8811-9ca682e0c1d5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d4f7624-882c-4cb3-8811-9ca682e0c1d5-workload-socket\") pod \"3d4f7624-882c-4cb3-8811-9ca682e0c1d5\" (UID: \"3d4f7624-882c-4cb3-8811-9ca682e0c1d5\") "
Nov 02 15:17:16.216783 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:16.216733    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d4f7624-882c-4cb3-8811-9ca682e0c1d5-workload-certs" pod "3d4f7624-882c-4cb3-8811-9ca682e0c1d5" (UID: "3d4f7624-882c-4cb3-8811-9ca682e0c1d5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:17:16.216783 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:16.216750    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3d4f7624-882c-4cb3-8811-9ca682e0c1d5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d4f7624-882c-4cb3-8811-9ca682e0c1d5-istio-podinfo\") pod \"3d4f7624-882c-4cb3-8811-9ca682e0c1d5\" (UID: \"3d4f7624-882c-4cb3-8811-9ca682e0c1d5\") "
Nov 02 15:17:16.216896 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:16.216836    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d4f7624-882c-4cb3-8811-9ca682e0c1d5-kube-api-access-smsjj\" (UniqueName: \"kubernetes.io/projected/3d4f7624-882c-4cb3-8811-9ca682e0c1d5-kube-api-access-smsjj\") pod \"3d4f7624-882c-4cb3-8811-9ca682e0c1d5\" (UID: \"3d4f7624-882c-4cb3-8811-9ca682e0c1d5\") "
Nov 02 15:17:16.216961 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:16.216913    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3d4f7624-882c-4cb3-8811-9ca682e0c1d5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d4f7624-882c-4cb3-8811-9ca682e0c1d5-istiod-ca-cert\") pod \"3d4f7624-882c-4cb3-8811-9ca682e0c1d5\" (UID: \"3d4f7624-882c-4cb3-8811-9ca682e0c1d5\") "
Nov 02 15:17:16.217013 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:16.216995    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d4f7624-882c-4cb3-8811-9ca682e0c1d5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d4f7624-882c-4cb3-8811-9ca682e0c1d5-istio-data\") pod \"3d4f7624-882c-4cb3-8811-9ca682e0c1d5\" (UID: \"3d4f7624-882c-4cb3-8811-9ca682e0c1d5\") "
Nov 02 15:17:16.217091 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:16.217034    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d4f7624-882c-4cb3-8811-9ca682e0c1d5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d4f7624-882c-4cb3-8811-9ca682e0c1d5-istio-envoy\") pod \"3d4f7624-882c-4cb3-8811-9ca682e0c1d5\" (UID: \"3d4f7624-882c-4cb3-8811-9ca682e0c1d5\") "
Nov 02 15:17:16.217273 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:16.217245    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d4f7624-882c-4cb3-8811-9ca682e0c1d5-workload-socket" pod "3d4f7624-882c-4cb3-8811-9ca682e0c1d5" (UID: "3d4f7624-882c-4cb3-8811-9ca682e0c1d5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:17:16.217471 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:16.217450    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d4f7624-882c-4cb3-8811-9ca682e0c1d5-istio-data" pod "3d4f7624-882c-4cb3-8811-9ca682e0c1d5" (UID: "3d4f7624-882c-4cb3-8811-9ca682e0c1d5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:17:16.217907 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:16.217889    5304 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d4f7624-882c-4cb3-8811-9ca682e0c1d5-credential-socket\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:17:16.217907 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:16.217912    5304 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d4f7624-882c-4cb3-8811-9ca682e0c1d5-workload-certs\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:17:16.218050 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:16.217929    5304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j4v6m\" (UniqueName: \"kubernetes.io/projected/1bd23848-eb1d-45fd-9d87-c11302b3c111-kube-api-access-j4v6m\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:17:16.218050 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:16.217944    5304 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bd23848-eb1d-45fd-9d87-c11302b3c111-credential-socket\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:17:16.218050 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:16.217959    5304 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1bd23848-eb1d-45fd-9d87-c11302b3c111-istio-envoy\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:17:16.218050 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:16.217982    5304 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1bd23848-eb1d-45fd-9d87-c11302b3c111-istiod-ca-cert\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:17:16.218050 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:16.217995    5304 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1bd23848-eb1d-45fd-9d87-c11302b3c111-workload-certs\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:17:16.218050 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:16.218008    5304 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bd23848-eb1d-45fd-9d87-c11302b3c111-workload-socket\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:17:16.218050 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:16.218021    5304 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d4f7624-882c-4cb3-8811-9ca682e0c1d5-workload-socket\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:17:16.218050 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:16.218037    5304 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1bd23848-eb1d-45fd-9d87-c11302b3c111-istio-podinfo\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:17:16.218050 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:16.218050    5304 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1bd23848-eb1d-45fd-9d87-c11302b3c111-istio-data\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:17:16.219547 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:16.218063    5304 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1bd23848-eb1d-45fd-9d87-c11302b3c111-istio-token\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:17:16.219547 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:16.218077    5304 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d4f7624-882c-4cb3-8811-9ca682e0c1d5-istio-data\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:17:16.219547 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:16.219086    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d4f7624-882c-4cb3-8811-9ca682e0c1d5-istiod-ca-cert" pod "3d4f7624-882c-4cb3-8811-9ca682e0c1d5" (UID: "3d4f7624-882c-4cb3-8811-9ca682e0c1d5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:17:16.219547 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:16.219345    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3d4f7624-882c-4cb3-8811-9ca682e0c1d5-istio-podinfo" pod "3d4f7624-882c-4cb3-8811-9ca682e0c1d5" (UID: "3d4f7624-882c-4cb3-8811-9ca682e0c1d5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 15:17:16.219742 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:16.219615    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d4f7624-882c-4cb3-8811-9ca682e0c1d5-istio-token" pod "3d4f7624-882c-4cb3-8811-9ca682e0c1d5" (UID: "3d4f7624-882c-4cb3-8811-9ca682e0c1d5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:17:16.219742 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:16.219645    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d4f7624-882c-4cb3-8811-9ca682e0c1d5-kube-api-access-smsjj" pod "3d4f7624-882c-4cb3-8811-9ca682e0c1d5" (UID: "3d4f7624-882c-4cb3-8811-9ca682e0c1d5"). InnerVolumeSpecName "kube-api-access-smsjj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:17:16.236497 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:16.236209    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d4f7624-882c-4cb3-8811-9ca682e0c1d5-istio-envoy" pod "3d4f7624-882c-4cb3-8811-9ca682e0c1d5" (UID: "3d4f7624-882c-4cb3-8811-9ca682e0c1d5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:17:16.284646 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:16.284615    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-dq45c"]
Nov 02 15:17:16.297901 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:16.297871    5304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-dq45c"]
Nov 02 15:17:16.319585 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:16.319551    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79e8599a-22fa-4df3-a35a-4d16c271cfe2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79e8599a-22fa-4df3-a35a-4d16c271cfe2-workload-certs\") pod \"79e8599a-22fa-4df3-a35a-4d16c271cfe2\" (UID: \"79e8599a-22fa-4df3-a35a-4d16c271cfe2\") "
Nov 02 15:17:16.319878 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:16.319855    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79e8599a-22fa-4df3-a35a-4d16c271cfe2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79e8599a-22fa-4df3-a35a-4d16c271cfe2-credential-socket\") pod \"79e8599a-22fa-4df3-a35a-4d16c271cfe2\" (UID: \"79e8599a-22fa-4df3-a35a-4d16c271cfe2\") "
Nov 02 15:17:16.320043 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:16.320028    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/79e8599a-22fa-4df3-a35a-4d16c271cfe2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79e8599a-22fa-4df3-a35a-4d16c271cfe2-istiod-ca-cert\") pod \"79e8599a-22fa-4df3-a35a-4d16c271cfe2\" (UID: \"79e8599a-22fa-4df3-a35a-4d16c271cfe2\") "
Nov 02 15:17:16.320238 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:16.320221    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79e8599a-22fa-4df3-a35a-4d16c271cfe2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79e8599a-22fa-4df3-a35a-4d16c271cfe2-workload-socket\") pod \"79e8599a-22fa-4df3-a35a-4d16c271cfe2\" (UID: \"79e8599a-22fa-4df3-a35a-4d16c271cfe2\") "
Nov 02 15:17:16.320526 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:16.320509    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/79e8599a-22fa-4df3-a35a-4d16c271cfe2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79e8599a-22fa-4df3-a35a-4d16c271cfe2-istio-podinfo\") pod \"79e8599a-22fa-4df3-a35a-4d16c271cfe2\" (UID: \"79e8599a-22fa-4df3-a35a-4d16c271cfe2\") "
Nov 02 15:17:16.320708 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:16.320692    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79e8599a-22fa-4df3-a35a-4d16c271cfe2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79e8599a-22fa-4df3-a35a-4d16c271cfe2-istio-data\") pod \"79e8599a-22fa-4df3-a35a-4d16c271cfe2\" (UID: \"79e8599a-22fa-4df3-a35a-4d16c271cfe2\") "
Nov 02 15:17:16.320838 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:16.320823    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79e8599a-22fa-4df3-a35a-4d16c271cfe2-kube-api-access-vqmt9\" (UniqueName: \"kubernetes.io/projected/79e8599a-22fa-4df3-a35a-4d16c271cfe2-kube-api-access-vqmt9\") pod \"79e8599a-22fa-4df3-a35a-4d16c271cfe2\" (UID: \"79e8599a-22fa-4df3-a35a-4d16c271cfe2\") "
Nov 02 15:17:16.320936 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:16.320924    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79e8599a-22fa-4df3-a35a-4d16c271cfe2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79e8599a-22fa-4df3-a35a-4d16c271cfe2-istio-envoy\") pod \"79e8599a-22fa-4df3-a35a-4d16c271cfe2\" (UID: \"79e8599a-22fa-4df3-a35a-4d16c271cfe2\") "
Nov 02 15:17:16.321150 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:16.321119    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79e8599a-22fa-4df3-a35a-4d16c271cfe2-istio-token\" (UniqueName: \"kubernetes.io/projected/79e8599a-22fa-4df3-a35a-4d16c271cfe2-istio-token\") pod \"79e8599a-22fa-4df3-a35a-4d16c271cfe2\" (UID: \"79e8599a-22fa-4df3-a35a-4d16c271cfe2\") "
Nov 02 15:17:16.321462 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:16.321435    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79e8599a-22fa-4df3-a35a-4d16c271cfe2-workload-socket" pod "79e8599a-22fa-4df3-a35a-4d16c271cfe2" (UID: "79e8599a-22fa-4df3-a35a-4d16c271cfe2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:17:16.322403 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:16.322379    5304 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d4f7624-882c-4cb3-8811-9ca682e0c1d5-istio-podinfo\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:17:16.322544 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:16.322528    5304 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79e8599a-22fa-4df3-a35a-4d16c271cfe2-workload-socket\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:17:16.322653 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:16.322637    5304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-smsjj\" (UniqueName: \"kubernetes.io/projected/3d4f7624-882c-4cb3-8811-9ca682e0c1d5-kube-api-access-smsjj\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:17:16.322755 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:16.322742    5304 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d4f7624-882c-4cb3-8811-9ca682e0c1d5-istiod-ca-cert\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:17:16.322843 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:16.322832    5304 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d4f7624-882c-4cb3-8811-9ca682e0c1d5-istio-envoy\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:17:16.322923 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:16.322911    5304 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d4f7624-882c-4cb3-8811-9ca682e0c1d5-istio-token\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:17:16.323245 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:16.323225    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79e8599a-22fa-4df3-a35a-4d16c271cfe2-workload-certs" pod "79e8599a-22fa-4df3-a35a-4d16c271cfe2" (UID: "79e8599a-22fa-4df3-a35a-4d16c271cfe2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:17:16.323557 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:16.323537    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79e8599a-22fa-4df3-a35a-4d16c271cfe2-credential-socket" pod "79e8599a-22fa-4df3-a35a-4d16c271cfe2" (UID: "79e8599a-22fa-4df3-a35a-4d16c271cfe2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:17:16.323955 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:16.323934    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79e8599a-22fa-4df3-a35a-4d16c271cfe2-istiod-ca-cert" pod "79e8599a-22fa-4df3-a35a-4d16c271cfe2" (UID: "79e8599a-22fa-4df3-a35a-4d16c271cfe2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:17:16.324322 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:16.324303    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79e8599a-22fa-4df3-a35a-4d16c271cfe2-istio-data" pod "79e8599a-22fa-4df3-a35a-4d16c271cfe2" (UID: "79e8599a-22fa-4df3-a35a-4d16c271cfe2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:17:16.328876 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:16.328852    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79e8599a-22fa-4df3-a35a-4d16c271cfe2-istio-token" pod "79e8599a-22fa-4df3-a35a-4d16c271cfe2" (UID: "79e8599a-22fa-4df3-a35a-4d16c271cfe2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:17:16.330731 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:16.330709    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/79e8599a-22fa-4df3-a35a-4d16c271cfe2-istio-podinfo" pod "79e8599a-22fa-4df3-a35a-4d16c271cfe2" (UID: "79e8599a-22fa-4df3-a35a-4d16c271cfe2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 15:17:16.339782 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:16.339736    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79e8599a-22fa-4df3-a35a-4d16c271cfe2-istio-envoy" pod "79e8599a-22fa-4df3-a35a-4d16c271cfe2" (UID: "79e8599a-22fa-4df3-a35a-4d16c271cfe2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:17:16.340330 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:16.340296    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79e8599a-22fa-4df3-a35a-4d16c271cfe2-kube-api-access-vqmt9" pod "79e8599a-22fa-4df3-a35a-4d16c271cfe2" (UID: "79e8599a-22fa-4df3-a35a-4d16c271cfe2"). InnerVolumeSpecName "kube-api-access-vqmt9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:17:16.423175 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:16.422098    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-k2p6m"]
Nov 02 15:17:16.424111 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:16.424082    5304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-k2p6m"]
Nov 02 15:17:16.424841 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:16.424811    5304 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79e8599a-22fa-4df3-a35a-4d16c271cfe2-istio-podinfo\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:17:16.424950 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:16.424843    5304 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79e8599a-22fa-4df3-a35a-4d16c271cfe2-istio-data\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:17:16.424950 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:16.424859    5304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vqmt9\" (UniqueName: \"kubernetes.io/projected/79e8599a-22fa-4df3-a35a-4d16c271cfe2-kube-api-access-vqmt9\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:17:16.424950 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:16.424873    5304 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79e8599a-22fa-4df3-a35a-4d16c271cfe2-istio-envoy\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:17:16.424950 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:16.424887    5304 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79e8599a-22fa-4df3-a35a-4d16c271cfe2-istio-token\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:17:16.424950 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:16.424899    5304 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79e8599a-22fa-4df3-a35a-4d16c271cfe2-workload-certs\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:17:16.424950 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:16.424911    5304 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79e8599a-22fa-4df3-a35a-4d16c271cfe2-credential-socket\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:17:16.424950 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:16.424924    5304 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79e8599a-22fa-4df3-a35a-4d16c271cfe2-istiod-ca-cert\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:17:16.598796 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:16.598542    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-bjrnm"]
Nov 02 15:17:16.602424 ip-10-0-135-138 kubenswrapper[5304]: W1102 15:17:16.602390    5304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97220a55_1205_4d67_90ce_49d6cdef5c43.slice/crio-8c337bfeaa73c82f070e30190adf0055a68a725ab26b07cab83321e43b6fe674 WatchSource:0}: Error finding container 8c337bfeaa73c82f070e30190adf0055a68a725ab26b07cab83321e43b6fe674: Status 404 returned error can't find the container with id 8c337bfeaa73c82f070e30190adf0055a68a725ab26b07cab83321e43b6fe674
Nov 02 15:17:16.612000 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:16.611955    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:17:16.612115 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:16.612064    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:17:16.612115 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:16.612108    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:17:16.894382 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:16.888539    5304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-zzvrc"]
Nov 02 15:17:16.894382 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:16.891917    5304 cpu_manager.go:414] "containerMap: removing container" podUID="79e8599a-22fa-4df3-a35a-4d16c271cfe2" containerName="istio-proxy"
Nov 02 15:17:16.894382 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:16.891942    5304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79e8599a-22fa-4df3-a35a-4d16c271cfe2" containerName="istio-proxy"
Nov 02 15:17:16.894382 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:16.891978    5304 cpu_manager.go:414] "containerMap: removing container" podUID="3d4f7624-882c-4cb3-8811-9ca682e0c1d5" containerName="istio-proxy"
Nov 02 15:17:16.894382 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:16.891986    5304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d4f7624-882c-4cb3-8811-9ca682e0c1d5" containerName="istio-proxy"
Nov 02 15:17:16.894382 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:16.891997    5304 cpu_manager.go:414] "containerMap: removing container" podUID="1bd23848-eb1d-45fd-9d87-c11302b3c111" containerName="istio-proxy"
Nov 02 15:17:16.894382 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:16.892006    5304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1bd23848-eb1d-45fd-9d87-c11302b3c111" containerName="istio-proxy"
Nov 02 15:17:16.894382 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:16.892242    5304 memory_manager.go:365] "RemoveStaleState removing state" podUID="79e8599a-22fa-4df3-a35a-4d16c271cfe2" containerName="istio-proxy"
Nov 02 15:17:16.894382 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:16.892259    5304 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d4f7624-882c-4cb3-8811-9ca682e0c1d5" containerName="istio-proxy"
Nov 02 15:17:16.894382 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:16.892268    5304 memory_manager.go:365] "RemoveStaleState removing state" podUID="1bd23848-eb1d-45fd-9d87-c11302b3c111" containerName="istio-proxy"
Nov 02 15:17:16.900855 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:16.899944    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-zzvrc"
Nov 02 15:17:16.948004 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:16.946672    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6057a4f9-4a74-4484-930f-501e9f6e6297-credential-socket\") pod \"e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-zzvrc\" (UID: \"6057a4f9-4a74-4484-930f-501e9f6e6297\") " pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-zzvrc"
Nov 02 15:17:16.948004 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:16.946732    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6057a4f9-4a74-4484-930f-501e9f6e6297-workload-certs\") pod \"e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-zzvrc\" (UID: \"6057a4f9-4a74-4484-930f-501e9f6e6297\") " pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-zzvrc"
Nov 02 15:17:16.948004 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:16.946794    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kwgl6\" (UniqueName: \"kubernetes.io/projected/6057a4f9-4a74-4484-930f-501e9f6e6297-kube-api-access-kwgl6\") pod \"e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-zzvrc\" (UID: \"6057a4f9-4a74-4484-930f-501e9f6e6297\") " pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-zzvrc"
Nov 02 15:17:16.948004 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:16.946842    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6057a4f9-4a74-4484-930f-501e9f6e6297-istio-podinfo\") pod \"e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-zzvrc\" (UID: \"6057a4f9-4a74-4484-930f-501e9f6e6297\") " pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-zzvrc"
Nov 02 15:17:16.948004 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:16.946911    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6057a4f9-4a74-4484-930f-501e9f6e6297-workload-socket\") pod \"e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-zzvrc\" (UID: \"6057a4f9-4a74-4484-930f-501e9f6e6297\") " pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-zzvrc"
Nov 02 15:17:16.948004 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:16.947069    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6057a4f9-4a74-4484-930f-501e9f6e6297-istio-envoy\") pod \"e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-zzvrc\" (UID: \"6057a4f9-4a74-4484-930f-501e9f6e6297\") " pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-zzvrc"
Nov 02 15:17:16.948004 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:16.947104    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6057a4f9-4a74-4484-930f-501e9f6e6297-istiod-ca-cert\") pod \"e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-zzvrc\" (UID: \"6057a4f9-4a74-4484-930f-501e9f6e6297\") " pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-zzvrc"
Nov 02 15:17:16.950459 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:16.950251    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6057a4f9-4a74-4484-930f-501e9f6e6297-istio-token\") pod \"e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-zzvrc\" (UID: \"6057a4f9-4a74-4484-930f-501e9f6e6297\") " pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-zzvrc"
Nov 02 15:17:16.950459 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:16.950331    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6057a4f9-4a74-4484-930f-501e9f6e6297-istio-data\") pod \"e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-zzvrc\" (UID: \"6057a4f9-4a74-4484-930f-501e9f6e6297\") " pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-zzvrc"
Nov 02 15:17:16.986278 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:16.986232    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-zzvrc"]
Nov 02 15:17:17.051792 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:17.051753    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6057a4f9-4a74-4484-930f-501e9f6e6297-istio-envoy\") pod \"e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-zzvrc\" (UID: \"6057a4f9-4a74-4484-930f-501e9f6e6297\") " pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-zzvrc"
Nov 02 15:17:17.052058 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:17.052029    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6057a4f9-4a74-4484-930f-501e9f6e6297-istiod-ca-cert\") pod \"e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-zzvrc\" (UID: \"6057a4f9-4a74-4484-930f-501e9f6e6297\") " pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-zzvrc"
Nov 02 15:17:17.052208 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:17.052089    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6057a4f9-4a74-4484-930f-501e9f6e6297-istio-token\") pod \"e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-zzvrc\" (UID: \"6057a4f9-4a74-4484-930f-501e9f6e6297\") " pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-zzvrc"
Nov 02 15:17:17.052208 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:17.052145    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6057a4f9-4a74-4484-930f-501e9f6e6297-istio-data\") pod \"e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-zzvrc\" (UID: \"6057a4f9-4a74-4484-930f-501e9f6e6297\") " pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-zzvrc"
Nov 02 15:17:17.052358 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:17.052224    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6057a4f9-4a74-4484-930f-501e9f6e6297-credential-socket\") pod \"e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-zzvrc\" (UID: \"6057a4f9-4a74-4484-930f-501e9f6e6297\") " pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-zzvrc"
Nov 02 15:17:17.052358 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:17.052262    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6057a4f9-4a74-4484-930f-501e9f6e6297-workload-certs\") pod \"e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-zzvrc\" (UID: \"6057a4f9-4a74-4484-930f-501e9f6e6297\") " pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-zzvrc"
Nov 02 15:17:17.052358 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:17.052331    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kwgl6\" (UniqueName: \"kubernetes.io/projected/6057a4f9-4a74-4484-930f-501e9f6e6297-kube-api-access-kwgl6\") pod \"e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-zzvrc\" (UID: \"6057a4f9-4a74-4484-930f-501e9f6e6297\") " pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-zzvrc"
Nov 02 15:17:17.052671 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:17.052382    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6057a4f9-4a74-4484-930f-501e9f6e6297-istio-podinfo\") pod \"e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-zzvrc\" (UID: \"6057a4f9-4a74-4484-930f-501e9f6e6297\") " pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-zzvrc"
Nov 02 15:17:17.052671 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:17.052460    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6057a4f9-4a74-4484-930f-501e9f6e6297-workload-socket\") pod \"e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-zzvrc\" (UID: \"6057a4f9-4a74-4484-930f-501e9f6e6297\") " pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-zzvrc"
Nov 02 15:17:17.053045 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:17.053022    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6057a4f9-4a74-4484-930f-501e9f6e6297-workload-socket\") pod \"e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-zzvrc\" (UID: \"6057a4f9-4a74-4484-930f-501e9f6e6297\") " pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-zzvrc"
Nov 02 15:17:17.053859 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:17.053614    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6057a4f9-4a74-4484-930f-501e9f6e6297-credential-socket\") pod \"e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-zzvrc\" (UID: \"6057a4f9-4a74-4484-930f-501e9f6e6297\") " pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-zzvrc"
Nov 02 15:17:17.054591 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:17.054006    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6057a4f9-4a74-4484-930f-501e9f6e6297-istio-data\") pod \"e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-zzvrc\" (UID: \"6057a4f9-4a74-4484-930f-501e9f6e6297\") " pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-zzvrc"
Nov 02 15:17:17.054591 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:17.054553    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6057a4f9-4a74-4484-930f-501e9f6e6297-workload-certs\") pod \"e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-zzvrc\" (UID: \"6057a4f9-4a74-4484-930f-501e9f6e6297\") " pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-zzvrc"
Nov 02 15:17:17.054748 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:17.054625    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6057a4f9-4a74-4484-930f-501e9f6e6297-istiod-ca-cert\") pod \"e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-zzvrc\" (UID: \"6057a4f9-4a74-4484-930f-501e9f6e6297\") " pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-zzvrc"
Nov 02 15:17:17.056341 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:17.056319    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6057a4f9-4a74-4484-930f-501e9f6e6297-istio-envoy\") pod \"e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-zzvrc\" (UID: \"6057a4f9-4a74-4484-930f-501e9f6e6297\") " pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-zzvrc"
Nov 02 15:17:17.058244 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:17.058219    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6057a4f9-4a74-4484-930f-501e9f6e6297-istio-podinfo\") pod \"e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-zzvrc\" (UID: \"6057a4f9-4a74-4484-930f-501e9f6e6297\") " pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-zzvrc"
Nov 02 15:17:17.080349 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:17.080304    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6057a4f9-4a74-4484-930f-501e9f6e6297-istio-token\") pod \"e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-zzvrc\" (UID: \"6057a4f9-4a74-4484-930f-501e9f6e6297\") " pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-zzvrc"
Nov 02 15:17:17.080548 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:17.080516    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kwgl6\" (UniqueName: \"kubernetes.io/projected/6057a4f9-4a74-4484-930f-501e9f6e6297-kube-api-access-kwgl6\") pod \"e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-zzvrc\" (UID: \"6057a4f9-4a74-4484-930f-501e9f6e6297\") " pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-zzvrc"
Nov 02 15:17:17.107369 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:17.107332    5304 generic.go:356] "Generic (PLEG): container finished" podID="294f0d9e-6f89-477c-b882-61af0d1bcb5d" containerID="65ae4b6158338164d9f26b447de78665d60d5d9499452284efb07562830609b7" exitCode=0
Nov 02 15:17:17.107511 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:17.107397    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-d7mbd" event={"ID":"294f0d9e-6f89-477c-b882-61af0d1bcb5d","Type":"ContainerDied","Data":"65ae4b6158338164d9f26b447de78665d60d5d9499452284efb07562830609b7"}
Nov 02 15:17:17.107938 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:17.107910    5304 scope.go:122] "RemoveContainer" containerID="65ae4b6158338164d9f26b447de78665d60d5d9499452284efb07562830609b7"
Nov 02 15:17:17.109953 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:17.109921    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:17:17.110085 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:17.110002    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:17:17.110085 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:17.110048    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:17:17.110085 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:17.110082    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-cwnwk"
Nov 02 15:17:17.110270 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:17.110116    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-cwnwk" event={"ID":"79e8599a-22fa-4df3-a35a-4d16c271cfe2","Type":"ContainerDied","Data":"6076748778bb853bb1a421f610a8aa9a94ae135b15c3c6200f836acf6aba3a66"}
Nov 02 15:17:17.110270 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:17.110179    5304 scope.go:122] "RemoveContainer" containerID="165e1e14e0dc19782087cbf0db0d14128fb6fd4ec462abe1c9b3ae9d0ea73fe3"
Nov 02 15:17:17.112535 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:17.112500    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-b88h7" event={"ID":"3d4f7624-882c-4cb3-8811-9ca682e0c1d5","Type":"ContainerDied","Data":"19ed571e5ff8bcda7823a7e117f1d5b1b014be7c2df13bbd46a1a7072ec89928"}
Nov 02 15:17:17.112718 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:17.112508    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-b88h7"
Nov 02 15:17:17.118409 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:17.118249    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-bjrnm" event={"ID":"97220a55-1205-4d67-90ce-49d6cdef5c43","Type":"ContainerStarted","Data":"b5e96d2568c382ccc710ae185bbb4990490c1c53625be6b6cec6a6e7ef49cf2e"}
Nov 02 15:17:17.118409 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:17.118295    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-bjrnm" event={"ID":"97220a55-1205-4d67-90ce-49d6cdef5c43","Type":"ContainerStarted","Data":"8c337bfeaa73c82f070e30190adf0055a68a725ab26b07cab83321e43b6fe674"}
Nov 02 15:17:17.133548 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:17.133489    5304 scope.go:122] "RemoveContainer" containerID="f308b7b6046e7500e8e6d2d6f77c217a3ab3773533f03a01f6b15aa9ca7b1d96"
Nov 02 15:17:17.215004 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:17.214840    5304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-bjrnm"
Nov 02 15:17:17.218211 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:17.218184    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-b88h7"]
Nov 02 15:17:17.225147 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:17.224708    5304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-b88h7"]
Nov 02 15:17:17.235291 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:17.235257    5304 patch_prober.go:29] interesting pod/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-bjrnm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body=
Nov 02 15:17:17.235458 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:17.235332    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-bjrnm" podUID="97220a55-1205-4d67-90ce-49d6cdef5c43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused"
Nov 02 15:17:17.263827 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:17.263775    5304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-bjrnm" podStartSLOduration=2.263761059 podStartE2EDuration="2.263761059s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:17:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:17:17.259320112 +0000 UTC m=+1237.253144522" watchObservedRunningTime="2026-11-02 15:17:17.263761059 +0000 UTC m=+1237.257585467"
Nov 02 15:17:17.293539 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:17.293516    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-zzvrc"
Nov 02 15:17:17.307415 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:17.302607    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-cwnwk"]
Nov 02 15:17:17.316597 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:17.313449    5304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-cwnwk"]
Nov 02 15:17:17.729341 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:17.729306    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-zzvrc"]
Nov 02 15:17:17.730578 ip-10-0-135-138 kubenswrapper[5304]: W1102 15:17:17.730552    5304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6057a4f9_4a74_4484_930f_501e9f6e6297.slice/crio-6a8344e2b6688133c2c9456924381bef60c40591b2c6e52ed07523c052093fcd WatchSource:0}: Error finding container 6a8344e2b6688133c2c9456924381bef60c40591b2c6e52ed07523c052093fcd: Status 404 returned error can't find the container with id 6a8344e2b6688133c2c9456924381bef60c40591b2c6e52ed07523c052093fcd
Nov 02 15:17:17.735326 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:17.735283    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:17:17.735882 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:17.735856    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:17:17.736036 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:17.736017    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:17:18.183568 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:18.183246    5304 patch_prober.go:29] interesting pod/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-w9hxk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Nov 02 15:17:18.183568 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:18.183322    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-w9hxk" podUID="c41d5c79-8a9b-4ad5-995c-818fd3faf115" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Nov 02 15:17:18.200515 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:18.200041    5304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1bd23848-eb1d-45fd-9d87-c11302b3c111" path="/var/lib/kubelet/pods/1bd23848-eb1d-45fd-9d87-c11302b3c111/volumes"
Nov 02 15:17:18.201762 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:18.201735    5304 generic.go:356] "Generic (PLEG): container finished" podID="c41d5c79-8a9b-4ad5-995c-818fd3faf115" containerID="5726645d5a40c50de38f15f9278fddfc748e28e031f3a911b0055733b1c724f2" exitCode=0
Nov 02 15:17:18.202735 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:18.202624    5304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3d4f7624-882c-4cb3-8811-9ca682e0c1d5" path="/var/lib/kubelet/pods/3d4f7624-882c-4cb3-8811-9ca682e0c1d5/volumes"
Nov 02 15:17:18.203531 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:18.203504    5304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="79e8599a-22fa-4df3-a35a-4d16c271cfe2" path="/var/lib/kubelet/pods/79e8599a-22fa-4df3-a35a-4d16c271cfe2/volumes"
Nov 02 15:17:18.204117 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:18.204101    5304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dbef5be7-9697-4eb4-b8c7-cd985e58c6b6" path="/var/lib/kubelet/pods/dbef5be7-9697-4eb4-b8c7-cd985e58c6b6/volumes"
Nov 02 15:17:18.205183 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:18.205160    5304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fdbb90e5-42c8-4409-95e1-a44ed4abead2" path="/var/lib/kubelet/pods/fdbb90e5-42c8-4409-95e1-a44ed4abead2/volumes"
Nov 02 15:17:18.205667 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:18.205641    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-w9hxk" event={"ID":"c41d5c79-8a9b-4ad5-995c-818fd3faf115","Type":"ContainerDied","Data":"5726645d5a40c50de38f15f9278fddfc748e28e031f3a911b0055733b1c724f2"}
Nov 02 15:17:18.205763 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:18.205700    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-zzvrc" event={"ID":"6057a4f9-4a74-4484-930f-501e9f6e6297","Type":"ContainerStarted","Data":"4dd9daf69f867ab40a288f81eccf8fde5b4951b498e13172f52546fb41cc36d3"}
Nov 02 15:17:18.205763 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:18.205728    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-zzvrc" event={"ID":"6057a4f9-4a74-4484-930f-501e9f6e6297","Type":"ContainerStarted","Data":"6a8344e2b6688133c2c9456924381bef60c40591b2c6e52ed07523c052093fcd"}
Nov 02 15:17:18.206120 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:18.206098    5304 scope.go:122] "RemoveContainer" containerID="5726645d5a40c50de38f15f9278fddfc748e28e031f3a911b0055733b1c724f2"
Nov 02 15:17:18.207826 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:18.207791    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:17:18.207902 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:18.207874    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:17:18.207953 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:18.207913    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:17:18.210860 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:18.210836    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-d7mbd" event={"ID":"294f0d9e-6f89-477c-b882-61af0d1bcb5d","Type":"ContainerStarted","Data":"70a548f0546c3d97932ef0ce8342fe10d633fa2f51f4be994d663d8c3eb5ebf1"}
Nov 02 15:17:18.215604 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:18.215577    5304 patch_prober.go:29] interesting pod/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-bjrnm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body=
Nov 02 15:17:18.215692 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:18.215632    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-bjrnm" podUID="97220a55-1205-4d67-90ce-49d6cdef5c43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused"
Nov 02 15:17:18.289110 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:18.289036    5304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-zzvrc" podStartSLOduration=2.2890139769999998 podStartE2EDuration="2.289013977s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:17:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:17:18.255545263 +0000 UTC m=+1238.249369697" watchObservedRunningTime="2026-11-02 15:17:18.289013977 +0000 UTC m=+1238.282838389"
Nov 02 15:17:18.297864 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:18.294929    5304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-zzvrc"
Nov 02 15:17:18.298613 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:18.298071    5304 patch_prober.go:29] interesting pod/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-zzvrc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body=
Nov 02 15:17:18.298613 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:18.298191    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-zzvrc" podUID="6057a4f9-4a74-4484-930f-501e9f6e6297" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused"
Nov 02 15:17:19.079314 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:19.079255    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-d7mbd"
Nov 02 15:17:19.204238 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:19.204205    5304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-d7mbd"
Nov 02 15:17:19.204647 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:19.204616    5304 patch_prober.go:29] interesting pod/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-d7mbd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Nov 02 15:17:19.206398 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:19.206368    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-d7mbd" podUID="294f0d9e-6f89-477c-b882-61af0d1bcb5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Nov 02 15:17:19.216946 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:19.216912    5304 patch_prober.go:29] interesting pod/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-bjrnm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body=
Nov 02 15:17:19.223431 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:19.223276    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-bjrnm" podUID="97220a55-1205-4d67-90ce-49d6cdef5c43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused"
Nov 02 15:17:19.239207 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:19.233326    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-w9hxk" event={"ID":"c41d5c79-8a9b-4ad5-995c-818fd3faf115","Type":"ContainerStarted","Data":"d5135bb67ac3138b8ad8647cec43c2b60d9644c5642d5ea923599b45f37d7719"}
Nov 02 15:17:19.326601 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:19.326553    5304 patch_prober.go:29] interesting pod/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-zzvrc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body=
Nov 02 15:17:19.326808 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:19.326622    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-zzvrc" podUID="6057a4f9-4a74-4484-930f-501e9f6e6297" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused"
Nov 02 15:17:19.900160 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:19.898200    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-7nj6n"]
Nov 02 15:17:19.900160 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:19.898461    5304 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-7nj6n" podUID="3ebb8197-273d-4e96-bec7-00c7a85a12a9" containerName="istio-proxy" containerID="cri-o://db9ed33df2feb8a3dd2cdb179234648679154d230db955b2cc925161eb56b931" gracePeriod=30
Nov 02 15:17:19.900160 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:19.899926    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-6gnx4"]
Nov 02 15:17:19.927264 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:19.900120    5304 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-6gnx4" podUID="2afb5fd0-d309-42eb-bd1e-9092ddeb166a" containerName="istio-proxy" containerID="cri-o://62ba7ae8f5ec5e03bc76221962b6d5947c0efeb6b89e89f81fff181293061504" gracePeriod=30
Nov 02 15:17:20.223206 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:20.222161    5304 patch_prober.go:29] interesting pod/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-d7mbd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Nov 02 15:17:20.223206 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:20.222220    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-d7mbd" podUID="294f0d9e-6f89-477c-b882-61af0d1bcb5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Nov 02 15:17:20.249341 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:20.227530    5304 patch_prober.go:29] interesting pod/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-bjrnm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body=
Nov 02 15:17:20.249341 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:20.227585    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-bjrnm" podUID="97220a55-1205-4d67-90ce-49d6cdef5c43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused"
Nov 02 15:17:20.249341 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:20.228108    5304 patch_prober.go:29] interesting pod/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-w9hxk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Nov 02 15:17:20.249341 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:20.228181    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-w9hxk" podUID="c41d5c79-8a9b-4ad5-995c-818fd3faf115" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Nov 02 15:17:20.308164 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:20.295237    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-739bf1/simulator-w2-739bf1-kserve-7c74f85465-ltk4t" podUID="cf6a1ded-28a3-49eb-878b-b4ad77a0bb5c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:17:20.501236 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:20.431769    5304 patch_prober.go:29] interesting pod/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-zzvrc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body=
Nov 02 15:17:20.501236 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:20.431897    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-zzvrc" podUID="6057a4f9-4a74-4484-930f-501e9f6e6297" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused"
Nov 02 15:17:20.786245 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:20.760906    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nkbrk" podUID="1f68d837-fff2-418b-84d7-c2d7eb21a57e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.24:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:17:20.791571 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:20.764997    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5dd8e9/simulator-w1-5dd8e9-kserve-6f9f755d6f-gnw5d" podUID="66b75e64-cb24-40be-9076-4984c132e208" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:17:20.791571 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:20.765026    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1cc1b2/distinct-w5-1cc1b2-kserve-5b7459bc6d-zl8xr" podUID="211ad15d-9234-4f57-b1b9-181dd2d74208" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded"
Nov 02 15:17:20.791571 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:20.765040    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5dd8e9/premium-w1-5dd8e9-kserve-67588456fb-j6lph" podUID="4ccc607a-14e4-43ca-8999-78100d4b799f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded"
Nov 02 15:17:20.791571 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:20.765055    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1cc1b2/premium-w5-1cc1b2-kserve-58db46f97c-cgdqg" podUID="210f620a-8ae7-4e4c-acc3-46f75ddc6105" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:17:20.816326 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:20.816293    5304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-w9hxk"
Nov 02 15:17:20.839262 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:20.829657    5304 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-6af50f/distinct-w0-6af50f-kserve-c6d994647-hbj9z" podUID="3dca5bf3-8e08-47a0-936e-205670d59ede" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/health\": context deadline exceeded"
Nov 02 15:17:21.042207 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:21.041295    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6af50f/distinct-w0-6af50f-kserve-c6d994647-hbj9z" podUID="3dca5bf3-8e08-47a0-936e-205670d59ede" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded"
Nov 02 15:17:21.154151 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:21.153870    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-624254/distinct-w3-624254-kserve-5465dc4765-dmq67" podUID="1a5aa910-a6de-417b-8638-81e4ec5c1825" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded"
Nov 02 15:17:21.154358 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:21.154299    5304 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-1cc1b2/premium-w5-1cc1b2-kserve-58db46f97c-cgdqg" podUID="210f620a-8ae7-4e4c-acc3-46f75ddc6105" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/health\": context deadline exceeded"
Nov 02 15:17:21.209152 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:21.208164    5304 patch_prober.go:29] interesting pod/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-d7mbd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Nov 02 15:17:21.209152 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:21.208237    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-d7mbd" podUID="294f0d9e-6f89-477c-b882-61af0d1bcb5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Nov 02 15:17:21.209152 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:21.208953    5304 patch_prober.go:29] interesting pod/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-w9hxk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Nov 02 15:17:21.209152 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:21.209004    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-w9hxk" podUID="c41d5c79-8a9b-4ad5-995c-818fd3faf115" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Nov 02 15:17:21.209152 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:21.209090    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-739bf1/distinct-2-w2-739bf1-kserve-5c58479c48-qgfkh" podUID="9746bf8a-91bd-4715-a02e-ae2e651ca54d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:17:21.218493 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:21.218462    5304 patch_prober.go:29] interesting pod/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-bjrnm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body=
Nov 02 15:17:21.218713 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:21.218687    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-bjrnm" podUID="97220a55-1205-4d67-90ce-49d6cdef5c43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused"
Nov 02 15:17:21.295662 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:21.295542    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-739bf1/simulator-w2-739bf1-kserve-7c74f85465-ltk4t" podUID="cf6a1ded-28a3-49eb-878b-b4ad77a0bb5c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded"
Nov 02 15:17:21.296140 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:21.295975    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-739bf1/simulator-w2-739bf1-kserve-7c74f85465-ltk4t"
Nov 02 15:17:21.312152 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:21.306767    5304 patch_prober.go:29] interesting pod/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-zzvrc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body=
Nov 02 15:17:21.312152 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:21.306830    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-zzvrc" podUID="6057a4f9-4a74-4484-930f-501e9f6e6297" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused"
Nov 02 15:17:21.388150 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:21.383380    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-624254/simulator-w3-624254-kserve-56866b7699-z47sd" podUID="345c710f-b414-4e62-86c7-d51ff4e5be15" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:17:21.388150 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:21.383805    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7ecfe1/distinct-w4-7ecfe1-kserve-6c847c647d-d2jvv" podUID="f1f1ca5e-cca7-4c54-939f-06f095defcdd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded"
Nov 02 15:17:21.605153 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:21.586432    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5113c2/premium-w6-5113c2-kserve-b84945cb5-pc6f8" podUID="9fb3e992-7bb0-497a-a82c-6e396ba241c1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:17:21.797070 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:21.797032    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1cc1b2/premium-w5-1cc1b2-kserve-58db46f97c-cgdqg" podUID="210f620a-8ae7-4e4c-acc3-46f75ddc6105" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:17:21.797672 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:21.797641    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5dd8e9/simulator-w1-5dd8e9-kserve-6f9f755d6f-gnw5d" podUID="66b75e64-cb24-40be-9076-4984c132e208" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": context deadline exceeded"
Nov 02 15:17:21.798119 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:21.798094    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1cc1b2/distinct-w5-1cc1b2-kserve-5b7459bc6d-zl8xr" podUID="211ad15d-9234-4f57-b1b9-181dd2d74208" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded"
Nov 02 15:17:21.798532 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:21.798511    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5dd8e9/premium-w1-5dd8e9-kserve-67588456fb-j6lph" podUID="4ccc607a-14e4-43ca-8999-78100d4b799f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded"
Nov 02 15:17:21.809151 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:21.798918    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-1cc1b2/premium-w5-1cc1b2-kserve-58db46f97c-cgdqg"
Nov 02 15:17:21.835173 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:21.815208    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nkbrk" podUID="1f68d837-fff2-418b-84d7-c2d7eb21a57e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.24:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:17:21.835173 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:21.818253    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5dd8e9/simulator-w1-5dd8e9-kserve-6f9f755d6f-gnw5d"
Nov 02 15:17:21.835173 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:21.818728    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-1cc1b2/distinct-w5-1cc1b2-kserve-5b7459bc6d-zl8xr"
Nov 02 15:17:21.835173 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:21.818785    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5dd8e9/premium-w1-5dd8e9-kserve-67588456fb-j6lph"
Nov 02 15:17:21.835173 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:21.818821    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nkbrk"
Nov 02 15:17:22.114211 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:22.058743    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6af50f/distinct-w0-6af50f-kserve-c6d994647-hbj9z" podUID="3dca5bf3-8e08-47a0-936e-205670d59ede" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:17:22.178164 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:22.175652    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-624254/distinct-w3-624254-kserve-5465dc4765-dmq67" podUID="1a5aa910-a6de-417b-8638-81e4ec5c1825" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:17:22.249476 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:22.222963    5304 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-624254/simulator-w3-624254-kserve-56866b7699-z47sd" podUID="345c710f-b414-4e62-86c7-d51ff4e5be15" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:17:22.249476 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:22.223657    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-739bf1/distinct-2-w2-739bf1-kserve-5c58479c48-qgfkh" podUID="9746bf8a-91bd-4715-a02e-ae2e651ca54d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded"
Nov 02 15:17:22.325159 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:22.305647    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-739bf1/simulator-w2-739bf1-kserve-7c74f85465-ltk4t" podUID="cf6a1ded-28a3-49eb-878b-b4ad77a0bb5c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:17:22.360877 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:22.360838    5304 patch_prober.go:29] interesting pod/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-zzvrc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body=
Nov 02 15:17:22.361174 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:22.361142    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-zzvrc" podUID="6057a4f9-4a74-4484-930f-501e9f6e6297" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused"
Nov 02 15:17:22.386165 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:22.361952    5304 patch_prober.go:29] interesting pod/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-bjrnm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body=
Nov 02 15:17:22.386472 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:22.386431    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-bjrnm" podUID="97220a55-1205-4d67-90ce-49d6cdef5c43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused"
Nov 02 15:17:22.442156 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:22.433953    5304 patch_prober.go:29] interesting pod/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-d7mbd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Nov 02 15:17:22.469153 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:22.364574    5304 patch_prober.go:29] interesting pod/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-w9hxk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Nov 02 15:17:22.469153 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:22.465439    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-w9hxk" podUID="c41d5c79-8a9b-4ad5-995c-818fd3faf115" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Nov 02 15:17:22.469153 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:22.434727    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7ecfe1/distinct-w4-7ecfe1-kserve-6c847c647d-d2jvv" podUID="f1f1ca5e-cca7-4c54-939f-06f095defcdd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded"
Nov 02 15:17:22.469153 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:22.465509    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-d7mbd" podUID="294f0d9e-6f89-477c-b882-61af0d1bcb5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Nov 02 15:17:22.469153 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:22.434700    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-624254/simulator-w3-624254-kserve-56866b7699-z47sd" podUID="345c710f-b414-4e62-86c7-d51ff4e5be15" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": context deadline exceeded"
Nov 02 15:17:22.807839 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:22.807730    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5dd8e9/premium-w1-5dd8e9-kserve-67588456fb-j6lph" podUID="4ccc607a-14e4-43ca-8999-78100d4b799f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:17:22.822446 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:22.816652    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1cc1b2/premium-w5-1cc1b2-kserve-58db46f97c-cgdqg" podUID="210f620a-8ae7-4e4c-acc3-46f75ddc6105" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:17:22.852204 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:22.826646    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5dd8e9/simulator-w1-5dd8e9-kserve-6f9f755d6f-gnw5d" podUID="66b75e64-cb24-40be-9076-4984c132e208" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": context deadline exceeded"
Nov 02 15:17:22.852784 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:22.828376    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-6af50f/distinct-w0-6af50f-kserve-c6d994647-hbj9z"
Nov 02 15:17:22.853048 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:22.853033    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-624254/distinct-w3-624254-kserve-5465dc4765-dmq67"
Nov 02 15:17:22.853193 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:22.853182    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-739bf1/distinct-2-w2-739bf1-kserve-5c58479c48-qgfkh"
Nov 02 15:17:22.853303 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:22.853294    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-7ecfe1/distinct-w4-7ecfe1-kserve-6c847c647d-d2jvv"
Nov 02 15:17:22.878301 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:22.878261    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-624254/simulator-w3-624254-kserve-56866b7699-z47sd"
Nov 02 15:17:22.889153 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:22.839438    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1cc1b2/distinct-w5-1cc1b2-kserve-5b7459bc6d-zl8xr" podUID="211ad15d-9234-4f57-b1b9-181dd2d74208" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded"
Nov 02 15:17:22.931166 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:22.840533    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nkbrk" podUID="1f68d837-fff2-418b-84d7-c2d7eb21a57e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.24:8000/ready\": context deadline exceeded"
Nov 02 15:17:22.957156 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:22.956388    5304 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-7ecfe1/distinct-w4-7ecfe1-kserve-6c847c647d-d2jvv" podUID="f1f1ca5e-cca7-4c54-939f-06f095defcdd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:17:23.127689 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:23.127646    5304 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nkbrk" podUID="1f68d837-fff2-418b-84d7-c2d7eb21a57e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.24:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:17:23.160437 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:23.156321    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6af50f/distinct-w0-6af50f-kserve-c6d994647-hbj9z" podUID="3dca5bf3-8e08-47a0-936e-205670d59ede" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:17:23.220401 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:23.201461    5304 patch_prober.go:29] interesting pod/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-w9hxk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Nov 02 15:17:23.220401 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:23.201523    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-w9hxk" podUID="c41d5c79-8a9b-4ad5-995c-818fd3faf115" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Nov 02 15:17:23.220401 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:23.202805    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-624254/distinct-w3-624254-kserve-5465dc4765-dmq67" podUID="1a5aa910-a6de-417b-8638-81e4ec5c1825" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:17:23.247155 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:23.228216    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-739bf1/distinct-2-w2-739bf1-kserve-5c58479c48-qgfkh" podUID="9746bf8a-91bd-4715-a02e-ae2e651ca54d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:17:23.275150 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:23.267836    5304 patch_prober.go:29] interesting pod/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-bjrnm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body=
Nov 02 15:17:23.275150 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:23.267907    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-bjrnm" podUID="97220a55-1205-4d67-90ce-49d6cdef5c43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused"
Nov 02 15:17:23.308200 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:23.295916    5304 patch_prober.go:29] interesting pod/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-d7mbd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Nov 02 15:17:23.308200 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:23.295982    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-d7mbd" podUID="294f0d9e-6f89-477c-b882-61af0d1bcb5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Nov 02 15:17:23.326116 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:23.326070    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-739bf1/simulator-w2-739bf1-kserve-7c74f85465-ltk4t" podUID="cf6a1ded-28a3-49eb-878b-b4ad77a0bb5c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:17:23.441597 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:23.441480    5304 patch_prober.go:29] interesting pod/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-zzvrc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body=
Nov 02 15:17:23.445337 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:23.445228    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-zzvrc" podUID="6057a4f9-4a74-4484-930f-501e9f6e6297" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused"
Nov 02 15:17:23.494448 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:23.494391    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5113c2/premium-w6-5113c2-kserve-b84945cb5-pc6f8" podUID="9fb3e992-7bb0-497a-a82c-6e396ba241c1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded"
Nov 02 15:17:23.495042 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:23.495012    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-624254/simulator-w3-624254-kserve-56866b7699-z47sd" podUID="345c710f-b414-4e62-86c7-d51ff4e5be15" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:17:23.495525 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:23.495502    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7ecfe1/distinct-w4-7ecfe1-kserve-6c847c647d-d2jvv" podUID="f1f1ca5e-cca7-4c54-939f-06f095defcdd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:17:23.917261 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:23.916245    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nkbrk" podUID="1f68d837-fff2-418b-84d7-c2d7eb21a57e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.24:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:17:23.921105 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:23.921065    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5dd8e9/premium-w1-5dd8e9-kserve-67588456fb-j6lph" podUID="4ccc607a-14e4-43ca-8999-78100d4b799f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded"
Nov 02 15:17:23.928152 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:23.925083    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1cc1b2/premium-w5-1cc1b2-kserve-58db46f97c-cgdqg" podUID="210f620a-8ae7-4e4c-acc3-46f75ddc6105" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded"
Nov 02 15:17:23.930295 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:23.930264    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5dd8e9/simulator-w1-5dd8e9-kserve-6f9f755d6f-gnw5d" podUID="66b75e64-cb24-40be-9076-4984c132e208" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": context deadline exceeded"
Nov 02 15:17:23.939751 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:23.939714    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1cc1b2/distinct-w5-1cc1b2-kserve-5b7459bc6d-zl8xr" podUID="211ad15d-9234-4f57-b1b9-181dd2d74208" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded"
Nov 02 15:17:24.071170 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:24.067328    5304 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-5dd8e9/premium-w1-5dd8e9-kserve-67588456fb-j6lph" podUID="4ccc607a-14e4-43ca-8999-78100d4b799f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:17:24.193351 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:24.193258    5304 patch_prober.go:29] interesting pod/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-w9hxk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Nov 02 15:17:24.193649 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:24.193617    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-w9hxk" podUID="c41d5c79-8a9b-4ad5-995c-818fd3faf115" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Nov 02 15:17:24.300209 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:24.214319    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-624254/distinct-w3-624254-kserve-5465dc4765-dmq67" podUID="1a5aa910-a6de-417b-8638-81e4ec5c1825" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:17:24.300209 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:24.241534    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-739bf1/distinct-2-w2-739bf1-kserve-5c58479c48-qgfkh" podUID="9746bf8a-91bd-4715-a02e-ae2e651ca54d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:17:24.300209 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:24.242407    5304 patch_prober.go:29] interesting pod/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-d7mbd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Nov 02 15:17:24.300209 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:24.242479    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-d7mbd" podUID="294f0d9e-6f89-477c-b882-61af0d1bcb5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Nov 02 15:17:24.300209 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:24.267216    5304 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-1cc1b2/distinct-w5-1cc1b2-kserve-5b7459bc6d-zl8xr" podUID="211ad15d-9234-4f57-b1b9-181dd2d74208" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/health\": context deadline exceeded"
Nov 02 15:17:24.300906 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:24.178254    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6af50f/distinct-w0-6af50f-kserve-c6d994647-hbj9z" podUID="3dca5bf3-8e08-47a0-936e-205670d59ede" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded"
Nov 02 15:17:24.328298 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:24.328252    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-739bf1/simulator-w2-739bf1-kserve-7c74f85465-ltk4t" podUID="cf6a1ded-28a3-49eb-878b-b4ad77a0bb5c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:17:24.376240 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:24.359648    5304 patch_prober.go:29] interesting pod/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-zzvrc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body=
Nov 02 15:17:24.376240 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:24.359725    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-zzvrc" podUID="6057a4f9-4a74-4484-930f-501e9f6e6297" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused"
Nov 02 15:17:24.454193 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:24.454068    5304 patch_prober.go:29] interesting pod/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-bjrnm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body=
Nov 02 15:17:24.460222 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:24.454442    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-bjrnm" podUID="97220a55-1205-4d67-90ce-49d6cdef5c43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused"
Nov 02 15:17:24.511158 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:24.508901    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7ecfe1/distinct-w4-7ecfe1-kserve-6c847c647d-d2jvv" podUID="f1f1ca5e-cca7-4c54-939f-06f095defcdd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:17:24.714378 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:24.692301    5304 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-739bf1/distinct-2-w2-739bf1-kserve-5c58479c48-qgfkh" podUID="9746bf8a-91bd-4715-a02e-ae2e651ca54d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:17:24.714960 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:24.714920    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5113c2/premium-w6-5113c2-kserve-b84945cb5-pc6f8" podUID="9fb3e992-7bb0-497a-a82c-6e396ba241c1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded"
Nov 02 15:17:24.715443 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:24.715418    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-624254/simulator-w3-624254-kserve-56866b7699-z47sd" podUID="345c710f-b414-4e62-86c7-d51ff4e5be15" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:17:24.964724 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:24.964626    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1cc1b2/premium-w5-1cc1b2-kserve-58db46f97c-cgdqg" podUID="210f620a-8ae7-4e4c-acc3-46f75ddc6105" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:17:24.967261 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:24.967231    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-0a7163-8dd8847c5-5s25z"
Nov 02 15:17:24.967496 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:24.967478    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-5113c2/premium-w6-5113c2-kserve-b84945cb5-pc6f8"
Nov 02 15:17:24.967645 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:24.967631    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nkbrk"
Nov 02 15:17:24.971234 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:24.971209    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-1cc1b2/distinct-w5-1cc1b2-kserve-5b7459bc6d-zl8xr"
Nov 02 15:17:24.971399 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:24.971384    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5dd8e9/simulator-w1-5dd8e9-kserve-6f9f755d6f-gnw5d"
Nov 02 15:17:24.998361 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:24.998322    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5dd8e9/premium-w1-5dd8e9-kserve-67588456fb-j6lph" podUID="4ccc607a-14e4-43ca-8999-78100d4b799f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:17:25.030600 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:25.028151    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-624254/simulator-w3-624254-kserve-56866b7699-z47sd"
Nov 02 15:17:25.127333 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:25.127300    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-7ecfe1/distinct-w4-7ecfe1-kserve-6c847c647d-d2jvv"
Nov 02 15:17:25.160884 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:25.160851    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-739bf1/distinct-2-w2-739bf1-kserve-5c58479c48-qgfkh"
Nov 02 15:17:25.163339 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:25.161006    5304 patch_prober.go:29] interesting pod/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-w9hxk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Nov 02 15:17:25.164825 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:25.164798    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-w9hxk" podUID="c41d5c79-8a9b-4ad5-995c-818fd3faf115" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Nov 02 15:17:25.165305 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:25.164358    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-5113c2/premium-w6-5113c2-kserve-b84945cb5-pc6f8"
Nov 02 15:17:25.173572 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:25.173538    5304 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-5113c2/premium-w6-5113c2-kserve-b84945cb5-pc6f8" podUID="9fb3e992-7bb0-497a-a82c-6e396ba241c1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:17:25.228218 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:25.227418    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-624254/distinct-w3-624254-kserve-5465dc4765-dmq67" podUID="1a5aa910-a6de-417b-8638-81e4ec5c1825" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:17:25.254321 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:25.229044    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-739bf1/simulator-w2-739bf1-kserve-7c74f85465-ltk4t"
Nov 02 15:17:25.279186 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:25.229951    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5dd8e9/premium-w1-5dd8e9-kserve-67588456fb-j6lph"
Nov 02 15:17:25.279186 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:25.275516    5304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-d7mbd"
Nov 02 15:17:25.279911 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:25.279810    5304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-bjrnm"
Nov 02 15:17:25.313260 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:25.298823    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-bjrnm"
Nov 02 15:17:25.313260 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:25.302169    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-bjrnm"
Nov 02 15:17:25.313260 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:25.303846    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-d7mbd"
Nov 02 15:17:25.313260 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:25.304335    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-1cc1b2/premium-w5-1cc1b2-kserve-58db46f97c-cgdqg"
Nov 02 15:17:25.313260 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:25.305539    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6af50f/distinct-w0-6af50f-kserve-c6d994647-hbj9z" podUID="3dca5bf3-8e08-47a0-936e-205670d59ede" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:17:25.322398 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:25.306380    5304 patch_prober.go:29] interesting pod/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-zzvrc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body=
Nov 02 15:17:25.322398 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:25.322359    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-zzvrc" podUID="6057a4f9-4a74-4484-930f-501e9f6e6297" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused"
Nov 02 15:17:25.324760 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:25.314683    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-624254/distinct-w3-624254-kserve-5465dc4765-dmq67"
Nov 02 15:17:25.326704 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:25.325842    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-6af50f/distinct-w0-6af50f-kserve-c6d994647-hbj9z"
Nov 02 15:17:25.704635 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:25.704602    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-7nj6n"
Nov 02 15:17:25.750366 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:17:25.750329    5304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k6fjk" containerName="extractor"
Nov 02 15:17:25.801689 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:25.801652    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-6gnx4"
Nov 02 15:17:25.802271 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:25.802237    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ebb8197-273d-4e96-bec7-00c7a85a12a9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ebb8197-273d-4e96-bec7-00c7a85a12a9-istio-envoy\") pod \"3ebb8197-273d-4e96-bec7-00c7a85a12a9\" (UID: \"3ebb8197-273d-4e96-bec7-00c7a85a12a9\") "
Nov 02 15:17:25.802418 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:25.802299    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ebb8197-273d-4e96-bec7-00c7a85a12a9-kube-api-access-7lggw\" (UniqueName: \"kubernetes.io/projected/3ebb8197-273d-4e96-bec7-00c7a85a12a9-kube-api-access-7lggw\") pod \"3ebb8197-273d-4e96-bec7-00c7a85a12a9\" (UID: \"3ebb8197-273d-4e96-bec7-00c7a85a12a9\") "
Nov 02 15:17:25.802418 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:25.802338    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3ebb8197-273d-4e96-bec7-00c7a85a12a9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ebb8197-273d-4e96-bec7-00c7a85a12a9-istiod-ca-cert\") pod \"3ebb8197-273d-4e96-bec7-00c7a85a12a9\" (UID: \"3ebb8197-273d-4e96-bec7-00c7a85a12a9\") "
Nov 02 15:17:25.802502 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:25.802472    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ebb8197-273d-4e96-bec7-00c7a85a12a9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ebb8197-273d-4e96-bec7-00c7a85a12a9-workload-socket\") pod \"3ebb8197-273d-4e96-bec7-00c7a85a12a9\" (UID: \"3ebb8197-273d-4e96-bec7-00c7a85a12a9\") "
Nov 02 15:17:25.802545 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:25.802500    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ebb8197-273d-4e96-bec7-00c7a85a12a9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ebb8197-273d-4e96-bec7-00c7a85a12a9-istio-data\") pod \"3ebb8197-273d-4e96-bec7-00c7a85a12a9\" (UID: \"3ebb8197-273d-4e96-bec7-00c7a85a12a9\") "
Nov 02 15:17:25.802586 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:25.802547    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ebb8197-273d-4e96-bec7-00c7a85a12a9-istio-token\" (UniqueName: \"kubernetes.io/projected/3ebb8197-273d-4e96-bec7-00c7a85a12a9-istio-token\") pod \"3ebb8197-273d-4e96-bec7-00c7a85a12a9\" (UID: \"3ebb8197-273d-4e96-bec7-00c7a85a12a9\") "
Nov 02 15:17:25.802637 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:25.802620    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ebb8197-273d-4e96-bec7-00c7a85a12a9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ebb8197-273d-4e96-bec7-00c7a85a12a9-workload-certs\") pod \"3ebb8197-273d-4e96-bec7-00c7a85a12a9\" (UID: \"3ebb8197-273d-4e96-bec7-00c7a85a12a9\") "
Nov 02 15:17:25.802691 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:25.802662    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3ebb8197-273d-4e96-bec7-00c7a85a12a9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ebb8197-273d-4e96-bec7-00c7a85a12a9-istio-podinfo\") pod \"3ebb8197-273d-4e96-bec7-00c7a85a12a9\" (UID: \"3ebb8197-273d-4e96-bec7-00c7a85a12a9\") "
Nov 02 15:17:25.802842 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:25.802828    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ebb8197-273d-4e96-bec7-00c7a85a12a9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ebb8197-273d-4e96-bec7-00c7a85a12a9-credential-socket\") pod \"3ebb8197-273d-4e96-bec7-00c7a85a12a9\" (UID: \"3ebb8197-273d-4e96-bec7-00c7a85a12a9\") "
Nov 02 15:17:25.804450 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:25.803896    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ebb8197-273d-4e96-bec7-00c7a85a12a9-istio-data" pod "3ebb8197-273d-4e96-bec7-00c7a85a12a9" (UID: "3ebb8197-273d-4e96-bec7-00c7a85a12a9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:17:25.804450 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:25.803912    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ebb8197-273d-4e96-bec7-00c7a85a12a9-istiod-ca-cert" pod "3ebb8197-273d-4e96-bec7-00c7a85a12a9" (UID: "3ebb8197-273d-4e96-bec7-00c7a85a12a9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:17:25.804450 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:25.804159    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ebb8197-273d-4e96-bec7-00c7a85a12a9-workload-socket" pod "3ebb8197-273d-4e96-bec7-00c7a85a12a9" (UID: "3ebb8197-273d-4e96-bec7-00c7a85a12a9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:17:25.804722 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:25.804698    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ebb8197-273d-4e96-bec7-00c7a85a12a9-credential-socket" pod "3ebb8197-273d-4e96-bec7-00c7a85a12a9" (UID: "3ebb8197-273d-4e96-bec7-00c7a85a12a9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:17:25.804722 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:25.804498    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ebb8197-273d-4e96-bec7-00c7a85a12a9-workload-certs" pod "3ebb8197-273d-4e96-bec7-00c7a85a12a9" (UID: "3ebb8197-273d-4e96-bec7-00c7a85a12a9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:17:25.805448 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:25.805358    5304 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ebb8197-273d-4e96-bec7-00c7a85a12a9-workload-certs\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:17:25.805448 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:25.805385    5304 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ebb8197-273d-4e96-bec7-00c7a85a12a9-credential-socket\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:17:25.805448 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:25.805399    5304 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ebb8197-273d-4e96-bec7-00c7a85a12a9-istiod-ca-cert\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:17:25.805448 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:25.805413    5304 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ebb8197-273d-4e96-bec7-00c7a85a12a9-workload-socket\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:17:25.805448 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:25.805427    5304 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ebb8197-273d-4e96-bec7-00c7a85a12a9-istio-data\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:17:25.806768 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:25.806741    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ebb8197-273d-4e96-bec7-00c7a85a12a9-kube-api-access-7lggw" pod "3ebb8197-273d-4e96-bec7-00c7a85a12a9" (UID: "3ebb8197-273d-4e96-bec7-00c7a85a12a9"). InnerVolumeSpecName "kube-api-access-7lggw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:17:25.807688 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:25.807609    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ebb8197-273d-4e96-bec7-00c7a85a12a9-istio-token" pod "3ebb8197-273d-4e96-bec7-00c7a85a12a9" (UID: "3ebb8197-273d-4e96-bec7-00c7a85a12a9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:17:25.807949 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:25.807912    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3ebb8197-273d-4e96-bec7-00c7a85a12a9-istio-podinfo" pod "3ebb8197-273d-4e96-bec7-00c7a85a12a9" (UID: "3ebb8197-273d-4e96-bec7-00c7a85a12a9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 15:17:25.811553 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:25.811502    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ebb8197-273d-4e96-bec7-00c7a85a12a9-istio-envoy" pod "3ebb8197-273d-4e96-bec7-00c7a85a12a9" (UID: "3ebb8197-273d-4e96-bec7-00c7a85a12a9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:17:25.906846 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:25.906757    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2afb5fd0-d309-42eb-bd1e-9092ddeb166a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2afb5fd0-d309-42eb-bd1e-9092ddeb166a-workload-certs\") pod \"2afb5fd0-d309-42eb-bd1e-9092ddeb166a\" (UID: \"2afb5fd0-d309-42eb-bd1e-9092ddeb166a\") "
Nov 02 15:17:25.907026 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:25.906855    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2afb5fd0-d309-42eb-bd1e-9092ddeb166a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2afb5fd0-d309-42eb-bd1e-9092ddeb166a-istiod-ca-cert\") pod \"2afb5fd0-d309-42eb-bd1e-9092ddeb166a\" (UID: \"2afb5fd0-d309-42eb-bd1e-9092ddeb166a\") "
Nov 02 15:17:25.907026 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:25.906974    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2afb5fd0-d309-42eb-bd1e-9092ddeb166a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2afb5fd0-d309-42eb-bd1e-9092ddeb166a-istio-podinfo\") pod \"2afb5fd0-d309-42eb-bd1e-9092ddeb166a\" (UID: \"2afb5fd0-d309-42eb-bd1e-9092ddeb166a\") "
Nov 02 15:17:25.907026 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:25.907010    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2afb5fd0-d309-42eb-bd1e-9092ddeb166a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2afb5fd0-d309-42eb-bd1e-9092ddeb166a-credential-socket\") pod \"2afb5fd0-d309-42eb-bd1e-9092ddeb166a\" (UID: \"2afb5fd0-d309-42eb-bd1e-9092ddeb166a\") "
Nov 02 15:17:25.907209 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:25.907088    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2afb5fd0-d309-42eb-bd1e-9092ddeb166a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2afb5fd0-d309-42eb-bd1e-9092ddeb166a-istio-data\") pod \"2afb5fd0-d309-42eb-bd1e-9092ddeb166a\" (UID: \"2afb5fd0-d309-42eb-bd1e-9092ddeb166a\") "
Nov 02 15:17:25.907209 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:25.907084    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2afb5fd0-d309-42eb-bd1e-9092ddeb166a-workload-certs" pod "2afb5fd0-d309-42eb-bd1e-9092ddeb166a" (UID: "2afb5fd0-d309-42eb-bd1e-9092ddeb166a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:17:25.907209 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:25.907156    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2afb5fd0-d309-42eb-bd1e-9092ddeb166a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2afb5fd0-d309-42eb-bd1e-9092ddeb166a-workload-socket\") pod \"2afb5fd0-d309-42eb-bd1e-9092ddeb166a\" (UID: \"2afb5fd0-d309-42eb-bd1e-9092ddeb166a\") "
Nov 02 15:17:25.907209 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:25.907198    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2afb5fd0-d309-42eb-bd1e-9092ddeb166a-istio-token\" (UniqueName: \"kubernetes.io/projected/2afb5fd0-d309-42eb-bd1e-9092ddeb166a-istio-token\") pod \"2afb5fd0-d309-42eb-bd1e-9092ddeb166a\" (UID: \"2afb5fd0-d309-42eb-bd1e-9092ddeb166a\") "
Nov 02 15:17:25.907419 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:25.907289    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2afb5fd0-d309-42eb-bd1e-9092ddeb166a-kube-api-access-tqcjm\" (UniqueName: \"kubernetes.io/projected/2afb5fd0-d309-42eb-bd1e-9092ddeb166a-kube-api-access-tqcjm\") pod \"2afb5fd0-d309-42eb-bd1e-9092ddeb166a\" (UID: \"2afb5fd0-d309-42eb-bd1e-9092ddeb166a\") "
Nov 02 15:17:25.907419 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:25.907336    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2afb5fd0-d309-42eb-bd1e-9092ddeb166a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2afb5fd0-d309-42eb-bd1e-9092ddeb166a-istio-envoy\") pod \"2afb5fd0-d309-42eb-bd1e-9092ddeb166a\" (UID: \"2afb5fd0-d309-42eb-bd1e-9092ddeb166a\") "
Nov 02 15:17:25.907419 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:25.907340    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2afb5fd0-d309-42eb-bd1e-9092ddeb166a-credential-socket" pod "2afb5fd0-d309-42eb-bd1e-9092ddeb166a" (UID: "2afb5fd0-d309-42eb-bd1e-9092ddeb166a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:17:25.907419 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:25.907355    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2afb5fd0-d309-42eb-bd1e-9092ddeb166a-istio-data" pod "2afb5fd0-d309-42eb-bd1e-9092ddeb166a" (UID: "2afb5fd0-d309-42eb-bd1e-9092ddeb166a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:17:25.907419 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:25.907387    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2afb5fd0-d309-42eb-bd1e-9092ddeb166a-istiod-ca-cert" pod "2afb5fd0-d309-42eb-bd1e-9092ddeb166a" (UID: "2afb5fd0-d309-42eb-bd1e-9092ddeb166a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:17:25.907668 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:25.907602    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2afb5fd0-d309-42eb-bd1e-9092ddeb166a-workload-socket" pod "2afb5fd0-d309-42eb-bd1e-9092ddeb166a" (UID: "2afb5fd0-d309-42eb-bd1e-9092ddeb166a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:17:25.908230 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:25.908203    5304 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2afb5fd0-d309-42eb-bd1e-9092ddeb166a-credential-socket\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:17:25.908370 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:25.908232    5304 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ebb8197-273d-4e96-bec7-00c7a85a12a9-istio-envoy\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:17:25.908370 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:25.908247    5304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7lggw\" (UniqueName: \"kubernetes.io/projected/3ebb8197-273d-4e96-bec7-00c7a85a12a9-kube-api-access-7lggw\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:17:25.908370 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:25.908263    5304 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2afb5fd0-d309-42eb-bd1e-9092ddeb166a-istio-data\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:17:25.908370 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:25.908279    5304 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2afb5fd0-d309-42eb-bd1e-9092ddeb166a-workload-socket\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:17:25.908370 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:25.908293    5304 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ebb8197-273d-4e96-bec7-00c7a85a12a9-istio-token\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:17:25.908370 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:25.908310    5304 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ebb8197-273d-4e96-bec7-00c7a85a12a9-istio-podinfo\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:17:25.908370 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:25.908323    5304 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2afb5fd0-d309-42eb-bd1e-9092ddeb166a-workload-certs\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:17:25.908370 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:25.908339    5304 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2afb5fd0-d309-42eb-bd1e-9092ddeb166a-istiod-ca-cert\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:17:25.910103 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:25.910077    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2afb5fd0-d309-42eb-bd1e-9092ddeb166a-istio-token" pod "2afb5fd0-d309-42eb-bd1e-9092ddeb166a" (UID: "2afb5fd0-d309-42eb-bd1e-9092ddeb166a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:17:25.910345 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:25.910322    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2afb5fd0-d309-42eb-bd1e-9092ddeb166a-istio-podinfo" pod "2afb5fd0-d309-42eb-bd1e-9092ddeb166a" (UID: "2afb5fd0-d309-42eb-bd1e-9092ddeb166a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 15:17:25.910455 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:25.910437    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2afb5fd0-d309-42eb-bd1e-9092ddeb166a-kube-api-access-tqcjm" pod "2afb5fd0-d309-42eb-bd1e-9092ddeb166a" (UID: "2afb5fd0-d309-42eb-bd1e-9092ddeb166a"). InnerVolumeSpecName "kube-api-access-tqcjm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:17:25.910610 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:25.910593    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2afb5fd0-d309-42eb-bd1e-9092ddeb166a-istio-envoy" pod "2afb5fd0-d309-42eb-bd1e-9092ddeb166a" (UID: "2afb5fd0-d309-42eb-bd1e-9092ddeb166a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:17:25.976617 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:25.975677    5304 generic.go:356] "Generic (PLEG): container finished" podID="3ebb8197-273d-4e96-bec7-00c7a85a12a9" containerID="db9ed33df2feb8a3dd2cdb179234648679154d230db955b2cc925161eb56b931" exitCode=0
Nov 02 15:17:25.976617 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:25.975733    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-7nj6n"
Nov 02 15:17:25.976617 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:25.975764    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-7nj6n" event={"ID":"3ebb8197-273d-4e96-bec7-00c7a85a12a9","Type":"ContainerDied","Data":"db9ed33df2feb8a3dd2cdb179234648679154d230db955b2cc925161eb56b931"}
Nov 02 15:17:25.976617 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:25.975849    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-7nj6n" event={"ID":"3ebb8197-273d-4e96-bec7-00c7a85a12a9","Type":"ContainerDied","Data":"196ab17e3dbd9dafa9eb688e338d605d2c808cb03ba2747bc309835883d6238b"}
Nov 02 15:17:25.976617 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:25.975882    5304 scope.go:122] "RemoveContainer" containerID="db9ed33df2feb8a3dd2cdb179234648679154d230db955b2cc925161eb56b931"
Nov 02 15:17:25.979201 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:25.978935    5304 generic.go:356] "Generic (PLEG): container finished" podID="2afb5fd0-d309-42eb-bd1e-9092ddeb166a" containerID="62ba7ae8f5ec5e03bc76221962b6d5947c0efeb6b89e89f81fff181293061504" exitCode=0
Nov 02 15:17:25.979201 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:25.978968    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-6gnx4" event={"ID":"2afb5fd0-d309-42eb-bd1e-9092ddeb166a","Type":"ContainerDied","Data":"62ba7ae8f5ec5e03bc76221962b6d5947c0efeb6b89e89f81fff181293061504"}
Nov 02 15:17:25.979201 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:25.978984    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-6gnx4"
Nov 02 15:17:25.979201 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:25.979042    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-6gnx4" event={"ID":"2afb5fd0-d309-42eb-bd1e-9092ddeb166a","Type":"ContainerDied","Data":"71f8bf70c2f9b7bab586387920e077e2eff4cb4c1dc646be4792ed251c1aba2d"}
Nov 02 15:17:25.991170 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:25.991032    5304 scope.go:122] "RemoveContainer" containerID="db9ed33df2feb8a3dd2cdb179234648679154d230db955b2cc925161eb56b931"
Nov 02 15:17:25.991455 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:17:25.991417    5304 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db9ed33df2feb8a3dd2cdb179234648679154d230db955b2cc925161eb56b931\": container with ID starting with db9ed33df2feb8a3dd2cdb179234648679154d230db955b2cc925161eb56b931 not found: ID does not exist" containerID="db9ed33df2feb8a3dd2cdb179234648679154d230db955b2cc925161eb56b931"
Nov 02 15:17:25.991584 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:25.991466    5304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db9ed33df2feb8a3dd2cdb179234648679154d230db955b2cc925161eb56b931"} err="failed to get container status \"db9ed33df2feb8a3dd2cdb179234648679154d230db955b2cc925161eb56b931\": rpc error: code = NotFound desc = could not find container \"db9ed33df2feb8a3dd2cdb179234648679154d230db955b2cc925161eb56b931\": container with ID starting with db9ed33df2feb8a3dd2cdb179234648679154d230db955b2cc925161eb56b931 not found: ID does not exist"
Nov 02 15:17:25.991584 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:25.991490    5304 scope.go:122] "RemoveContainer" containerID="62ba7ae8f5ec5e03bc76221962b6d5947c0efeb6b89e89f81fff181293061504"
Nov 02 15:17:26.003624 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:26.003600    5304 scope.go:122] "RemoveContainer" containerID="62ba7ae8f5ec5e03bc76221962b6d5947c0efeb6b89e89f81fff181293061504"
Nov 02 15:17:26.003995 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:17:26.003964    5304 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"62ba7ae8f5ec5e03bc76221962b6d5947c0efeb6b89e89f81fff181293061504\": container with ID starting with 62ba7ae8f5ec5e03bc76221962b6d5947c0efeb6b89e89f81fff181293061504 not found: ID does not exist" containerID="62ba7ae8f5ec5e03bc76221962b6d5947c0efeb6b89e89f81fff181293061504"
Nov 02 15:17:26.004090 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:26.004005    5304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62ba7ae8f5ec5e03bc76221962b6d5947c0efeb6b89e89f81fff181293061504"} err="failed to get container status \"62ba7ae8f5ec5e03bc76221962b6d5947c0efeb6b89e89f81fff181293061504\": rpc error: code = NotFound desc = could not find container \"62ba7ae8f5ec5e03bc76221962b6d5947c0efeb6b89e89f81fff181293061504\": container with ID starting with 62ba7ae8f5ec5e03bc76221962b6d5947c0efeb6b89e89f81fff181293061504 not found: ID does not exist"
Nov 02 15:17:26.010563 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:26.010531    5304 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2afb5fd0-d309-42eb-bd1e-9092ddeb166a-istio-podinfo\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:17:26.010667 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:26.010562    5304 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2afb5fd0-d309-42eb-bd1e-9092ddeb166a-istio-token\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:17:26.010667 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:26.010581    5304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tqcjm\" (UniqueName: \"kubernetes.io/projected/2afb5fd0-d309-42eb-bd1e-9092ddeb166a-kube-api-access-tqcjm\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:17:26.010667 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:26.010602    5304 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2afb5fd0-d309-42eb-bd1e-9092ddeb166a-istio-envoy\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:17:26.106321 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:26.106286    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-7nj6n"]
Nov 02 15:17:26.113689 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:26.111871    5304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-7nj6n"]
Nov 02 15:17:26.128540 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:26.128505    5304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3ebb8197-273d-4e96-bec7-00c7a85a12a9" path="/var/lib/kubelet/pods/3ebb8197-273d-4e96-bec7-00c7a85a12a9/volumes"
Nov 02 15:17:26.145331 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:26.145298    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-6gnx4"]
Nov 02 15:17:26.151191 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:26.151156    5304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-6gnx4"]
Nov 02 15:17:26.163747 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:26.163676    5304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-w9hxk"
Nov 02 15:17:26.164231 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:26.164203    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-w9hxk"
Nov 02 15:17:26.164987 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:26.164964    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-w9hxk"
Nov 02 15:17:26.301981 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:26.301950    5304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-zzvrc"
Nov 02 15:17:26.302506 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:26.302476    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-zzvrc"
Nov 02 15:17:26.303730 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:26.303709    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-zzvrc"
Nov 02 15:17:28.120940 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:28.120910    5304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2afb5fd0-d309-42eb-bd1e-9092ddeb166a" path="/var/lib/kubelet/pods/2afb5fd0-d309-42eb-bd1e-9092ddeb166a/volumes"
Nov 02 15:17:36.234353 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:36.234312    5304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-0a7163-6c989cd9bd-xhgf5"]
Nov 02 15:17:36.235307 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:36.235283    5304 cpu_manager.go:414] "containerMap: removing container" podUID="2afb5fd0-d309-42eb-bd1e-9092ddeb166a" containerName="istio-proxy"
Nov 02 15:17:36.235429 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:36.235308    5304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2afb5fd0-d309-42eb-bd1e-9092ddeb166a" containerName="istio-proxy"
Nov 02 15:17:36.235429 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:36.235340    5304 cpu_manager.go:414] "containerMap: removing container" podUID="3ebb8197-273d-4e96-bec7-00c7a85a12a9" containerName="istio-proxy"
Nov 02 15:17:36.235429 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:36.235349    5304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ebb8197-273d-4e96-bec7-00c7a85a12a9" containerName="istio-proxy"
Nov 02 15:17:36.235601 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:36.235537    5304 memory_manager.go:365] "RemoveStaleState removing state" podUID="2afb5fd0-d309-42eb-bd1e-9092ddeb166a" containerName="istio-proxy"
Nov 02 15:17:36.235601 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:36.235554    5304 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ebb8197-273d-4e96-bec7-00c7a85a12a9" containerName="istio-proxy"
Nov 02 15:17:36.238866 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:36.238840    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-0a7163-6c989cd9bd-xhgf5"
Nov 02 15:17:36.273149 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:36.269484    5304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w5-0a7163-serving-cert\""
Nov 02 15:17:36.384150 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:36.359428    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-0a7163-6c989cd9bd-xhgf5"]
Nov 02 15:17:36.387967 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:36.386303    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mzd78\" (UniqueName: \"kubernetes.io/projected/34a8288b-adc6-424b-a30b-38f023185c86-kube-api-access-mzd78\") pod \"maas-api-e2e-shared-b-w5-0a7163-6c989cd9bd-xhgf5\" (UID: \"34a8288b-adc6-424b-a30b-38f023185c86\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-0a7163-6c989cd9bd-xhgf5"
Nov 02 15:17:36.387967 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:36.386404    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/34a8288b-adc6-424b-a30b-38f023185c86-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w5-0a7163-6c989cd9bd-xhgf5\" (UID: \"34a8288b-adc6-424b-a30b-38f023185c86\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-0a7163-6c989cd9bd-xhgf5"
Nov 02 15:17:36.387967 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:36.386617    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/34a8288b-adc6-424b-a30b-38f023185c86-maas-api-tls\") pod \"maas-api-e2e-shared-b-w5-0a7163-6c989cd9bd-xhgf5\" (UID: \"34a8288b-adc6-424b-a30b-38f023185c86\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-0a7163-6c989cd9bd-xhgf5"
Nov 02 15:17:36.387967 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:36.386698    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/34a8288b-adc6-424b-a30b-38f023185c86-metrics-certs\") pod \"maas-api-e2e-shared-b-w5-0a7163-6c989cd9bd-xhgf5\" (UID: \"34a8288b-adc6-424b-a30b-38f023185c86\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-0a7163-6c989cd9bd-xhgf5"
Nov 02 15:17:36.525611 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:36.514327    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mzd78\" (UniqueName: \"kubernetes.io/projected/34a8288b-adc6-424b-a30b-38f023185c86-kube-api-access-mzd78\") pod \"maas-api-e2e-shared-b-w5-0a7163-6c989cd9bd-xhgf5\" (UID: \"34a8288b-adc6-424b-a30b-38f023185c86\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-0a7163-6c989cd9bd-xhgf5"
Nov 02 15:17:36.525611 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:36.514422    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/34a8288b-adc6-424b-a30b-38f023185c86-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w5-0a7163-6c989cd9bd-xhgf5\" (UID: \"34a8288b-adc6-424b-a30b-38f023185c86\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-0a7163-6c989cd9bd-xhgf5"
Nov 02 15:17:36.525611 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:36.514590    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/34a8288b-adc6-424b-a30b-38f023185c86-maas-api-tls\") pod \"maas-api-e2e-shared-b-w5-0a7163-6c989cd9bd-xhgf5\" (UID: \"34a8288b-adc6-424b-a30b-38f023185c86\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-0a7163-6c989cd9bd-xhgf5"
Nov 02 15:17:36.525611 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:36.514659    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/34a8288b-adc6-424b-a30b-38f023185c86-metrics-certs\") pod \"maas-api-e2e-shared-b-w5-0a7163-6c989cd9bd-xhgf5\" (UID: \"34a8288b-adc6-424b-a30b-38f023185c86\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-0a7163-6c989cd9bd-xhgf5"
Nov 02 15:17:36.575149 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:17:36.572517    5304 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-0a7163-serving-cert: secret "maas-api-e2e-shared-b-w5-0a7163-serving-cert" not found
Nov 02 15:17:36.575149 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:17:36.572624    5304 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/34a8288b-adc6-424b-a30b-38f023185c86-maas-api-tls podName:34a8288b-adc6-424b-a30b-38f023185c86 nodeName:}" failed. No retries permitted until 2026-11-02 15:17:37.072599322 +0000 UTC m=+1257.066423729 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/34a8288b-adc6-424b-a30b-38f023185c86-maas-api-tls") pod "maas-api-e2e-shared-b-w5-0a7163-6c989cd9bd-xhgf5" (UID: "34a8288b-adc6-424b-a30b-38f023185c86") : secret "maas-api-e2e-shared-b-w5-0a7163-serving-cert" not found
Nov 02 15:17:36.606517 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:36.602015    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/34a8288b-adc6-424b-a30b-38f023185c86-metrics-certs\") pod \"maas-api-e2e-shared-b-w5-0a7163-6c989cd9bd-xhgf5\" (UID: \"34a8288b-adc6-424b-a30b-38f023185c86\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-0a7163-6c989cd9bd-xhgf5"
Nov 02 15:17:36.672292 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:36.637547    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/34a8288b-adc6-424b-a30b-38f023185c86-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w5-0a7163-6c989cd9bd-xhgf5\" (UID: \"34a8288b-adc6-424b-a30b-38f023185c86\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-0a7163-6c989cd9bd-xhgf5"
Nov 02 15:17:36.682260 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:36.680530    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mzd78\" (UniqueName: \"kubernetes.io/projected/34a8288b-adc6-424b-a30b-38f023185c86-kube-api-access-mzd78\") pod \"maas-api-e2e-shared-b-w5-0a7163-6c989cd9bd-xhgf5\" (UID: \"34a8288b-adc6-424b-a30b-38f023185c86\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-0a7163-6c989cd9bd-xhgf5"
Nov 02 15:17:37.169153 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:37.154327    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/34a8288b-adc6-424b-a30b-38f023185c86-maas-api-tls\") pod \"maas-api-e2e-shared-b-w5-0a7163-6c989cd9bd-xhgf5\" (UID: \"34a8288b-adc6-424b-a30b-38f023185c86\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-0a7163-6c989cd9bd-xhgf5"
Nov 02 15:17:37.211148 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:37.205089    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/34a8288b-adc6-424b-a30b-38f023185c86-maas-api-tls\") pod \"maas-api-e2e-shared-b-w5-0a7163-6c989cd9bd-xhgf5\" (UID: \"34a8288b-adc6-424b-a30b-38f023185c86\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-0a7163-6c989cd9bd-xhgf5"
Nov 02 15:17:37.391161 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:37.383250    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-739bf1/simulator-w2-739bf1-kserve-7c74f85465-ltk4t" podUID="cf6a1ded-28a3-49eb-878b-b4ad77a0bb5c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:17:37.480224 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:37.476633    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-0a7163-6c989cd9bd-xhgf5"
Nov 02 15:17:37.556165 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:37.553251    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7ecfe1/distinct-w4-7ecfe1-kserve-6c847c647d-d2jvv" podUID="f1f1ca5e-cca7-4c54-939f-06f095defcdd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:17:37.588161 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:37.573221    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-624254/simulator-w3-624254-kserve-56866b7699-z47sd" podUID="345c710f-b414-4e62-86c7-d51ff4e5be15" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:17:37.777985 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:37.777751    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nkbrk" podUID="1f68d837-fff2-418b-84d7-c2d7eb21a57e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.24:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:17:37.834406 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:37.833406    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1cc1b2/premium-w5-1cc1b2-kserve-58db46f97c-cgdqg" podUID="210f620a-8ae7-4e4c-acc3-46f75ddc6105" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:17:37.901325 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:37.901284    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1cc1b2/distinct-w5-1cc1b2-kserve-5b7459bc6d-zl8xr" podUID="211ad15d-9234-4f57-b1b9-181dd2d74208" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded"
Nov 02 15:17:37.926240 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:37.925220    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5dd8e9/premium-w1-5dd8e9-kserve-67588456fb-j6lph" podUID="4ccc607a-14e4-43ca-8999-78100d4b799f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:17:37.963202 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:37.956230    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5dd8e9/simulator-w1-5dd8e9-kserve-6f9f755d6f-gnw5d" podUID="66b75e64-cb24-40be-9076-4984c132e208" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:17:38.205613 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:38.203566    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5113c2/premium-w6-5113c2-kserve-b84945cb5-pc6f8" podUID="9fb3e992-7bb0-497a-a82c-6e396ba241c1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:17:38.255234 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:38.224700    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-624254/distinct-w3-624254-kserve-5465dc4765-dmq67" podUID="1a5aa910-a6de-417b-8638-81e4ec5c1825" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:17:38.264183 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:38.255780    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-739bf1/distinct-2-w2-739bf1-kserve-5c58479c48-qgfkh" podUID="9746bf8a-91bd-4715-a02e-ae2e651ca54d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:17:38.447202 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:38.309945    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6af50f/distinct-w0-6af50f-kserve-c6d994647-hbj9z" podUID="3dca5bf3-8e08-47a0-936e-205670d59ede" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:17:38.447202 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:38.385221    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-739bf1/simulator-w2-739bf1-kserve-7c74f85465-ltk4t" podUID="cf6a1ded-28a3-49eb-878b-b4ad77a0bb5c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:17:38.559151 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:38.558493    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7ecfe1/distinct-w4-7ecfe1-kserve-6c847c647d-d2jvv" podUID="f1f1ca5e-cca7-4c54-939f-06f095defcdd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:17:38.784605 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:38.784511    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-739bf1/simulator-w2-739bf1-kserve-7c74f85465-ltk4t"
Nov 02 15:17:38.953147 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:38.943506    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-7ecfe1/distinct-w4-7ecfe1-kserve-6c847c647d-d2jvv"
Nov 02 15:17:38.953147 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:38.843855    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1cc1b2/premium-w5-1cc1b2-kserve-58db46f97c-cgdqg" podUID="210f620a-8ae7-4e4c-acc3-46f75ddc6105" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded"
Nov 02 15:17:38.953147 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:38.882441    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-624254/simulator-w3-624254-kserve-56866b7699-z47sd" podUID="345c710f-b414-4e62-86c7-d51ff4e5be15" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": context deadline exceeded"
Nov 02 15:17:38.953147 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:38.889359    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nkbrk" podUID="1f68d837-fff2-418b-84d7-c2d7eb21a57e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.24:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:17:38.953147 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:38.949000    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-1cc1b2/premium-w5-1cc1b2-kserve-58db46f97c-cgdqg"
Nov 02 15:17:38.953147 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:38.949078    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-624254/simulator-w3-624254-kserve-56866b7699-z47sd"
Nov 02 15:17:38.953147 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:38.949556    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nkbrk"
Nov 02 15:17:38.984151 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:38.976570    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5dd8e9/simulator-w1-5dd8e9-kserve-6f9f755d6f-gnw5d" podUID="66b75e64-cb24-40be-9076-4984c132e208" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": context deadline exceeded"
Nov 02 15:17:39.124302 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:38.994483    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5dd8e9/simulator-w1-5dd8e9-kserve-6f9f755d6f-gnw5d"
Nov 02 15:17:39.124302 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:39.051424    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1cc1b2/distinct-w5-1cc1b2-kserve-5b7459bc6d-zl8xr" podUID="211ad15d-9234-4f57-b1b9-181dd2d74208" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded"
Nov 02 15:17:39.124302 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:39.080387    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5dd8e9/premium-w1-5dd8e9-kserve-67588456fb-j6lph" podUID="4ccc607a-14e4-43ca-8999-78100d4b799f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded"
Nov 02 15:17:39.124302 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:39.082384    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-1cc1b2/distinct-w5-1cc1b2-kserve-5b7459bc6d-zl8xr"
Nov 02 15:17:39.124302 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:39.082456    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5dd8e9/premium-w1-5dd8e9-kserve-67588456fb-j6lph"
Nov 02 15:17:39.254910 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:39.254864    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-624254/distinct-w3-624254-kserve-5465dc4765-dmq67" podUID="1a5aa910-a6de-417b-8638-81e4ec5c1825" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:17:39.268769 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:39.268657    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-624254/distinct-w3-624254-kserve-5465dc4765-dmq67"
Nov 02 15:17:39.319149 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:39.317279    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-739bf1/distinct-2-w2-739bf1-kserve-5c58479c48-qgfkh" podUID="9746bf8a-91bd-4715-a02e-ae2e651ca54d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:17:39.319149 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:39.317739    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-739bf1/distinct-2-w2-739bf1-kserve-5c58479c48-qgfkh"
Nov 02 15:17:39.374155 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:39.365442    5304 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-5dd8e9/simulator-w1-5dd8e9-kserve-6f9f755d6f-gnw5d" podUID="66b75e64-cb24-40be-9076-4984c132e208" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/health\": context deadline exceeded"
Nov 02 15:17:39.390207 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:39.384217    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-0a7163-6c989cd9bd-xhgf5"]
Nov 02 15:17:39.390207 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:39.384358    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-624254/distinct-w3-624254-kserve-5465dc4765-dmq67"
Nov 02 15:17:39.419171 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:39.410277    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5dd8e9/premium-w1-5dd8e9-kserve-67588456fb-j6lph"
Nov 02 15:17:39.419171 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:39.410362    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-1cc1b2/distinct-w5-1cc1b2-kserve-5b7459bc6d-zl8xr"
Nov 02 15:17:39.421979 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:39.421928    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-739bf1/simulator-w2-739bf1-kserve-7c74f85465-ltk4t" podUID="cf6a1ded-28a3-49eb-878b-b4ad77a0bb5c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:17:39.517302 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:39.516939    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5dd8e9/simulator-w1-5dd8e9-kserve-6f9f755d6f-gnw5d"
Nov 02 15:17:39.517951 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:39.517933    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-739bf1/distinct-2-w2-739bf1-kserve-5c58479c48-qgfkh"
Nov 02 15:17:39.574090 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:39.559423    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-624254/simulator-w3-624254-kserve-56866b7699-z47sd"
Nov 02 15:17:39.574090 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:39.566402    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nkbrk"
Nov 02 15:17:39.618199 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:39.598098    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-1cc1b2/premium-w5-1cc1b2-kserve-58db46f97c-cgdqg"
Nov 02 15:17:39.622706 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:39.621214    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7ecfe1/distinct-w4-7ecfe1-kserve-6c847c647d-d2jvv" podUID="f1f1ca5e-cca7-4c54-939f-06f095defcdd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:17:39.682872 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:39.682420    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-7ecfe1/distinct-w4-7ecfe1-kserve-6c847c647d-d2jvv"
Nov 02 15:17:39.683677 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:39.683480    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-739bf1/simulator-w2-739bf1-kserve-7c74f85465-ltk4t"
Nov 02 15:17:40.678147 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:40.677890    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-0a7163-6c989cd9bd-xhgf5" event={"ID":"34a8288b-adc6-424b-a30b-38f023185c86","Type":"ContainerStarted","Data":"d9acb6dbd5f54b30c15ce3b101055c388a0b8a35443bf5ac415ca57978ae962b"}
Nov 02 15:17:40.678147 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:40.677951    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-0a7163-6c989cd9bd-xhgf5" event={"ID":"34a8288b-adc6-424b-a30b-38f023185c86","Type":"ContainerStarted","Data":"21b8135b8f2ca6ab05e72ed55b4420802c7b68fc52763309066acff413109c35"}
Nov 02 15:17:40.735961 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:40.735934    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-64phg"]
Nov 02 15:17:40.736257 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:40.736239    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-0a7163-6c989cd9bd-xhgf5"
Nov 02 15:17:40.736594 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:40.736568    5304 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-64phg" podUID="e60cf723-3d4a-4bcc-bc72-0c984eba466b" containerName="istio-proxy" containerID="cri-o://06e5c446c033df27152ed7a158a49a0d299c958e4afe3e00a3cb047bfdf5315d" gracePeriod=30
Nov 02 15:17:40.797345 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:40.793945    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-qqqdn"]
Nov 02 15:17:40.797345 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:40.794707    5304 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-qqqdn" podUID="59fee359-49bd-4a5f-b5da-e9e22c9328f1" containerName="istio-proxy" containerID="cri-o://564c908aa5ce1b803c9b8f326631537260db9011013e1d0d4f96662f86df5638" gracePeriod=30
Nov 02 15:17:40.816974 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:40.812267    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-k9kqx"]
Nov 02 15:17:40.816974 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:40.812536    5304 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-k9kqx" podUID="7f0b208d-d18a-4a38-952c-4a1abb1b30a0" containerName="istio-proxy" containerID="cri-o://df5313aa08c19e33bc82d9ee5d39df53edec151bc8e4706fed0c3eca101ff271" gracePeriod=30
Nov 02 15:17:40.816974 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:40.814136    5304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-0a7163-6c989cd9bd-xhgf5" podStartSLOduration=4.215686658 podStartE2EDuration="4.814093525s" totalImagesPullingTime="598.406867ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:17:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:17:40.742970826 +0000 UTC m=+1260.736795237" watchObservedRunningTime="2026-11-02 15:17:40.814093525 +0000 UTC m=+1260.807917935"
Nov 02 15:17:46.189963 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:46.189915    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-64phg"
Nov 02 15:17:46.226676 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:46.226635    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-qqqdn"
Nov 02 15:17:46.265986 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:46.265954    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-k9kqx"
Nov 02 15:17:46.340565 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:46.340528    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f0b208d-d18a-4a38-952c-4a1abb1b30a0-istio-token\" (UniqueName: \"kubernetes.io/projected/7f0b208d-d18a-4a38-952c-4a1abb1b30a0-istio-token\") pod \"7f0b208d-d18a-4a38-952c-4a1abb1b30a0\" (UID: \"7f0b208d-d18a-4a38-952c-4a1abb1b30a0\") "
Nov 02 15:17:46.340738 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:46.340581    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e60cf723-3d4a-4bcc-bc72-0c984eba466b-istio-token\" (UniqueName: \"kubernetes.io/projected/e60cf723-3d4a-4bcc-bc72-0c984eba466b-istio-token\") pod \"e60cf723-3d4a-4bcc-bc72-0c984eba466b\" (UID: \"e60cf723-3d4a-4bcc-bc72-0c984eba466b\") "
Nov 02 15:17:46.340738 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:46.340611    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/59fee359-49bd-4a5f-b5da-e9e22c9328f1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59fee359-49bd-4a5f-b5da-e9e22c9328f1-istiod-ca-cert\") pod \"59fee359-49bd-4a5f-b5da-e9e22c9328f1\" (UID: \"59fee359-49bd-4a5f-b5da-e9e22c9328f1\") "
Nov 02 15:17:46.340738 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:46.340649    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59fee359-49bd-4a5f-b5da-e9e22c9328f1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59fee359-49bd-4a5f-b5da-e9e22c9328f1-istio-data\") pod \"59fee359-49bd-4a5f-b5da-e9e22c9328f1\" (UID: \"59fee359-49bd-4a5f-b5da-e9e22c9328f1\") "
Nov 02 15:17:46.340738 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:46.340687    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7f0b208d-d18a-4a38-952c-4a1abb1b30a0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f0b208d-d18a-4a38-952c-4a1abb1b30a0-istio-podinfo\") pod \"7f0b208d-d18a-4a38-952c-4a1abb1b30a0\" (UID: \"7f0b208d-d18a-4a38-952c-4a1abb1b30a0\") "
Nov 02 15:17:46.340900 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:46.340743    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f0b208d-d18a-4a38-952c-4a1abb1b30a0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f0b208d-d18a-4a38-952c-4a1abb1b30a0-workload-socket\") pod \"7f0b208d-d18a-4a38-952c-4a1abb1b30a0\" (UID: \"7f0b208d-d18a-4a38-952c-4a1abb1b30a0\") "
Nov 02 15:17:46.340900 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:46.340817    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f0b208d-d18a-4a38-952c-4a1abb1b30a0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f0b208d-d18a-4a38-952c-4a1abb1b30a0-istio-data\") pod \"7f0b208d-d18a-4a38-952c-4a1abb1b30a0\" (UID: \"7f0b208d-d18a-4a38-952c-4a1abb1b30a0\") "
Nov 02 15:17:46.340900 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:46.340851    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59fee359-49bd-4a5f-b5da-e9e22c9328f1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59fee359-49bd-4a5f-b5da-e9e22c9328f1-credential-socket\") pod \"59fee359-49bd-4a5f-b5da-e9e22c9328f1\" (UID: \"59fee359-49bd-4a5f-b5da-e9e22c9328f1\") "
Nov 02 15:17:46.340900 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:46.340891    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e60cf723-3d4a-4bcc-bc72-0c984eba466b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e60cf723-3d4a-4bcc-bc72-0c984eba466b-workload-socket\") pod \"e60cf723-3d4a-4bcc-bc72-0c984eba466b\" (UID: \"e60cf723-3d4a-4bcc-bc72-0c984eba466b\") "
Nov 02 15:17:46.341078 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:46.340926    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/59fee359-49bd-4a5f-b5da-e9e22c9328f1-kube-api-access-9xr5q\" (UniqueName: \"kubernetes.io/projected/59fee359-49bd-4a5f-b5da-e9e22c9328f1-kube-api-access-9xr5q\") pod \"59fee359-49bd-4a5f-b5da-e9e22c9328f1\" (UID: \"59fee359-49bd-4a5f-b5da-e9e22c9328f1\") "
Nov 02 15:17:46.341078 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:46.340964    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/59fee359-49bd-4a5f-b5da-e9e22c9328f1-istio-token\" (UniqueName: \"kubernetes.io/projected/59fee359-49bd-4a5f-b5da-e9e22c9328f1-istio-token\") pod \"59fee359-49bd-4a5f-b5da-e9e22c9328f1\" (UID: \"59fee359-49bd-4a5f-b5da-e9e22c9328f1\") "
Nov 02 15:17:46.341078 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:46.341014    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e60cf723-3d4a-4bcc-bc72-0c984eba466b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e60cf723-3d4a-4bcc-bc72-0c984eba466b-istiod-ca-cert\") pod \"e60cf723-3d4a-4bcc-bc72-0c984eba466b\" (UID: \"e60cf723-3d4a-4bcc-bc72-0c984eba466b\") "
Nov 02 15:17:46.341078 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:46.341024    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59fee359-49bd-4a5f-b5da-e9e22c9328f1-istiod-ca-cert" pod "59fee359-49bd-4a5f-b5da-e9e22c9328f1" (UID: "59fee359-49bd-4a5f-b5da-e9e22c9328f1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:17:46.341078 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:46.341066    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f0b208d-d18a-4a38-952c-4a1abb1b30a0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f0b208d-d18a-4a38-952c-4a1abb1b30a0-workload-certs\") pod \"7f0b208d-d18a-4a38-952c-4a1abb1b30a0\" (UID: \"7f0b208d-d18a-4a38-952c-4a1abb1b30a0\") "
Nov 02 15:17:46.341404 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:46.341113    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f0b208d-d18a-4a38-952c-4a1abb1b30a0-kube-api-access-s8w4k\" (UniqueName: \"kubernetes.io/projected/7f0b208d-d18a-4a38-952c-4a1abb1b30a0-kube-api-access-s8w4k\") pod \"7f0b208d-d18a-4a38-952c-4a1abb1b30a0\" (UID: \"7f0b208d-d18a-4a38-952c-4a1abb1b30a0\") "
Nov 02 15:17:46.341404 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:46.341211    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e60cf723-3d4a-4bcc-bc72-0c984eba466b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e60cf723-3d4a-4bcc-bc72-0c984eba466b-credential-socket\") pod \"e60cf723-3d4a-4bcc-bc72-0c984eba466b\" (UID: \"e60cf723-3d4a-4bcc-bc72-0c984eba466b\") "
Nov 02 15:17:46.341404 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:46.341262    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e60cf723-3d4a-4bcc-bc72-0c984eba466b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e60cf723-3d4a-4bcc-bc72-0c984eba466b-istio-data\") pod \"e60cf723-3d4a-4bcc-bc72-0c984eba466b\" (UID: \"e60cf723-3d4a-4bcc-bc72-0c984eba466b\") "
Nov 02 15:17:46.341404 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:46.341324    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59fee359-49bd-4a5f-b5da-e9e22c9328f1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59fee359-49bd-4a5f-b5da-e9e22c9328f1-workload-certs\") pod \"59fee359-49bd-4a5f-b5da-e9e22c9328f1\" (UID: \"59fee359-49bd-4a5f-b5da-e9e22c9328f1\") "
Nov 02 15:17:46.341404 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:46.341350    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59fee359-49bd-4a5f-b5da-e9e22c9328f1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59fee359-49bd-4a5f-b5da-e9e22c9328f1-workload-socket\") pod \"59fee359-49bd-4a5f-b5da-e9e22c9328f1\" (UID: \"59fee359-49bd-4a5f-b5da-e9e22c9328f1\") "
Nov 02 15:17:46.341404 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:46.341404    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e60cf723-3d4a-4bcc-bc72-0c984eba466b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e60cf723-3d4a-4bcc-bc72-0c984eba466b-workload-certs\") pod \"e60cf723-3d4a-4bcc-bc72-0c984eba466b\" (UID: \"e60cf723-3d4a-4bcc-bc72-0c984eba466b\") "
Nov 02 15:17:46.341714 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:46.341444    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f0b208d-d18a-4a38-952c-4a1abb1b30a0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f0b208d-d18a-4a38-952c-4a1abb1b30a0-credential-socket\") pod \"7f0b208d-d18a-4a38-952c-4a1abb1b30a0\" (UID: \"7f0b208d-d18a-4a38-952c-4a1abb1b30a0\") "
Nov 02 15:17:46.341714 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:46.341448    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59fee359-49bd-4a5f-b5da-e9e22c9328f1-istio-data" pod "59fee359-49bd-4a5f-b5da-e9e22c9328f1" (UID: "59fee359-49bd-4a5f-b5da-e9e22c9328f1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:17:46.341714 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:46.341435    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f0b208d-d18a-4a38-952c-4a1abb1b30a0-workload-socket" pod "7f0b208d-d18a-4a38-952c-4a1abb1b30a0" (UID: "7f0b208d-d18a-4a38-952c-4a1abb1b30a0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:17:46.341714 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:46.341511    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7f0b208d-d18a-4a38-952c-4a1abb1b30a0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f0b208d-d18a-4a38-952c-4a1abb1b30a0-istiod-ca-cert\") pod \"7f0b208d-d18a-4a38-952c-4a1abb1b30a0\" (UID: \"7f0b208d-d18a-4a38-952c-4a1abb1b30a0\") "
Nov 02 15:17:46.341714 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:46.341556    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e60cf723-3d4a-4bcc-bc72-0c984eba466b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e60cf723-3d4a-4bcc-bc72-0c984eba466b-istio-podinfo\") pod \"e60cf723-3d4a-4bcc-bc72-0c984eba466b\" (UID: \"e60cf723-3d4a-4bcc-bc72-0c984eba466b\") "
Nov 02 15:17:46.341714 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:46.341639    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/59fee359-49bd-4a5f-b5da-e9e22c9328f1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59fee359-49bd-4a5f-b5da-e9e22c9328f1-istio-podinfo\") pod \"59fee359-49bd-4a5f-b5da-e9e22c9328f1\" (UID: \"59fee359-49bd-4a5f-b5da-e9e22c9328f1\") "
Nov 02 15:17:46.341714 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:46.341665    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59fee359-49bd-4a5f-b5da-e9e22c9328f1-credential-socket" pod "59fee359-49bd-4a5f-b5da-e9e22c9328f1" (UID: "59fee359-49bd-4a5f-b5da-e9e22c9328f1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:17:46.341714 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:46.341708    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e60cf723-3d4a-4bcc-bc72-0c984eba466b-kube-api-access-4wbfr\" (UniqueName: \"kubernetes.io/projected/e60cf723-3d4a-4bcc-bc72-0c984eba466b-kube-api-access-4wbfr\") pod \"e60cf723-3d4a-4bcc-bc72-0c984eba466b\" (UID: \"e60cf723-3d4a-4bcc-bc72-0c984eba466b\") "
Nov 02 15:17:46.342143 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:46.341730    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e60cf723-3d4a-4bcc-bc72-0c984eba466b-workload-socket" pod "e60cf723-3d4a-4bcc-bc72-0c984eba466b" (UID: "e60cf723-3d4a-4bcc-bc72-0c984eba466b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:17:46.342143 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:46.341748    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59fee359-49bd-4a5f-b5da-e9e22c9328f1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59fee359-49bd-4a5f-b5da-e9e22c9328f1-istio-envoy\") pod \"59fee359-49bd-4a5f-b5da-e9e22c9328f1\" (UID: \"59fee359-49bd-4a5f-b5da-e9e22c9328f1\") "
Nov 02 15:17:46.342143 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:46.341828    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e60cf723-3d4a-4bcc-bc72-0c984eba466b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e60cf723-3d4a-4bcc-bc72-0c984eba466b-istio-envoy\") pod \"e60cf723-3d4a-4bcc-bc72-0c984eba466b\" (UID: \"e60cf723-3d4a-4bcc-bc72-0c984eba466b\") "
Nov 02 15:17:46.342143 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:46.341860    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f0b208d-d18a-4a38-952c-4a1abb1b30a0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f0b208d-d18a-4a38-952c-4a1abb1b30a0-istio-envoy\") pod \"7f0b208d-d18a-4a38-952c-4a1abb1b30a0\" (UID: \"7f0b208d-d18a-4a38-952c-4a1abb1b30a0\") "
Nov 02 15:17:46.342143 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:46.341875    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59fee359-49bd-4a5f-b5da-e9e22c9328f1-workload-certs" pod "59fee359-49bd-4a5f-b5da-e9e22c9328f1" (UID: "59fee359-49bd-4a5f-b5da-e9e22c9328f1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:17:46.342143 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:46.342038    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f0b208d-d18a-4a38-952c-4a1abb1b30a0-istiod-ca-cert" pod "7f0b208d-d18a-4a38-952c-4a1abb1b30a0" (UID: "7f0b208d-d18a-4a38-952c-4a1abb1b30a0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:17:46.342471 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:46.342296    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59fee359-49bd-4a5f-b5da-e9e22c9328f1-workload-socket" pod "59fee359-49bd-4a5f-b5da-e9e22c9328f1" (UID: "59fee359-49bd-4a5f-b5da-e9e22c9328f1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:17:46.342526 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:46.342474    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e60cf723-3d4a-4bcc-bc72-0c984eba466b-workload-certs" pod "e60cf723-3d4a-4bcc-bc72-0c984eba466b" (UID: "e60cf723-3d4a-4bcc-bc72-0c984eba466b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:17:46.346053 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:46.342788    5304 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59fee359-49bd-4a5f-b5da-e9e22c9328f1-istiod-ca-cert\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:17:46.346053 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:46.342816    5304 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59fee359-49bd-4a5f-b5da-e9e22c9328f1-istio-data\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:17:46.346053 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:46.342851    5304 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f0b208d-d18a-4a38-952c-4a1abb1b30a0-workload-socket\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:17:46.346053 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:46.342871    5304 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59fee359-49bd-4a5f-b5da-e9e22c9328f1-credential-socket\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:17:46.346053 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:46.342885    5304 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e60cf723-3d4a-4bcc-bc72-0c984eba466b-workload-socket\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:17:46.346053 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:46.342900    5304 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59fee359-49bd-4a5f-b5da-e9e22c9328f1-workload-certs\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:17:46.346053 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:46.342914    5304 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59fee359-49bd-4a5f-b5da-e9e22c9328f1-workload-socket\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:17:46.346053 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:46.342926    5304 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e60cf723-3d4a-4bcc-bc72-0c984eba466b-workload-certs\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:17:46.346053 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:46.342938    5304 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f0b208d-d18a-4a38-952c-4a1abb1b30a0-istiod-ca-cert\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:17:46.346053 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:46.343554    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e60cf723-3d4a-4bcc-bc72-0c984eba466b-istio-token" pod "e60cf723-3d4a-4bcc-bc72-0c984eba466b" (UID: "e60cf723-3d4a-4bcc-bc72-0c984eba466b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:17:46.346053 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:46.343865    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f0b208d-d18a-4a38-952c-4a1abb1b30a0-credential-socket" pod "7f0b208d-d18a-4a38-952c-4a1abb1b30a0" (UID: "7f0b208d-d18a-4a38-952c-4a1abb1b30a0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:17:46.346053 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:46.344936    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59fee359-49bd-4a5f-b5da-e9e22c9328f1-istio-envoy" pod "59fee359-49bd-4a5f-b5da-e9e22c9328f1" (UID: "59fee359-49bd-4a5f-b5da-e9e22c9328f1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:17:46.346053 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:46.345047    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f0b208d-d18a-4a38-952c-4a1abb1b30a0-istio-token" pod "7f0b208d-d18a-4a38-952c-4a1abb1b30a0" (UID: "7f0b208d-d18a-4a38-952c-4a1abb1b30a0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:17:46.346053 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:46.345284    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f0b208d-d18a-4a38-952c-4a1abb1b30a0-workload-certs" pod "7f0b208d-d18a-4a38-952c-4a1abb1b30a0" (UID: "7f0b208d-d18a-4a38-952c-4a1abb1b30a0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:17:46.346053 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:46.345296    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e60cf723-3d4a-4bcc-bc72-0c984eba466b-istio-data" pod "e60cf723-3d4a-4bcc-bc72-0c984eba466b" (UID: "e60cf723-3d4a-4bcc-bc72-0c984eba466b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:17:46.346053 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:46.345377    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e60cf723-3d4a-4bcc-bc72-0c984eba466b-istiod-ca-cert" pod "e60cf723-3d4a-4bcc-bc72-0c984eba466b" (UID: "e60cf723-3d4a-4bcc-bc72-0c984eba466b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:17:46.346053 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:46.346007    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e60cf723-3d4a-4bcc-bc72-0c984eba466b-credential-socket" pod "e60cf723-3d4a-4bcc-bc72-0c984eba466b" (UID: "e60cf723-3d4a-4bcc-bc72-0c984eba466b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:17:46.352780 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:46.346165    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f0b208d-d18a-4a38-952c-4a1abb1b30a0-kube-api-access-s8w4k" pod "7f0b208d-d18a-4a38-952c-4a1abb1b30a0" (UID: "7f0b208d-d18a-4a38-952c-4a1abb1b30a0"). InnerVolumeSpecName "kube-api-access-s8w4k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:17:46.352780 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:46.347314    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e60cf723-3d4a-4bcc-bc72-0c984eba466b-kube-api-access-4wbfr" pod "e60cf723-3d4a-4bcc-bc72-0c984eba466b" (UID: "e60cf723-3d4a-4bcc-bc72-0c984eba466b"). InnerVolumeSpecName "kube-api-access-4wbfr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:17:46.352780 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:46.348626    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f0b208d-d18a-4a38-952c-4a1abb1b30a0-istio-data" pod "7f0b208d-d18a-4a38-952c-4a1abb1b30a0" (UID: "7f0b208d-d18a-4a38-952c-4a1abb1b30a0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:17:46.352780 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:46.348653    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f0b208d-d18a-4a38-952c-4a1abb1b30a0-istio-envoy" pod "7f0b208d-d18a-4a38-952c-4a1abb1b30a0" (UID: "7f0b208d-d18a-4a38-952c-4a1abb1b30a0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:17:46.352780 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:46.350899    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7f0b208d-d18a-4a38-952c-4a1abb1b30a0-istio-podinfo" pod "7f0b208d-d18a-4a38-952c-4a1abb1b30a0" (UID: "7f0b208d-d18a-4a38-952c-4a1abb1b30a0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 15:17:46.353491 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:46.352888    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/59fee359-49bd-4a5f-b5da-e9e22c9328f1-istio-podinfo" pod "59fee359-49bd-4a5f-b5da-e9e22c9328f1" (UID: "59fee359-49bd-4a5f-b5da-e9e22c9328f1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 15:17:46.353491 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:46.353030    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e60cf723-3d4a-4bcc-bc72-0c984eba466b-istio-podinfo" pod "e60cf723-3d4a-4bcc-bc72-0c984eba466b" (UID: "e60cf723-3d4a-4bcc-bc72-0c984eba466b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 15:17:46.353491 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:46.353149    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59fee359-49bd-4a5f-b5da-e9e22c9328f1-kube-api-access-9xr5q" pod "59fee359-49bd-4a5f-b5da-e9e22c9328f1" (UID: "59fee359-49bd-4a5f-b5da-e9e22c9328f1"). InnerVolumeSpecName "kube-api-access-9xr5q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:17:46.353491 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:46.353203    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59fee359-49bd-4a5f-b5da-e9e22c9328f1-istio-token" pod "59fee359-49bd-4a5f-b5da-e9e22c9328f1" (UID: "59fee359-49bd-4a5f-b5da-e9e22c9328f1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:17:46.354831 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:46.354789    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e60cf723-3d4a-4bcc-bc72-0c984eba466b-istio-envoy" pod "e60cf723-3d4a-4bcc-bc72-0c984eba466b" (UID: "e60cf723-3d4a-4bcc-bc72-0c984eba466b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:17:46.444905 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:46.444871    5304 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59fee359-49bd-4a5f-b5da-e9e22c9328f1-istio-podinfo\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:17:46.444905 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:46.444900    5304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4wbfr\" (UniqueName: \"kubernetes.io/projected/e60cf723-3d4a-4bcc-bc72-0c984eba466b-kube-api-access-4wbfr\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:17:46.444905 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:46.444912    5304 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59fee359-49bd-4a5f-b5da-e9e22c9328f1-istio-envoy\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:17:46.445146 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:46.444920    5304 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e60cf723-3d4a-4bcc-bc72-0c984eba466b-istio-envoy\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:17:46.445146 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:46.444928    5304 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f0b208d-d18a-4a38-952c-4a1abb1b30a0-istio-envoy\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:17:46.445146 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:46.444936    5304 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f0b208d-d18a-4a38-952c-4a1abb1b30a0-istio-token\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:17:46.445146 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:46.444944    5304 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e60cf723-3d4a-4bcc-bc72-0c984eba466b-istio-token\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:17:46.445146 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:46.444952    5304 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f0b208d-d18a-4a38-952c-4a1abb1b30a0-istio-podinfo\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:17:46.445146 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:46.444959    5304 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f0b208d-d18a-4a38-952c-4a1abb1b30a0-istio-data\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:17:46.445146 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:46.444974    5304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9xr5q\" (UniqueName: \"kubernetes.io/projected/59fee359-49bd-4a5f-b5da-e9e22c9328f1-kube-api-access-9xr5q\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:17:46.445146 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:46.444982    5304 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59fee359-49bd-4a5f-b5da-e9e22c9328f1-istio-token\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:17:46.445146 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:46.444991    5304 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e60cf723-3d4a-4bcc-bc72-0c984eba466b-istiod-ca-cert\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:17:46.445146 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:46.444998    5304 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f0b208d-d18a-4a38-952c-4a1abb1b30a0-workload-certs\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:17:46.445146 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:46.445007    5304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s8w4k\" (UniqueName: \"kubernetes.io/projected/7f0b208d-d18a-4a38-952c-4a1abb1b30a0-kube-api-access-s8w4k\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:17:46.445146 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:46.445014    5304 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e60cf723-3d4a-4bcc-bc72-0c984eba466b-credential-socket\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:17:46.445146 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:46.445022    5304 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e60cf723-3d4a-4bcc-bc72-0c984eba466b-istio-data\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:17:46.445146 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:46.445031    5304 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f0b208d-d18a-4a38-952c-4a1abb1b30a0-credential-socket\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:17:46.445146 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:46.445039    5304 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e60cf723-3d4a-4bcc-bc72-0c984eba466b-istio-podinfo\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:17:46.625326 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:46.625240    5304 patch_prober.go:29] interesting pod/e2e-api-a-ad276f-openshift-default-7cb6d54d77-rps78 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 15:17:46.625516 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:46.625324    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-ad276f-openshift-default-7cb6d54d77-rps78" podUID="350b279b-88a8-4318-b49c-d5001077f702" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:17:46.726211 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:46.726171    5304 generic.go:356] "Generic (PLEG): container finished" podID="e60cf723-3d4a-4bcc-bc72-0c984eba466b" containerID="06e5c446c033df27152ed7a158a49a0d299c958e4afe3e00a3cb047bfdf5315d" exitCode=0
Nov 02 15:17:46.726594 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:46.726219    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-64phg"
Nov 02 15:17:46.726594 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:46.726312    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-64phg" event={"ID":"e60cf723-3d4a-4bcc-bc72-0c984eba466b","Type":"ContainerDied","Data":"06e5c446c033df27152ed7a158a49a0d299c958e4afe3e00a3cb047bfdf5315d"}
Nov 02 15:17:46.726594 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:46.726368    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-64phg" event={"ID":"e60cf723-3d4a-4bcc-bc72-0c984eba466b","Type":"ContainerDied","Data":"5c01915173619b9f33e66de69a407baae4f0718870e919607a2c261d05f71137"}
Nov 02 15:17:46.726594 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:46.726400    5304 scope.go:122] "RemoveContainer" containerID="06e5c446c033df27152ed7a158a49a0d299c958e4afe3e00a3cb047bfdf5315d"
Nov 02 15:17:46.730214 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:46.729364    5304 generic.go:356] "Generic (PLEG): container finished" podID="7f0b208d-d18a-4a38-952c-4a1abb1b30a0" containerID="df5313aa08c19e33bc82d9ee5d39df53edec151bc8e4706fed0c3eca101ff271" exitCode=0
Nov 02 15:17:46.730214 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:46.729453    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-k9kqx" event={"ID":"7f0b208d-d18a-4a38-952c-4a1abb1b30a0","Type":"ContainerDied","Data":"df5313aa08c19e33bc82d9ee5d39df53edec151bc8e4706fed0c3eca101ff271"}
Nov 02 15:17:46.730214 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:46.729495    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-k9kqx" event={"ID":"7f0b208d-d18a-4a38-952c-4a1abb1b30a0","Type":"ContainerDied","Data":"a91340c00d2fc77aa513da268cf7cd5a98f0c6c8067da974ee62797409c759db"}
Nov 02 15:17:46.730214 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:46.729573    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-k9kqx"
Nov 02 15:17:46.731663 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:46.731638    5304 generic.go:356] "Generic (PLEG): container finished" podID="59fee359-49bd-4a5f-b5da-e9e22c9328f1" containerID="564c908aa5ce1b803c9b8f326631537260db9011013e1d0d4f96662f86df5638" exitCode=0
Nov 02 15:17:46.733612 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:46.733238    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-qqqdn" event={"ID":"59fee359-49bd-4a5f-b5da-e9e22c9328f1","Type":"ContainerDied","Data":"564c908aa5ce1b803c9b8f326631537260db9011013e1d0d4f96662f86df5638"}
Nov 02 15:17:46.733918 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:46.733882    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-qqqdn" event={"ID":"59fee359-49bd-4a5f-b5da-e9e22c9328f1","Type":"ContainerDied","Data":"32184248414a7e61c1bffda7ecc38e9e69aac76e3d6bc80d6c7570ea0f8dae8b"}
Nov 02 15:17:46.734040 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:46.733268    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-qqqdn"
Nov 02 15:17:46.764015 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:46.763991    5304 scope.go:122] "RemoveContainer" containerID="06e5c446c033df27152ed7a158a49a0d299c958e4afe3e00a3cb047bfdf5315d"
Nov 02 15:17:46.764449 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:17:46.764426    5304 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"06e5c446c033df27152ed7a158a49a0d299c958e4afe3e00a3cb047bfdf5315d\": container with ID starting with 06e5c446c033df27152ed7a158a49a0d299c958e4afe3e00a3cb047bfdf5315d not found: ID does not exist" containerID="06e5c446c033df27152ed7a158a49a0d299c958e4afe3e00a3cb047bfdf5315d"
Nov 02 15:17:46.764610 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:46.764465    5304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"06e5c446c033df27152ed7a158a49a0d299c958e4afe3e00a3cb047bfdf5315d"} err="failed to get container status \"06e5c446c033df27152ed7a158a49a0d299c958e4afe3e00a3cb047bfdf5315d\": rpc error: code = NotFound desc = could not find container \"06e5c446c033df27152ed7a158a49a0d299c958e4afe3e00a3cb047bfdf5315d\": container with ID starting with 06e5c446c033df27152ed7a158a49a0d299c958e4afe3e00a3cb047bfdf5315d not found: ID does not exist"
Nov 02 15:17:46.764610 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:46.764492    5304 scope.go:122] "RemoveContainer" containerID="df5313aa08c19e33bc82d9ee5d39df53edec151bc8e4706fed0c3eca101ff271"
Nov 02 15:17:46.778840 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:46.778817    5304 scope.go:122] "RemoveContainer" containerID="df5313aa08c19e33bc82d9ee5d39df53edec151bc8e4706fed0c3eca101ff271"
Nov 02 15:17:46.779361 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:17:46.779338    5304 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df5313aa08c19e33bc82d9ee5d39df53edec151bc8e4706fed0c3eca101ff271\": container with ID starting with df5313aa08c19e33bc82d9ee5d39df53edec151bc8e4706fed0c3eca101ff271 not found: ID does not exist" containerID="df5313aa08c19e33bc82d9ee5d39df53edec151bc8e4706fed0c3eca101ff271"
Nov 02 15:17:46.779449 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:46.779376    5304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df5313aa08c19e33bc82d9ee5d39df53edec151bc8e4706fed0c3eca101ff271"} err="failed to get container status \"df5313aa08c19e33bc82d9ee5d39df53edec151bc8e4706fed0c3eca101ff271\": rpc error: code = NotFound desc = could not find container \"df5313aa08c19e33bc82d9ee5d39df53edec151bc8e4706fed0c3eca101ff271\": container with ID starting with df5313aa08c19e33bc82d9ee5d39df53edec151bc8e4706fed0c3eca101ff271 not found: ID does not exist"
Nov 02 15:17:46.779449 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:46.779396    5304 scope.go:122] "RemoveContainer" containerID="564c908aa5ce1b803c9b8f326631537260db9011013e1d0d4f96662f86df5638"
Nov 02 15:17:46.788075 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:46.788048    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-64phg"]
Nov 02 15:17:46.800334 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:46.800288    5304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-64phg"]
Nov 02 15:17:46.823434 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:46.823394    5304 scope.go:122] "RemoveContainer" containerID="564c908aa5ce1b803c9b8f326631537260db9011013e1d0d4f96662f86df5638"
Nov 02 15:17:46.823842 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:17:46.823806    5304 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"564c908aa5ce1b803c9b8f326631537260db9011013e1d0d4f96662f86df5638\": container with ID starting with 564c908aa5ce1b803c9b8f326631537260db9011013e1d0d4f96662f86df5638 not found: ID does not exist" containerID="564c908aa5ce1b803c9b8f326631537260db9011013e1d0d4f96662f86df5638"
Nov 02 15:17:46.823966 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:46.823852    5304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"564c908aa5ce1b803c9b8f326631537260db9011013e1d0d4f96662f86df5638"} err="failed to get container status \"564c908aa5ce1b803c9b8f326631537260db9011013e1d0d4f96662f86df5638\": rpc error: code = NotFound desc = could not find container \"564c908aa5ce1b803c9b8f326631537260db9011013e1d0d4f96662f86df5638\": container with ID starting with 564c908aa5ce1b803c9b8f326631537260db9011013e1d0d4f96662f86df5638 not found: ID does not exist"
Nov 02 15:17:46.865548 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:46.865485    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-k9kqx"]
Nov 02 15:17:46.871830 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:46.871803    5304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-k9kqx"]
Nov 02 15:17:46.918062 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:46.917981    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-qqqdn"]
Nov 02 15:17:46.923894 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:46.923865    5304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-qqqdn"]
Nov 02 15:17:47.053377 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:47.053346    5304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-3d7df16e/e2e-lifecycle-model-3d7df16e-kserve-6c967d4fb5-sh4rx"]
Nov 02 15:17:47.053984 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:47.053966    5304 cpu_manager.go:414] "containerMap: removing container" podUID="e60cf723-3d4a-4bcc-bc72-0c984eba466b" containerName="istio-proxy"
Nov 02 15:17:47.053984 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:47.053984    5304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e60cf723-3d4a-4bcc-bc72-0c984eba466b" containerName="istio-proxy"
Nov 02 15:17:47.054175 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:47.054029    5304 cpu_manager.go:414] "containerMap: removing container" podUID="59fee359-49bd-4a5f-b5da-e9e22c9328f1" containerName="istio-proxy"
Nov 02 15:17:47.054175 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:47.054033    5304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="59fee359-49bd-4a5f-b5da-e9e22c9328f1" containerName="istio-proxy"
Nov 02 15:17:47.054175 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:47.054045    5304 cpu_manager.go:414] "containerMap: removing container" podUID="7f0b208d-d18a-4a38-952c-4a1abb1b30a0" containerName="istio-proxy"
Nov 02 15:17:47.054175 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:47.054051    5304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f0b208d-d18a-4a38-952c-4a1abb1b30a0" containerName="istio-proxy"
Nov 02 15:17:47.054175 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:47.054145    5304 memory_manager.go:365] "RemoveStaleState removing state" podUID="59fee359-49bd-4a5f-b5da-e9e22c9328f1" containerName="istio-proxy"
Nov 02 15:17:47.054175 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:47.054154    5304 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f0b208d-d18a-4a38-952c-4a1abb1b30a0" containerName="istio-proxy"
Nov 02 15:17:47.054175 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:47.054163    5304 memory_manager.go:365] "RemoveStaleState removing state" podUID="e60cf723-3d4a-4bcc-bc72-0c984eba466b" containerName="istio-proxy"
Nov 02 15:17:47.076724 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:47.076690    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-3d7df16e/e2e-lifecycle-model-3d7df16e-kserve-6c967d4fb5-sh4rx"
Nov 02 15:17:47.091738 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:47.091709    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-3d7df16e/e2e-lifecycle-model-3d7df16e-kserve-6c967d4fb5-sh4rx"]
Nov 02 15:17:47.102168 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:47.102143    5304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-3d7df16e\"/\"default-dockercfg-d444f\""
Nov 02 15:17:47.102367 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:47.102165    5304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-3d7df16e\"/\"e2e-lifecycle-model-3d7df16e-kserve-self-signed-certs\""
Nov 02 15:17:47.103241 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:47.103214    5304 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-3d7df16e\"/\"openshift-service-ca.crt\""
Nov 02 15:17:47.103241 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:47.103223    5304 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-3d7df16e\"/\"kube-root-ca.crt\""
Nov 02 15:17:47.158186 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:47.158156    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a786ec8c-9093-4152-8e99-4d334bd5928a-dshm\") pod \"e2e-lifecycle-model-3d7df16e-kserve-6c967d4fb5-sh4rx\" (UID: \"a786ec8c-9093-4152-8e99-4d334bd5928a\") " pod="ai-tenant-e2e-mt-3d7df16e/e2e-lifecycle-model-3d7df16e-kserve-6c967d4fb5-sh4rx"
Nov 02 15:17:47.267289 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:47.158273    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a786ec8c-9093-4152-8e99-4d334bd5928a-model-cache\") pod \"e2e-lifecycle-model-3d7df16e-kserve-6c967d4fb5-sh4rx\" (UID: \"a786ec8c-9093-4152-8e99-4d334bd5928a\") " pod="ai-tenant-e2e-mt-3d7df16e/e2e-lifecycle-model-3d7df16e-kserve-6c967d4fb5-sh4rx"
Nov 02 15:17:47.267289 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:47.158343    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a786ec8c-9093-4152-8e99-4d334bd5928a-tls-certs\") pod \"e2e-lifecycle-model-3d7df16e-kserve-6c967d4fb5-sh4rx\" (UID: \"a786ec8c-9093-4152-8e99-4d334bd5928a\") " pod="ai-tenant-e2e-mt-3d7df16e/e2e-lifecycle-model-3d7df16e-kserve-6c967d4fb5-sh4rx"
Nov 02 15:17:47.267289 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:47.158432    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a786ec8c-9093-4152-8e99-4d334bd5928a-home\") pod \"e2e-lifecycle-model-3d7df16e-kserve-6c967d4fb5-sh4rx\" (UID: \"a786ec8c-9093-4152-8e99-4d334bd5928a\") " pod="ai-tenant-e2e-mt-3d7df16e/e2e-lifecycle-model-3d7df16e-kserve-6c967d4fb5-sh4rx"
Nov 02 15:17:47.267289 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:47.158559    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a786ec8c-9093-4152-8e99-4d334bd5928a-tmp-dir\") pod \"e2e-lifecycle-model-3d7df16e-kserve-6c967d4fb5-sh4rx\" (UID: \"a786ec8c-9093-4152-8e99-4d334bd5928a\") " pod="ai-tenant-e2e-mt-3d7df16e/e2e-lifecycle-model-3d7df16e-kserve-6c967d4fb5-sh4rx"
Nov 02 15:17:47.267289 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:47.158663    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6ll2c\" (UniqueName: \"kubernetes.io/projected/a786ec8c-9093-4152-8e99-4d334bd5928a-kube-api-access-6ll2c\") pod \"e2e-lifecycle-model-3d7df16e-kserve-6c967d4fb5-sh4rx\" (UID: \"a786ec8c-9093-4152-8e99-4d334bd5928a\") " pod="ai-tenant-e2e-mt-3d7df16e/e2e-lifecycle-model-3d7df16e-kserve-6c967d4fb5-sh4rx"
Nov 02 15:17:47.267289 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:47.260068    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a786ec8c-9093-4152-8e99-4d334bd5928a-tmp-dir\") pod \"e2e-lifecycle-model-3d7df16e-kserve-6c967d4fb5-sh4rx\" (UID: \"a786ec8c-9093-4152-8e99-4d334bd5928a\") " pod="ai-tenant-e2e-mt-3d7df16e/e2e-lifecycle-model-3d7df16e-kserve-6c967d4fb5-sh4rx"
Nov 02 15:17:47.267289 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:47.260276    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6ll2c\" (UniqueName: \"kubernetes.io/projected/a786ec8c-9093-4152-8e99-4d334bd5928a-kube-api-access-6ll2c\") pod \"e2e-lifecycle-model-3d7df16e-kserve-6c967d4fb5-sh4rx\" (UID: \"a786ec8c-9093-4152-8e99-4d334bd5928a\") " pod="ai-tenant-e2e-mt-3d7df16e/e2e-lifecycle-model-3d7df16e-kserve-6c967d4fb5-sh4rx"
Nov 02 15:17:47.267289 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:47.260363    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a786ec8c-9093-4152-8e99-4d334bd5928a-dshm\") pod \"e2e-lifecycle-model-3d7df16e-kserve-6c967d4fb5-sh4rx\" (UID: \"a786ec8c-9093-4152-8e99-4d334bd5928a\") " pod="ai-tenant-e2e-mt-3d7df16e/e2e-lifecycle-model-3d7df16e-kserve-6c967d4fb5-sh4rx"
Nov 02 15:17:47.267289 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:47.260444    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a786ec8c-9093-4152-8e99-4d334bd5928a-model-cache\") pod \"e2e-lifecycle-model-3d7df16e-kserve-6c967d4fb5-sh4rx\" (UID: \"a786ec8c-9093-4152-8e99-4d334bd5928a\") " pod="ai-tenant-e2e-mt-3d7df16e/e2e-lifecycle-model-3d7df16e-kserve-6c967d4fb5-sh4rx"
Nov 02 15:17:47.267289 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:47.260496    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a786ec8c-9093-4152-8e99-4d334bd5928a-tls-certs\") pod \"e2e-lifecycle-model-3d7df16e-kserve-6c967d4fb5-sh4rx\" (UID: \"a786ec8c-9093-4152-8e99-4d334bd5928a\") " pod="ai-tenant-e2e-mt-3d7df16e/e2e-lifecycle-model-3d7df16e-kserve-6c967d4fb5-sh4rx"
Nov 02 15:17:47.267289 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:47.260503    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a786ec8c-9093-4152-8e99-4d334bd5928a-tmp-dir\") pod \"e2e-lifecycle-model-3d7df16e-kserve-6c967d4fb5-sh4rx\" (UID: \"a786ec8c-9093-4152-8e99-4d334bd5928a\") " pod="ai-tenant-e2e-mt-3d7df16e/e2e-lifecycle-model-3d7df16e-kserve-6c967d4fb5-sh4rx"
Nov 02 15:17:47.267289 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:47.260649    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a786ec8c-9093-4152-8e99-4d334bd5928a-home\") pod \"e2e-lifecycle-model-3d7df16e-kserve-6c967d4fb5-sh4rx\" (UID: \"a786ec8c-9093-4152-8e99-4d334bd5928a\") " pod="ai-tenant-e2e-mt-3d7df16e/e2e-lifecycle-model-3d7df16e-kserve-6c967d4fb5-sh4rx"
Nov 02 15:17:47.267289 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:47.260720    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a786ec8c-9093-4152-8e99-4d334bd5928a-model-cache\") pod \"e2e-lifecycle-model-3d7df16e-kserve-6c967d4fb5-sh4rx\" (UID: \"a786ec8c-9093-4152-8e99-4d334bd5928a\") " pod="ai-tenant-e2e-mt-3d7df16e/e2e-lifecycle-model-3d7df16e-kserve-6c967d4fb5-sh4rx"
Nov 02 15:17:47.267289 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:47.260909    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a786ec8c-9093-4152-8e99-4d334bd5928a-home\") pod \"e2e-lifecycle-model-3d7df16e-kserve-6c967d4fb5-sh4rx\" (UID: \"a786ec8c-9093-4152-8e99-4d334bd5928a\") " pod="ai-tenant-e2e-mt-3d7df16e/e2e-lifecycle-model-3d7df16e-kserve-6c967d4fb5-sh4rx"
Nov 02 15:17:47.267289 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:47.262508    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a786ec8c-9093-4152-8e99-4d334bd5928a-dshm\") pod \"e2e-lifecycle-model-3d7df16e-kserve-6c967d4fb5-sh4rx\" (UID: \"a786ec8c-9093-4152-8e99-4d334bd5928a\") " pod="ai-tenant-e2e-mt-3d7df16e/e2e-lifecycle-model-3d7df16e-kserve-6c967d4fb5-sh4rx"
Nov 02 15:17:47.267289 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:47.262827    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a786ec8c-9093-4152-8e99-4d334bd5928a-tls-certs\") pod \"e2e-lifecycle-model-3d7df16e-kserve-6c967d4fb5-sh4rx\" (UID: \"a786ec8c-9093-4152-8e99-4d334bd5928a\") " pod="ai-tenant-e2e-mt-3d7df16e/e2e-lifecycle-model-3d7df16e-kserve-6c967d4fb5-sh4rx"
Nov 02 15:17:47.283400 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:47.283373    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6ll2c\" (UniqueName: \"kubernetes.io/projected/a786ec8c-9093-4152-8e99-4d334bd5928a-kube-api-access-6ll2c\") pod \"e2e-lifecycle-model-3d7df16e-kserve-6c967d4fb5-sh4rx\" (UID: \"a786ec8c-9093-4152-8e99-4d334bd5928a\") " pod="ai-tenant-e2e-mt-3d7df16e/e2e-lifecycle-model-3d7df16e-kserve-6c967d4fb5-sh4rx"
Nov 02 15:17:47.403866 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:47.403773    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-3d7df16e/e2e-lifecycle-model-3d7df16e-kserve-6c967d4fb5-sh4rx"
Nov 02 15:17:47.699947 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:47.699906    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-0a7163-6c989cd9bd-xhgf5"
Nov 02 15:17:47.748677 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:47.748648    5304 generic.go:356] "Generic (PLEG): container finished" podID="350b279b-88a8-4318-b49c-d5001077f702" containerID="68e340f6a7203903e085d2bee7707621c57f9f1d6b1e7b75aa965438a9764cba" exitCode=0
Nov 02 15:17:47.748838 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:47.748717    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-ad276f-openshift-default-7cb6d54d77-rps78" event={"ID":"350b279b-88a8-4318-b49c-d5001077f702","Type":"ContainerDied","Data":"68e340f6a7203903e085d2bee7707621c57f9f1d6b1e7b75aa965438a9764cba"}
Nov 02 15:17:47.749205 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:47.749190    5304 scope.go:122] "RemoveContainer" containerID="68e340f6a7203903e085d2bee7707621c57f9f1d6b1e7b75aa965438a9764cba"
Nov 02 15:17:47.752981 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:47.752936    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:17:47.753376 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:47.753326    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:17:47.753586 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:47.753440    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:17:47.779110 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:47.779077    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-3d7df16e/e2e-lifecycle-model-3d7df16e-kserve-6c967d4fb5-sh4rx"]
Nov 02 15:17:48.120501 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:48.120470    5304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="59fee359-49bd-4a5f-b5da-e9e22c9328f1" path="/var/lib/kubelet/pods/59fee359-49bd-4a5f-b5da-e9e22c9328f1/volumes"
Nov 02 15:17:48.121001 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:48.120986    5304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7f0b208d-d18a-4a38-952c-4a1abb1b30a0" path="/var/lib/kubelet/pods/7f0b208d-d18a-4a38-952c-4a1abb1b30a0/volumes"
Nov 02 15:17:48.121515 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:48.121500    5304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e60cf723-3d4a-4bcc-bc72-0c984eba466b" path="/var/lib/kubelet/pods/e60cf723-3d4a-4bcc-bc72-0c984eba466b/volumes"
Nov 02 15:17:48.768839 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:48.768800    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-3d7df16e/e2e-lifecycle-model-3d7df16e-kserve-6c967d4fb5-sh4rx" event={"ID":"a786ec8c-9093-4152-8e99-4d334bd5928a","Type":"ContainerStarted","Data":"78f3c45e5202b15d51ab630f29cf5d7aed4054651814d316a18a63dcb4137410"}
Nov 02 15:17:48.769421 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:48.769399    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-3d7df16e/e2e-lifecycle-model-3d7df16e-kserve-6c967d4fb5-sh4rx" event={"ID":"a786ec8c-9093-4152-8e99-4d334bd5928a","Type":"ContainerStarted","Data":"e487a98b8131c7054fc7afdc912474415723b0f155c121d4b7ba8858a2d24166"}
Nov 02 15:17:48.771096 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:48.771074    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-ad276f-openshift-default-7cb6d54d77-rps78" event={"ID":"350b279b-88a8-4318-b49c-d5001077f702","Type":"ContainerStarted","Data":"04794ea5bf11354bb2d6fa52c5c14cc251ecd5c4fb600f33337bf644e004114b"}
Nov 02 15:17:48.821180 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:48.821112    5304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-mt-3d7df16e/e2e-lifecycle-model-3d7df16e-kserve-6c967d4fb5-sh4rx" podStartSLOduration=1.8210984030000001 podStartE2EDuration="1.821098403s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:17:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:17:48.817056646 +0000 UTC m=+1268.810881055" watchObservedRunningTime="2026-11-02 15:17:48.821098403 +0000 UTC m=+1268.814922827"
Nov 02 15:17:49.403861 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:49.403834    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-3d7df16e/e2e-lifecycle-model-3d7df16e-kserve-6c967d4fb5-sh4rx"
Nov 02 15:17:49.491435 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:49.491396    5304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-ad276f-openshift-default-7cb6d54d77-rps78"
Nov 02 15:17:49.491657 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:49.491521    5304 patch_prober.go:29] interesting pod/e2e-api-a-ad276f-openshift-default-7cb6d54d77-rps78 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body=
Nov 02 15:17:49.491657 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:49.491562    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ad276f-openshift-default-7cb6d54d77-rps78" podUID="350b279b-88a8-4318-b49c-d5001077f702" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused"
Nov 02 15:17:50.491083 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:50.491048    5304 patch_prober.go:29] interesting pod/e2e-api-a-ad276f-openshift-default-7cb6d54d77-rps78 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body=
Nov 02 15:17:50.491528 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:50.491110    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ad276f-openshift-default-7cb6d54d77-rps78" podUID="350b279b-88a8-4318-b49c-d5001077f702" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused"
Nov 02 15:17:51.490728 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:51.490639    5304 patch_prober.go:29] interesting pod/e2e-api-a-ad276f-openshift-default-7cb6d54d77-rps78 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body=
Nov 02 15:17:51.490728 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:51.490694    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ad276f-openshift-default-7cb6d54d77-rps78" podUID="350b279b-88a8-4318-b49c-d5001077f702" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused"
Nov 02 15:17:52.491257 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:52.491222    5304 patch_prober.go:29] interesting pod/e2e-api-a-ad276f-openshift-default-7cb6d54d77-rps78 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body=
Nov 02 15:17:52.491655 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:52.491274    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ad276f-openshift-default-7cb6d54d77-rps78" podUID="350b279b-88a8-4318-b49c-d5001077f702" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused"
Nov 02 15:17:53.491363 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:53.491321    5304 patch_prober.go:29] interesting pod/e2e-api-a-ad276f-openshift-default-7cb6d54d77-rps78 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body=
Nov 02 15:17:53.491786 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:53.491376    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ad276f-openshift-default-7cb6d54d77-rps78" podUID="350b279b-88a8-4318-b49c-d5001077f702" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused"
Nov 02 15:17:54.096644 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:54.096607    5304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-5f9166-openshift-default-5d4b56b869-skdww"]
Nov 02 15:17:54.101661 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:54.101630    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-5f9166-openshift-default-5d4b56b869-skdww"
Nov 02 15:17:54.135112 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:54.134186    5304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-5f9166-openshift-default-dockercfg-ptgjn\""
Nov 02 15:17:54.158551 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:54.158515    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-5f9166-openshift-default-5d4b56b869-skdww"]
Nov 02 15:17:54.202900 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:54.202857    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c34a9870-db4c-462a-827f-95abdfc1f68b-workload-certs\") pod \"e2e-api-b-5f9166-openshift-default-5d4b56b869-skdww\" (UID: \"c34a9870-db4c-462a-827f-95abdfc1f68b\") " pod="openshift-ingress/e2e-api-b-5f9166-openshift-default-5d4b56b869-skdww"
Nov 02 15:17:54.203098 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:54.202984    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c34a9870-db4c-462a-827f-95abdfc1f68b-istio-podinfo\") pod \"e2e-api-b-5f9166-openshift-default-5d4b56b869-skdww\" (UID: \"c34a9870-db4c-462a-827f-95abdfc1f68b\") " pod="openshift-ingress/e2e-api-b-5f9166-openshift-default-5d4b56b869-skdww"
Nov 02 15:17:54.203098 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:54.203026    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c34a9870-db4c-462a-827f-95abdfc1f68b-credential-socket\") pod \"e2e-api-b-5f9166-openshift-default-5d4b56b869-skdww\" (UID: \"c34a9870-db4c-462a-827f-95abdfc1f68b\") " pod="openshift-ingress/e2e-api-b-5f9166-openshift-default-5d4b56b869-skdww"
Nov 02 15:17:54.203251 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:54.203166    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c34a9870-db4c-462a-827f-95abdfc1f68b-istio-envoy\") pod \"e2e-api-b-5f9166-openshift-default-5d4b56b869-skdww\" (UID: \"c34a9870-db4c-462a-827f-95abdfc1f68b\") " pod="openshift-ingress/e2e-api-b-5f9166-openshift-default-5d4b56b869-skdww"
Nov 02 15:17:54.203251 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:54.203204    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c34a9870-db4c-462a-827f-95abdfc1f68b-istiod-ca-cert\") pod \"e2e-api-b-5f9166-openshift-default-5d4b56b869-skdww\" (UID: \"c34a9870-db4c-462a-827f-95abdfc1f68b\") " pod="openshift-ingress/e2e-api-b-5f9166-openshift-default-5d4b56b869-skdww"
Nov 02 15:17:54.203251 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:54.203249    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c34a9870-db4c-462a-827f-95abdfc1f68b-istio-token\") pod \"e2e-api-b-5f9166-openshift-default-5d4b56b869-skdww\" (UID: \"c34a9870-db4c-462a-827f-95abdfc1f68b\") " pod="openshift-ingress/e2e-api-b-5f9166-openshift-default-5d4b56b869-skdww"
Nov 02 15:17:54.203417 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:54.203278    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c34a9870-db4c-462a-827f-95abdfc1f68b-istio-data\") pod \"e2e-api-b-5f9166-openshift-default-5d4b56b869-skdww\" (UID: \"c34a9870-db4c-462a-827f-95abdfc1f68b\") " pod="openshift-ingress/e2e-api-b-5f9166-openshift-default-5d4b56b869-skdww"
Nov 02 15:17:54.203474 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:54.203434    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c34a9870-db4c-462a-827f-95abdfc1f68b-workload-socket\") pod \"e2e-api-b-5f9166-openshift-default-5d4b56b869-skdww\" (UID: \"c34a9870-db4c-462a-827f-95abdfc1f68b\") " pod="openshift-ingress/e2e-api-b-5f9166-openshift-default-5d4b56b869-skdww"
Nov 02 15:17:54.203531 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:54.203479    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zhh54\" (UniqueName: \"kubernetes.io/projected/c34a9870-db4c-462a-827f-95abdfc1f68b-kube-api-access-zhh54\") pod \"e2e-api-b-5f9166-openshift-default-5d4b56b869-skdww\" (UID: \"c34a9870-db4c-462a-827f-95abdfc1f68b\") " pod="openshift-ingress/e2e-api-b-5f9166-openshift-default-5d4b56b869-skdww"
Nov 02 15:17:54.305586 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:54.305545    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c34a9870-db4c-462a-827f-95abdfc1f68b-istio-envoy\") pod \"e2e-api-b-5f9166-openshift-default-5d4b56b869-skdww\" (UID: \"c34a9870-db4c-462a-827f-95abdfc1f68b\") " pod="openshift-ingress/e2e-api-b-5f9166-openshift-default-5d4b56b869-skdww"
Nov 02 15:17:54.306296 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:54.306269    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c34a9870-db4c-462a-827f-95abdfc1f68b-istiod-ca-cert\") pod \"e2e-api-b-5f9166-openshift-default-5d4b56b869-skdww\" (UID: \"c34a9870-db4c-462a-827f-95abdfc1f68b\") " pod="openshift-ingress/e2e-api-b-5f9166-openshift-default-5d4b56b869-skdww"
Nov 02 15:17:54.322150 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:54.318262    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c34a9870-db4c-462a-827f-95abdfc1f68b-istio-token\") pod \"e2e-api-b-5f9166-openshift-default-5d4b56b869-skdww\" (UID: \"c34a9870-db4c-462a-827f-95abdfc1f68b\") " pod="openshift-ingress/e2e-api-b-5f9166-openshift-default-5d4b56b869-skdww"
Nov 02 15:17:54.322150 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:54.318322    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c34a9870-db4c-462a-827f-95abdfc1f68b-istio-data\") pod \"e2e-api-b-5f9166-openshift-default-5d4b56b869-skdww\" (UID: \"c34a9870-db4c-462a-827f-95abdfc1f68b\") " pod="openshift-ingress/e2e-api-b-5f9166-openshift-default-5d4b56b869-skdww"
Nov 02 15:17:54.322150 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:54.318512    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c34a9870-db4c-462a-827f-95abdfc1f68b-workload-socket\") pod \"e2e-api-b-5f9166-openshift-default-5d4b56b869-skdww\" (UID: \"c34a9870-db4c-462a-827f-95abdfc1f68b\") " pod="openshift-ingress/e2e-api-b-5f9166-openshift-default-5d4b56b869-skdww"
Nov 02 15:17:54.322150 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:54.318570    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zhh54\" (UniqueName: \"kubernetes.io/projected/c34a9870-db4c-462a-827f-95abdfc1f68b-kube-api-access-zhh54\") pod \"e2e-api-b-5f9166-openshift-default-5d4b56b869-skdww\" (UID: \"c34a9870-db4c-462a-827f-95abdfc1f68b\") " pod="openshift-ingress/e2e-api-b-5f9166-openshift-default-5d4b56b869-skdww"
Nov 02 15:17:54.322150 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:54.318647    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c34a9870-db4c-462a-827f-95abdfc1f68b-workload-certs\") pod \"e2e-api-b-5f9166-openshift-default-5d4b56b869-skdww\" (UID: \"c34a9870-db4c-462a-827f-95abdfc1f68b\") " pod="openshift-ingress/e2e-api-b-5f9166-openshift-default-5d4b56b869-skdww"
Nov 02 15:17:54.322150 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:54.318761    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c34a9870-db4c-462a-827f-95abdfc1f68b-istio-podinfo\") pod \"e2e-api-b-5f9166-openshift-default-5d4b56b869-skdww\" (UID: \"c34a9870-db4c-462a-827f-95abdfc1f68b\") " pod="openshift-ingress/e2e-api-b-5f9166-openshift-default-5d4b56b869-skdww"
Nov 02 15:17:54.322150 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:54.318799    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c34a9870-db4c-462a-827f-95abdfc1f68b-credential-socket\") pod \"e2e-api-b-5f9166-openshift-default-5d4b56b869-skdww\" (UID: \"c34a9870-db4c-462a-827f-95abdfc1f68b\") " pod="openshift-ingress/e2e-api-b-5f9166-openshift-default-5d4b56b869-skdww"
Nov 02 15:17:54.322150 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:54.319304    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c34a9870-db4c-462a-827f-95abdfc1f68b-credential-socket\") pod \"e2e-api-b-5f9166-openshift-default-5d4b56b869-skdww\" (UID: \"c34a9870-db4c-462a-827f-95abdfc1f68b\") " pod="openshift-ingress/e2e-api-b-5f9166-openshift-default-5d4b56b869-skdww"
Nov 02 15:17:54.322150 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:54.309902    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c34a9870-db4c-462a-827f-95abdfc1f68b-istiod-ca-cert\") pod \"e2e-api-b-5f9166-openshift-default-5d4b56b869-skdww\" (UID: \"c34a9870-db4c-462a-827f-95abdfc1f68b\") " pod="openshift-ingress/e2e-api-b-5f9166-openshift-default-5d4b56b869-skdww"
Nov 02 15:17:54.322150 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:54.319901    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c34a9870-db4c-462a-827f-95abdfc1f68b-istio-data\") pod \"e2e-api-b-5f9166-openshift-default-5d4b56b869-skdww\" (UID: \"c34a9870-db4c-462a-827f-95abdfc1f68b\") " pod="openshift-ingress/e2e-api-b-5f9166-openshift-default-5d4b56b869-skdww"
Nov 02 15:17:54.322150 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:54.320161    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c34a9870-db4c-462a-827f-95abdfc1f68b-workload-socket\") pod \"e2e-api-b-5f9166-openshift-default-5d4b56b869-skdww\" (UID: \"c34a9870-db4c-462a-827f-95abdfc1f68b\") " pod="openshift-ingress/e2e-api-b-5f9166-openshift-default-5d4b56b869-skdww"
Nov 02 15:17:54.322150 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:54.320651    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c34a9870-db4c-462a-827f-95abdfc1f68b-workload-certs\") pod \"e2e-api-b-5f9166-openshift-default-5d4b56b869-skdww\" (UID: \"c34a9870-db4c-462a-827f-95abdfc1f68b\") " pod="openshift-ingress/e2e-api-b-5f9166-openshift-default-5d4b56b869-skdww"
Nov 02 15:17:54.343803 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:54.343558    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c34a9870-db4c-462a-827f-95abdfc1f68b-istio-token\") pod \"e2e-api-b-5f9166-openshift-default-5d4b56b869-skdww\" (UID: \"c34a9870-db4c-462a-827f-95abdfc1f68b\") " pod="openshift-ingress/e2e-api-b-5f9166-openshift-default-5d4b56b869-skdww"
Nov 02 15:17:54.351810 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:54.351740    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c34a9870-db4c-462a-827f-95abdfc1f68b-istio-podinfo\") pod \"e2e-api-b-5f9166-openshift-default-5d4b56b869-skdww\" (UID: \"c34a9870-db4c-462a-827f-95abdfc1f68b\") " pod="openshift-ingress/e2e-api-b-5f9166-openshift-default-5d4b56b869-skdww"
Nov 02 15:17:54.352571 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:54.352544    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zhh54\" (UniqueName: \"kubernetes.io/projected/c34a9870-db4c-462a-827f-95abdfc1f68b-kube-api-access-zhh54\") pod \"e2e-api-b-5f9166-openshift-default-5d4b56b869-skdww\" (UID: \"c34a9870-db4c-462a-827f-95abdfc1f68b\") " pod="openshift-ingress/e2e-api-b-5f9166-openshift-default-5d4b56b869-skdww"
Nov 02 15:17:54.362785 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:54.362755    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c34a9870-db4c-462a-827f-95abdfc1f68b-istio-envoy\") pod \"e2e-api-b-5f9166-openshift-default-5d4b56b869-skdww\" (UID: \"c34a9870-db4c-462a-827f-95abdfc1f68b\") " pod="openshift-ingress/e2e-api-b-5f9166-openshift-default-5d4b56b869-skdww"
Nov 02 15:17:54.421198 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:54.419921    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-5f9166-openshift-default-5d4b56b869-skdww"
Nov 02 15:17:54.587832 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:54.577402    5304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-ad276f-openshift-default-7cb6d54d77-rps78"
Nov 02 15:17:54.587832 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:54.578470    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-ad276f-openshift-default-7cb6d54d77-rps78"
Nov 02 15:17:54.587832 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:54.581198    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-ad276f-openshift-default-7cb6d54d77-rps78"
Nov 02 15:17:54.806042 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:54.805997    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-5f9166-openshift-default-5d4b56b869-skdww"]
Nov 02 15:17:54.809368 ip-10-0-135-138 kubenswrapper[5304]: W1102 15:17:54.809314    5304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc34a9870_db4c_462a_827f_95abdfc1f68b.slice/crio-ff2d4854556f6225c48e381df0d0b7bc547d3e8ebba40c57afb3c9466778ff6a WatchSource:0}: Error finding container ff2d4854556f6225c48e381df0d0b7bc547d3e8ebba40c57afb3c9466778ff6a: Status 404 returned error can't find the container with id ff2d4854556f6225c48e381df0d0b7bc547d3e8ebba40c57afb3c9466778ff6a
Nov 02 15:17:54.812506 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:54.812474    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:17:54.812807 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:54.812785    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:17:54.812956 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:54.812935    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:17:54.820078 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:17:54.818466    5304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k6fjk" containerName="extractor"
Nov 02 15:17:54.823697 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:54.823669    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-5f9166-openshift-default-5d4b56b869-skdww" event={"ID":"c34a9870-db4c-462a-827f-95abdfc1f68b","Type":"ContainerStarted","Data":"ff2d4854556f6225c48e381df0d0b7bc547d3e8ebba40c57afb3c9466778ff6a"}
Nov 02 15:17:55.833066 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:55.833022    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-5f9166-openshift-default-5d4b56b869-skdww" event={"ID":"c34a9870-db4c-462a-827f-95abdfc1f68b","Type":"ContainerStarted","Data":"5c8aa0912102f7f4905ec45c8ba28cc09a31ea54fa487ffa50530f601472732a"}
Nov 02 15:17:55.895909 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:55.895847    5304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-5f9166-openshift-default-5d4b56b869-skdww" podStartSLOduration=1.895832583 podStartE2EDuration="1.895832583s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:17:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:17:55.892481513 +0000 UTC m=+1275.886305922" watchObservedRunningTime="2026-11-02 15:17:55.895832583 +0000 UTC m=+1275.889656992"
Nov 02 15:17:56.421464 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:56.421431    5304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-5f9166-openshift-default-5d4b56b869-skdww"
Nov 02 15:17:56.428555 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:56.428527    5304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-5f9166-openshift-default-5d4b56b869-skdww"
Nov 02 15:17:56.840589 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:56.840548    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-5f9166-openshift-default-5d4b56b869-skdww"
Nov 02 15:17:56.841680 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:56.841660    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-5f9166-openshift-default-5d4b56b869-skdww"
Nov 02 15:17:57.404314 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:57.404280    5304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-3d7df16e/e2e-lifecycle-model-3d7df16e-kserve-6c967d4fb5-sh4rx"
Nov 02 15:17:57.417389 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:57.417363    5304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-3d7df16e/e2e-lifecycle-model-3d7df16e-kserve-6c967d4fb5-sh4rx"
Nov 02 15:17:57.858992 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:17:57.858966    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-3d7df16e/e2e-lifecycle-model-3d7df16e-kserve-6c967d4fb5-sh4rx"
Nov 02 15:18:00.259998 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:00.259962    5304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-5f9166-85b8cf5dc-2q6d2"]
Nov 02 15:18:00.263461 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:00.263439    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-5f9166-85b8cf5dc-2q6d2"
Nov 02 15:18:00.286053 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:00.286020    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-5f9166-85b8cf5dc-2q6d2"]
Nov 02 15:18:00.293116 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:00.293078    5304 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-5f9166\""
Nov 02 15:18:00.293294 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:00.293091    5304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-5f9166-dockercfg-nr5w5\""
Nov 02 15:18:00.388289 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:00.388250    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6530de8-45d4-4be1-9c37-e4c29d8a3ace-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-5f9166-85b8cf5dc-2q6d2\" (UID: \"c6530de8-45d4-4be1-9c37-e4c29d8a3ace\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-5f9166-85b8cf5dc-2q6d2"
Nov 02 15:18:00.436396 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:00.394605    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tdlhh\" (UniqueName: \"kubernetes.io/projected/c6530de8-45d4-4be1-9c37-e4c29d8a3ace-kube-api-access-tdlhh\") pod \"payload-pre-processing-e2e-api-b-5f9166-85b8cf5dc-2q6d2\" (UID: \"c6530de8-45d4-4be1-9c37-e4c29d8a3ace\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-5f9166-85b8cf5dc-2q6d2"
Nov 02 15:18:00.570959 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:00.570861    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tdlhh\" (UniqueName: \"kubernetes.io/projected/c6530de8-45d4-4be1-9c37-e4c29d8a3ace-kube-api-access-tdlhh\") pod \"payload-pre-processing-e2e-api-b-5f9166-85b8cf5dc-2q6d2\" (UID: \"c6530de8-45d4-4be1-9c37-e4c29d8a3ace\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-5f9166-85b8cf5dc-2q6d2"
Nov 02 15:18:00.571187 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:00.571057    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6530de8-45d4-4be1-9c37-e4c29d8a3ace-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-5f9166-85b8cf5dc-2q6d2\" (UID: \"c6530de8-45d4-4be1-9c37-e4c29d8a3ace\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-5f9166-85b8cf5dc-2q6d2"
Nov 02 15:18:00.572644 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:00.572605    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6530de8-45d4-4be1-9c37-e4c29d8a3ace-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-5f9166-85b8cf5dc-2q6d2\" (UID: \"c6530de8-45d4-4be1-9c37-e4c29d8a3ace\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-5f9166-85b8cf5dc-2q6d2"
Nov 02 15:18:00.604116 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:00.604079    5304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-5f9166-8c7954fff-tq5g2"]
Nov 02 15:18:00.606108 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:00.606078    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tdlhh\" (UniqueName: \"kubernetes.io/projected/c6530de8-45d4-4be1-9c37-e4c29d8a3ace-kube-api-access-tdlhh\") pod \"payload-pre-processing-e2e-api-b-5f9166-85b8cf5dc-2q6d2\" (UID: \"c6530de8-45d4-4be1-9c37-e4c29d8a3ace\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-5f9166-85b8cf5dc-2q6d2"
Nov 02 15:18:00.614382 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:00.614362    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-5f9166-8c7954fff-tq5g2"]
Nov 02 15:18:00.614632 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:00.614586    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-5f9166-8c7954fff-tq5g2"
Nov 02 15:18:00.672638 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:00.672608    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3bdd3ae-9a9c-4945-b820-96d56f208aa3-plugins-config-volume\") pod \"payload-processing-e2e-api-b-5f9166-8c7954fff-tq5g2\" (UID: \"b3bdd3ae-9a9c-4945-b820-96d56f208aa3\") " pod="openshift-ingress/payload-processing-e2e-api-b-5f9166-8c7954fff-tq5g2"
Nov 02 15:18:00.672810 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:00.672743    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-glzsf\" (UniqueName: \"kubernetes.io/projected/b3bdd3ae-9a9c-4945-b820-96d56f208aa3-kube-api-access-glzsf\") pod \"payload-processing-e2e-api-b-5f9166-8c7954fff-tq5g2\" (UID: \"b3bdd3ae-9a9c-4945-b820-96d56f208aa3\") " pod="openshift-ingress/payload-processing-e2e-api-b-5f9166-8c7954fff-tq5g2"
Nov 02 15:18:00.774708 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:00.774671    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-glzsf\" (UniqueName: \"kubernetes.io/projected/b3bdd3ae-9a9c-4945-b820-96d56f208aa3-kube-api-access-glzsf\") pod \"payload-processing-e2e-api-b-5f9166-8c7954fff-tq5g2\" (UID: \"b3bdd3ae-9a9c-4945-b820-96d56f208aa3\") " pod="openshift-ingress/payload-processing-e2e-api-b-5f9166-8c7954fff-tq5g2"
Nov 02 15:18:00.774983 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:00.774963    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3bdd3ae-9a9c-4945-b820-96d56f208aa3-plugins-config-volume\") pod \"payload-processing-e2e-api-b-5f9166-8c7954fff-tq5g2\" (UID: \"b3bdd3ae-9a9c-4945-b820-96d56f208aa3\") " pod="openshift-ingress/payload-processing-e2e-api-b-5f9166-8c7954fff-tq5g2"
Nov 02 15:18:00.775848 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:00.775820    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3bdd3ae-9a9c-4945-b820-96d56f208aa3-plugins-config-volume\") pod \"payload-processing-e2e-api-b-5f9166-8c7954fff-tq5g2\" (UID: \"b3bdd3ae-9a9c-4945-b820-96d56f208aa3\") " pod="openshift-ingress/payload-processing-e2e-api-b-5f9166-8c7954fff-tq5g2"
Nov 02 15:18:00.798224 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:00.798188    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-glzsf\" (UniqueName: \"kubernetes.io/projected/b3bdd3ae-9a9c-4945-b820-96d56f208aa3-kube-api-access-glzsf\") pod \"payload-processing-e2e-api-b-5f9166-8c7954fff-tq5g2\" (UID: \"b3bdd3ae-9a9c-4945-b820-96d56f208aa3\") " pod="openshift-ingress/payload-processing-e2e-api-b-5f9166-8c7954fff-tq5g2"
Nov 02 15:18:00.877787 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:00.877746    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-5f9166-85b8cf5dc-2q6d2"
Nov 02 15:18:00.939468 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:00.939439    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-5f9166-8c7954fff-tq5g2"
Nov 02 15:18:01.226636 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:01.226563    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-5f9166-85b8cf5dc-2q6d2"]
Nov 02 15:18:01.230456 ip-10-0-135-138 kubenswrapper[5304]: W1102 15:18:01.230426    5304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6530de8_45d4_4be1_9c37_e4c29d8a3ace.slice/crio-36ff9910bebed1bcb612df23d3d335b4173999e7802778397635f7d53b2937ef WatchSource:0}: Error finding container 36ff9910bebed1bcb612df23d3d335b4173999e7802778397635f7d53b2937ef: Status 404 returned error can't find the container with id 36ff9910bebed1bcb612df23d3d335b4173999e7802778397635f7d53b2937ef
Nov 02 15:18:01.306611 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:01.306581    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-5f9166-8c7954fff-tq5g2"]
Nov 02 15:18:01.311467 ip-10-0-135-138 kubenswrapper[5304]: W1102 15:18:01.311442    5304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3bdd3ae_9a9c_4945_b820_96d56f208aa3.slice/crio-bfac8978a576d6c7fc7042fe18d24b6dacf0c506e26f17be99abae82e26d0adb WatchSource:0}: Error finding container bfac8978a576d6c7fc7042fe18d24b6dacf0c506e26f17be99abae82e26d0adb: Status 404 returned error can't find the container with id bfac8978a576d6c7fc7042fe18d24b6dacf0c506e26f17be99abae82e26d0adb
Nov 02 15:18:01.876756 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:01.876715    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-5f9166-8c7954fff-tq5g2" event={"ID":"b3bdd3ae-9a9c-4945-b820-96d56f208aa3","Type":"ContainerStarted","Data":"2078e55337110072c4297383f22c973ac713512124e9c54753772d7291891809"}
Nov 02 15:18:01.876949 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:01.876815    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-5f9166-8c7954fff-tq5g2" event={"ID":"b3bdd3ae-9a9c-4945-b820-96d56f208aa3","Type":"ContainerStarted","Data":"bfac8978a576d6c7fc7042fe18d24b6dacf0c506e26f17be99abae82e26d0adb"}
Nov 02 15:18:01.876949 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:01.876844    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-5f9166-8c7954fff-tq5g2"
Nov 02 15:18:01.878510 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:01.878479    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-5f9166-85b8cf5dc-2q6d2" event={"ID":"c6530de8-45d4-4be1-9c37-e4c29d8a3ace","Type":"ContainerStarted","Data":"d614fd9c13ea98f71be766c10b51acda536172b0177ed930a069944922f2e3c3"}
Nov 02 15:18:01.878662 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:01.878524    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-5f9166-85b8cf5dc-2q6d2" event={"ID":"c6530de8-45d4-4be1-9c37-e4c29d8a3ace","Type":"ContainerStarted","Data":"36ff9910bebed1bcb612df23d3d335b4173999e7802778397635f7d53b2937ef"}
Nov 02 15:18:01.878662 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:01.878570    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-5f9166-85b8cf5dc-2q6d2"
Nov 02 15:18:01.922694 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:01.922645    5304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-5f9166-8c7954fff-tq5g2" podStartSLOduration=1.9226317910000001 podStartE2EDuration="1.922631791s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:18:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:18:01.919756565 +0000 UTC m=+1281.913580976" watchObservedRunningTime="2026-11-02 15:18:01.922631791 +0000 UTC m=+1281.916456200"
Nov 02 15:18:01.959959 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:01.959904    5304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-5f9166-85b8cf5dc-2q6d2" podStartSLOduration=1.959891308 podStartE2EDuration="1.959891308s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:18:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:18:01.959240931 +0000 UTC m=+1281.953065340" watchObservedRunningTime="2026-11-02 15:18:01.959891308 +0000 UTC m=+1281.953715716"
Nov 02 15:18:02.025443 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:02.025410    5304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-lpbsq"]
Nov 02 15:18:02.028670 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:02.028652    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-lpbsq"
Nov 02 15:18:02.054490 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:02.054457    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-lpbsq"]
Nov 02 15:18:02.095774 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:02.095720    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6e1530b5-8c7e-4a83-af8d-67bb9d476b50-istio-token\") pod \"e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-lpbsq\" (UID: \"6e1530b5-8c7e-4a83-af8d-67bb9d476b50\") " pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-lpbsq"
Nov 02 15:18:02.095943 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:02.095801    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e1530b5-8c7e-4a83-af8d-67bb9d476b50-credential-socket\") pod \"e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-lpbsq\" (UID: \"6e1530b5-8c7e-4a83-af8d-67bb9d476b50\") " pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-lpbsq"
Nov 02 15:18:02.095943 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:02.095921    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6e1530b5-8c7e-4a83-af8d-67bb9d476b50-istio-data\") pod \"e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-lpbsq\" (UID: \"6e1530b5-8c7e-4a83-af8d-67bb9d476b50\") " pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-lpbsq"
Nov 02 15:18:02.096032 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:02.095996    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6e1530b5-8c7e-4a83-af8d-67bb9d476b50-istio-podinfo\") pod \"e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-lpbsq\" (UID: \"6e1530b5-8c7e-4a83-af8d-67bb9d476b50\") " pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-lpbsq"
Nov 02 15:18:02.096069 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:02.096037    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6e1530b5-8c7e-4a83-af8d-67bb9d476b50-istio-envoy\") pod \"e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-lpbsq\" (UID: \"6e1530b5-8c7e-4a83-af8d-67bb9d476b50\") " pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-lpbsq"
Nov 02 15:18:02.096105 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:02.096077    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e1530b5-8c7e-4a83-af8d-67bb9d476b50-workload-socket\") pod \"e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-lpbsq\" (UID: \"6e1530b5-8c7e-4a83-af8d-67bb9d476b50\") " pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-lpbsq"
Nov 02 15:18:02.096199 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:02.096112    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6e1530b5-8c7e-4a83-af8d-67bb9d476b50-workload-certs\") pod \"e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-lpbsq\" (UID: \"6e1530b5-8c7e-4a83-af8d-67bb9d476b50\") " pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-lpbsq"
Nov 02 15:18:02.096199 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:02.096159    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-87cgx\" (UniqueName: \"kubernetes.io/projected/6e1530b5-8c7e-4a83-af8d-67bb9d476b50-kube-api-access-87cgx\") pod \"e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-lpbsq\" (UID: \"6e1530b5-8c7e-4a83-af8d-67bb9d476b50\") " pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-lpbsq"
Nov 02 15:18:02.096313 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:02.096204    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6e1530b5-8c7e-4a83-af8d-67bb9d476b50-istiod-ca-cert\") pod \"e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-lpbsq\" (UID: \"6e1530b5-8c7e-4a83-af8d-67bb9d476b50\") " pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-lpbsq"
Nov 02 15:18:02.198406 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:02.198298    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6e1530b5-8c7e-4a83-af8d-67bb9d476b50-istio-data\") pod \"e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-lpbsq\" (UID: \"6e1530b5-8c7e-4a83-af8d-67bb9d476b50\") " pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-lpbsq"
Nov 02 15:18:02.198406 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:02.198396    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6e1530b5-8c7e-4a83-af8d-67bb9d476b50-istio-podinfo\") pod \"e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-lpbsq\" (UID: \"6e1530b5-8c7e-4a83-af8d-67bb9d476b50\") " pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-lpbsq"
Nov 02 15:18:02.198775 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:02.198741    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6e1530b5-8c7e-4a83-af8d-67bb9d476b50-istio-envoy\") pod \"e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-lpbsq\" (UID: \"6e1530b5-8c7e-4a83-af8d-67bb9d476b50\") " pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-lpbsq"
Nov 02 15:18:02.198919 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:02.198750    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6e1530b5-8c7e-4a83-af8d-67bb9d476b50-istio-data\") pod \"e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-lpbsq\" (UID: \"6e1530b5-8c7e-4a83-af8d-67bb9d476b50\") " pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-lpbsq"
Nov 02 15:18:02.198919 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:02.198867    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e1530b5-8c7e-4a83-af8d-67bb9d476b50-workload-socket\") pod \"e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-lpbsq\" (UID: \"6e1530b5-8c7e-4a83-af8d-67bb9d476b50\") " pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-lpbsq"
Nov 02 15:18:02.199063 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:02.198932    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6e1530b5-8c7e-4a83-af8d-67bb9d476b50-workload-certs\") pod \"e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-lpbsq\" (UID: \"6e1530b5-8c7e-4a83-af8d-67bb9d476b50\") " pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-lpbsq"
Nov 02 15:18:02.199063 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:02.198962    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87cgx\" (UniqueName: \"kubernetes.io/projected/6e1530b5-8c7e-4a83-af8d-67bb9d476b50-kube-api-access-87cgx\") pod \"e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-lpbsq\" (UID: \"6e1530b5-8c7e-4a83-af8d-67bb9d476b50\") " pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-lpbsq"
Nov 02 15:18:02.199063 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:02.199008    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6e1530b5-8c7e-4a83-af8d-67bb9d476b50-istiod-ca-cert\") pod \"e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-lpbsq\" (UID: \"6e1530b5-8c7e-4a83-af8d-67bb9d476b50\") " pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-lpbsq"
Nov 02 15:18:02.199246 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:02.199166    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e1530b5-8c7e-4a83-af8d-67bb9d476b50-workload-socket\") pod \"e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-lpbsq\" (UID: \"6e1530b5-8c7e-4a83-af8d-67bb9d476b50\") " pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-lpbsq"
Nov 02 15:18:02.199305 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:02.199289    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6e1530b5-8c7e-4a83-af8d-67bb9d476b50-istio-token\") pod \"e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-lpbsq\" (UID: \"6e1530b5-8c7e-4a83-af8d-67bb9d476b50\") " pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-lpbsq"
Nov 02 15:18:02.199370 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:02.199346    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e1530b5-8c7e-4a83-af8d-67bb9d476b50-credential-socket\") pod \"e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-lpbsq\" (UID: \"6e1530b5-8c7e-4a83-af8d-67bb9d476b50\") " pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-lpbsq"
Nov 02 15:18:02.199763 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:02.199736    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6e1530b5-8c7e-4a83-af8d-67bb9d476b50-workload-certs\") pod \"e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-lpbsq\" (UID: \"6e1530b5-8c7e-4a83-af8d-67bb9d476b50\") " pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-lpbsq"
Nov 02 15:18:02.199916 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:02.199759    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e1530b5-8c7e-4a83-af8d-67bb9d476b50-credential-socket\") pod \"e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-lpbsq\" (UID: \"6e1530b5-8c7e-4a83-af8d-67bb9d476b50\") " pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-lpbsq"
Nov 02 15:18:02.201413 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:02.201365    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6e1530b5-8c7e-4a83-af8d-67bb9d476b50-istio-envoy\") pod \"e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-lpbsq\" (UID: \"6e1530b5-8c7e-4a83-af8d-67bb9d476b50\") " pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-lpbsq"
Nov 02 15:18:02.201513 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:02.201486    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6e1530b5-8c7e-4a83-af8d-67bb9d476b50-istio-podinfo\") pod \"e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-lpbsq\" (UID: \"6e1530b5-8c7e-4a83-af8d-67bb9d476b50\") " pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-lpbsq"
Nov 02 15:18:02.203068 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:02.203048    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6e1530b5-8c7e-4a83-af8d-67bb9d476b50-istiod-ca-cert\") pod \"e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-lpbsq\" (UID: \"6e1530b5-8c7e-4a83-af8d-67bb9d476b50\") " pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-lpbsq"
Nov 02 15:18:02.225917 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:02.225889    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-87cgx\" (UniqueName: \"kubernetes.io/projected/6e1530b5-8c7e-4a83-af8d-67bb9d476b50-kube-api-access-87cgx\") pod \"e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-lpbsq\" (UID: \"6e1530b5-8c7e-4a83-af8d-67bb9d476b50\") " pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-lpbsq"
Nov 02 15:18:02.226070 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:02.226031    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6e1530b5-8c7e-4a83-af8d-67bb9d476b50-istio-token\") pod \"e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-lpbsq\" (UID: \"6e1530b5-8c7e-4a83-af8d-67bb9d476b50\") " pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-lpbsq"
Nov 02 15:18:02.346490 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:02.346455    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-lpbsq"
Nov 02 15:18:02.706367 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:02.706298    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-lpbsq"]
Nov 02 15:18:02.710087 ip-10-0-135-138 kubenswrapper[5304]: W1102 15:18:02.710060    5304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e1530b5_8c7e_4a83_af8d_67bb9d476b50.slice/crio-f63e61314607e851ca06981af8b5e81f4c03757d95a58557e132f032e489a5d4 WatchSource:0}: Error finding container f63e61314607e851ca06981af8b5e81f4c03757d95a58557e132f032e489a5d4: Status 404 returned error can't find the container with id f63e61314607e851ca06981af8b5e81f4c03757d95a58557e132f032e489a5d4
Nov 02 15:18:02.712771 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:02.712713    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:18:02.712894 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:02.712828    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:18:02.712894 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:02.712876    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:18:02.890401 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:02.890359    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-lpbsq" event={"ID":"6e1530b5-8c7e-4a83-af8d-67bb9d476b50","Type":"ContainerStarted","Data":"028c1e2af928f940991002829c6a2a71071b358ea9a5bf46f5dafec298d82351"}
Nov 02 15:18:02.890626 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:02.890421    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-lpbsq" event={"ID":"6e1530b5-8c7e-4a83-af8d-67bb9d476b50","Type":"ContainerStarted","Data":"f63e61314607e851ca06981af8b5e81f4c03757d95a58557e132f032e489a5d4"}
Nov 02 15:18:02.944524 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:02.944450    5304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-lpbsq" podStartSLOduration=1.944427245 podStartE2EDuration="1.944427245s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:18:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:18:02.936905474 +0000 UTC m=+1282.930729886" watchObservedRunningTime="2026-11-02 15:18:02.944427245 +0000 UTC m=+1282.938251655"
Nov 02 15:18:03.347068 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:03.347025    5304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-lpbsq"
Nov 02 15:18:03.348478 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:03.348455    5304 patch_prober.go:29] interesting pod/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-lpbsq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body=
Nov 02 15:18:03.348545 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:03.348506    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-lpbsq" podUID="6e1530b5-8c7e-4a83-af8d-67bb9d476b50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused"
Nov 02 15:18:04.349666 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:04.349626    5304 patch_prober.go:29] interesting pod/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-lpbsq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body=
Nov 02 15:18:04.350317 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:04.350283    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-lpbsq" podUID="6e1530b5-8c7e-4a83-af8d-67bb9d476b50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused"
Nov 02 15:18:05.351150 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:05.350730    5304 patch_prober.go:29] interesting pod/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-lpbsq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body=
Nov 02 15:18:05.351150 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:05.350812    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-lpbsq" podUID="6e1530b5-8c7e-4a83-af8d-67bb9d476b50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused"
Nov 02 15:18:06.356147 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:06.348726    5304 patch_prober.go:29] interesting pod/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-lpbsq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body=
Nov 02 15:18:06.356147 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:06.348799    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-lpbsq" podUID="6e1530b5-8c7e-4a83-af8d-67bb9d476b50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused"
Nov 02 15:18:07.350611 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:07.350569    5304 patch_prober.go:29] interesting pod/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-lpbsq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body=
Nov 02 15:18:07.350914 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:07.350874    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-lpbsq" podUID="6e1530b5-8c7e-4a83-af8d-67bb9d476b50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused"
Nov 02 15:18:08.349157 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:08.348720    5304 patch_prober.go:29] interesting pod/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-lpbsq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body=
Nov 02 15:18:08.349157 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:08.348796    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-lpbsq" podUID="6e1530b5-8c7e-4a83-af8d-67bb9d476b50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused"
Nov 02 15:18:09.355213 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:09.352399    5304 patch_prober.go:29] interesting pod/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-lpbsq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body=
Nov 02 15:18:09.355213 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:09.352507    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-lpbsq" podUID="6e1530b5-8c7e-4a83-af8d-67bb9d476b50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused"
Nov 02 15:18:10.352798 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:10.352762    5304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-lpbsq"
Nov 02 15:18:10.353180 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:10.353156    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-lpbsq"
Nov 02 15:18:10.353844 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:10.353827    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-lpbsq"
Nov 02 15:18:12.893859 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:12.893830    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-5f9166-8c7954fff-tq5g2"
Nov 02 15:18:12.894355 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:12.893899    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-5f9166-85b8cf5dc-2q6d2"
Nov 02 15:18:17.173854 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:17.173816    5304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-xtxns"]
Nov 02 15:18:17.179121 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:17.179092    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-xtxns"
Nov 02 15:18:17.202321 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:17.202292    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-xtxns"]
Nov 02 15:18:17.296054 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:17.296007    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/abefd6e8-6072-45e0-9585-7c16ae3b6b35-istio-data\") pod \"e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-xtxns\" (UID: \"abefd6e8-6072-45e0-9585-7c16ae3b6b35\") " pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-xtxns"
Nov 02 15:18:17.296245 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:17.296158    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/abefd6e8-6072-45e0-9585-7c16ae3b6b35-istio-token\") pod \"e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-xtxns\" (UID: \"abefd6e8-6072-45e0-9585-7c16ae3b6b35\") " pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-xtxns"
Nov 02 15:18:17.296303 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:17.296279    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/abefd6e8-6072-45e0-9585-7c16ae3b6b35-credential-socket\") pod \"e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-xtxns\" (UID: \"abefd6e8-6072-45e0-9585-7c16ae3b6b35\") " pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-xtxns"
Nov 02 15:18:17.296355 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:17.296330    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/abefd6e8-6072-45e0-9585-7c16ae3b6b35-istiod-ca-cert\") pod \"e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-xtxns\" (UID: \"abefd6e8-6072-45e0-9585-7c16ae3b6b35\") " pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-xtxns"
Nov 02 15:18:17.296507 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:17.296489    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/abefd6e8-6072-45e0-9585-7c16ae3b6b35-workload-certs\") pod \"e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-xtxns\" (UID: \"abefd6e8-6072-45e0-9585-7c16ae3b6b35\") " pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-xtxns"
Nov 02 15:18:17.296601 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:17.296585    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7rb6z\" (UniqueName: \"kubernetes.io/projected/abefd6e8-6072-45e0-9585-7c16ae3b6b35-kube-api-access-7rb6z\") pod \"e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-xtxns\" (UID: \"abefd6e8-6072-45e0-9585-7c16ae3b6b35\") " pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-xtxns"
Nov 02 15:18:17.296693 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:17.296677    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/abefd6e8-6072-45e0-9585-7c16ae3b6b35-workload-socket\") pod \"e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-xtxns\" (UID: \"abefd6e8-6072-45e0-9585-7c16ae3b6b35\") " pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-xtxns"
Nov 02 15:18:17.296754 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:17.296738    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/abefd6e8-6072-45e0-9585-7c16ae3b6b35-istio-podinfo\") pod \"e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-xtxns\" (UID: \"abefd6e8-6072-45e0-9585-7c16ae3b6b35\") " pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-xtxns"
Nov 02 15:18:17.296878 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:17.296865    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/abefd6e8-6072-45e0-9585-7c16ae3b6b35-istio-envoy\") pod \"e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-xtxns\" (UID: \"abefd6e8-6072-45e0-9585-7c16ae3b6b35\") " pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-xtxns"
Nov 02 15:18:17.399177 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:17.399116    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/abefd6e8-6072-45e0-9585-7c16ae3b6b35-workload-certs\") pod \"e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-xtxns\" (UID: \"abefd6e8-6072-45e0-9585-7c16ae3b6b35\") " pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-xtxns"
Nov 02 15:18:17.399380 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:17.399233    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7rb6z\" (UniqueName: \"kubernetes.io/projected/abefd6e8-6072-45e0-9585-7c16ae3b6b35-kube-api-access-7rb6z\") pod \"e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-xtxns\" (UID: \"abefd6e8-6072-45e0-9585-7c16ae3b6b35\") " pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-xtxns"
Nov 02 15:18:17.399380 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:17.399308    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/abefd6e8-6072-45e0-9585-7c16ae3b6b35-workload-socket\") pod \"e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-xtxns\" (UID: \"abefd6e8-6072-45e0-9585-7c16ae3b6b35\") " pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-xtxns"
Nov 02 15:18:17.399380 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:17.399362    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/abefd6e8-6072-45e0-9585-7c16ae3b6b35-istio-podinfo\") pod \"e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-xtxns\" (UID: \"abefd6e8-6072-45e0-9585-7c16ae3b6b35\") " pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-xtxns"
Nov 02 15:18:17.399587 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:17.399489    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/abefd6e8-6072-45e0-9585-7c16ae3b6b35-istio-envoy\") pod \"e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-xtxns\" (UID: \"abefd6e8-6072-45e0-9585-7c16ae3b6b35\") " pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-xtxns"
Nov 02 15:18:17.399587 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:17.399545    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/abefd6e8-6072-45e0-9585-7c16ae3b6b35-istio-data\") pod \"e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-xtxns\" (UID: \"abefd6e8-6072-45e0-9585-7c16ae3b6b35\") " pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-xtxns"
Nov 02 15:18:17.399587 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:17.399572    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/abefd6e8-6072-45e0-9585-7c16ae3b6b35-workload-certs\") pod \"e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-xtxns\" (UID: \"abefd6e8-6072-45e0-9585-7c16ae3b6b35\") " pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-xtxns"
Nov 02 15:18:17.399828 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:17.399808    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/abefd6e8-6072-45e0-9585-7c16ae3b6b35-workload-socket\") pod \"e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-xtxns\" (UID: \"abefd6e8-6072-45e0-9585-7c16ae3b6b35\") " pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-xtxns"
Nov 02 15:18:17.400372 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:17.400349    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/abefd6e8-6072-45e0-9585-7c16ae3b6b35-istio-data\") pod \"e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-xtxns\" (UID: \"abefd6e8-6072-45e0-9585-7c16ae3b6b35\") " pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-xtxns"
Nov 02 15:18:17.400720 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:17.400695    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/abefd6e8-6072-45e0-9585-7c16ae3b6b35-istio-token\") pod \"e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-xtxns\" (UID: \"abefd6e8-6072-45e0-9585-7c16ae3b6b35\") " pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-xtxns"
Nov 02 15:18:17.400803 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:17.400780    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/abefd6e8-6072-45e0-9585-7c16ae3b6b35-credential-socket\") pod \"e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-xtxns\" (UID: \"abefd6e8-6072-45e0-9585-7c16ae3b6b35\") " pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-xtxns"
Nov 02 15:18:17.400856 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:17.400822    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/abefd6e8-6072-45e0-9585-7c16ae3b6b35-istiod-ca-cert\") pod \"e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-xtxns\" (UID: \"abefd6e8-6072-45e0-9585-7c16ae3b6b35\") " pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-xtxns"
Nov 02 15:18:17.401293 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:17.401268    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/abefd6e8-6072-45e0-9585-7c16ae3b6b35-credential-socket\") pod \"e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-xtxns\" (UID: \"abefd6e8-6072-45e0-9585-7c16ae3b6b35\") " pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-xtxns"
Nov 02 15:18:17.401667 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:17.401642    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/abefd6e8-6072-45e0-9585-7c16ae3b6b35-istiod-ca-cert\") pod \"e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-xtxns\" (UID: \"abefd6e8-6072-45e0-9585-7c16ae3b6b35\") " pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-xtxns"
Nov 02 15:18:17.402607 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:17.402562    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/abefd6e8-6072-45e0-9585-7c16ae3b6b35-istio-envoy\") pod \"e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-xtxns\" (UID: \"abefd6e8-6072-45e0-9585-7c16ae3b6b35\") " pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-xtxns"
Nov 02 15:18:17.403804 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:17.403773    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/abefd6e8-6072-45e0-9585-7c16ae3b6b35-istio-podinfo\") pod \"e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-xtxns\" (UID: \"abefd6e8-6072-45e0-9585-7c16ae3b6b35\") " pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-xtxns"
Nov 02 15:18:17.491062 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:17.489145    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/abefd6e8-6072-45e0-9585-7c16ae3b6b35-istio-token\") pod \"e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-xtxns\" (UID: \"abefd6e8-6072-45e0-9585-7c16ae3b6b35\") " pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-xtxns"
Nov 02 15:18:17.491062 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:17.490758    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7rb6z\" (UniqueName: \"kubernetes.io/projected/abefd6e8-6072-45e0-9585-7c16ae3b6b35-kube-api-access-7rb6z\") pod \"e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-xtxns\" (UID: \"abefd6e8-6072-45e0-9585-7c16ae3b6b35\") " pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-xtxns"
Nov 02 15:18:17.500253 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:17.500228    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-xtxns"
Nov 02 15:18:17.982668 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:17.976855    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:18:17.982668 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:17.976952    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:18:17.982668 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:17.976998    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:18:18.010232 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:18.010200    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-xtxns"]
Nov 02 15:18:18.015905 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:18.015872    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-xtxns" event={"ID":"abefd6e8-6072-45e0-9585-7c16ae3b6b35","Type":"ContainerStarted","Data":"5058c79345a2be62594db9c382adee6d374c86c3db0b4a018712fb3b6036df59"}
Nov 02 15:18:19.095043 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:19.087262    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-xtxns" event={"ID":"abefd6e8-6072-45e0-9585-7c16ae3b6b35","Type":"ContainerStarted","Data":"1ef1c5c5df17bc3b3bfd3c8cfb4c51ddf09afca3930e7d5312c47a35249e9ee3"}
Nov 02 15:18:19.164470 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:19.164400    5304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-xtxns" podStartSLOduration=2.164377581 podStartE2EDuration="2.164377581s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:18:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:18:19.160764601 +0000 UTC m=+1299.154589011" watchObservedRunningTime="2026-11-02 15:18:19.164377581 +0000 UTC m=+1299.158201991"
Nov 02 15:18:19.249904 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:19.249868    5304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-501738-openshift-default-54c4d5d859-bwbmq"]
Nov 02 15:18:19.277445 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:19.277413    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-501738-openshift-default-54c4d5d859-bwbmq"]
Nov 02 15:18:19.277818 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:19.277786    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-501738-openshift-default-54c4d5d859-bwbmq"
Nov 02 15:18:19.354298 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:19.344554    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7d14663-db08-4ae9-9dae-98f56d51f8a4-workload-socket\") pod \"e2e-shared-a-w4-501738-openshift-default-54c4d5d859-bwbmq\" (UID: \"f7d14663-db08-4ae9-9dae-98f56d51f8a4\") " pod="openshift-ingress/e2e-shared-a-w4-501738-openshift-default-54c4d5d859-bwbmq"
Nov 02 15:18:19.354298 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:19.344651    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7d14663-db08-4ae9-9dae-98f56d51f8a4-istio-token\") pod \"e2e-shared-a-w4-501738-openshift-default-54c4d5d859-bwbmq\" (UID: \"f7d14663-db08-4ae9-9dae-98f56d51f8a4\") " pod="openshift-ingress/e2e-shared-a-w4-501738-openshift-default-54c4d5d859-bwbmq"
Nov 02 15:18:19.354298 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:19.344678    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7d14663-db08-4ae9-9dae-98f56d51f8a4-workload-certs\") pod \"e2e-shared-a-w4-501738-openshift-default-54c4d5d859-bwbmq\" (UID: \"f7d14663-db08-4ae9-9dae-98f56d51f8a4\") " pod="openshift-ingress/e2e-shared-a-w4-501738-openshift-default-54c4d5d859-bwbmq"
Nov 02 15:18:19.354298 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:19.344750    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7d14663-db08-4ae9-9dae-98f56d51f8a4-istio-podinfo\") pod \"e2e-shared-a-w4-501738-openshift-default-54c4d5d859-bwbmq\" (UID: \"f7d14663-db08-4ae9-9dae-98f56d51f8a4\") " pod="openshift-ingress/e2e-shared-a-w4-501738-openshift-default-54c4d5d859-bwbmq"
Nov 02 15:18:19.354298 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:19.344779    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7d14663-db08-4ae9-9dae-98f56d51f8a4-istio-envoy\") pod \"e2e-shared-a-w4-501738-openshift-default-54c4d5d859-bwbmq\" (UID: \"f7d14663-db08-4ae9-9dae-98f56d51f8a4\") " pod="openshift-ingress/e2e-shared-a-w4-501738-openshift-default-54c4d5d859-bwbmq"
Nov 02 15:18:19.354298 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:19.344849    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nggqv\" (UniqueName: \"kubernetes.io/projected/f7d14663-db08-4ae9-9dae-98f56d51f8a4-kube-api-access-nggqv\") pod \"e2e-shared-a-w4-501738-openshift-default-54c4d5d859-bwbmq\" (UID: \"f7d14663-db08-4ae9-9dae-98f56d51f8a4\") " pod="openshift-ingress/e2e-shared-a-w4-501738-openshift-default-54c4d5d859-bwbmq"
Nov 02 15:18:19.354298 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:19.344924    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7d14663-db08-4ae9-9dae-98f56d51f8a4-credential-socket\") pod \"e2e-shared-a-w4-501738-openshift-default-54c4d5d859-bwbmq\" (UID: \"f7d14663-db08-4ae9-9dae-98f56d51f8a4\") " pod="openshift-ingress/e2e-shared-a-w4-501738-openshift-default-54c4d5d859-bwbmq"
Nov 02 15:18:19.354298 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:19.344946    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7d14663-db08-4ae9-9dae-98f56d51f8a4-istio-data\") pod \"e2e-shared-a-w4-501738-openshift-default-54c4d5d859-bwbmq\" (UID: \"f7d14663-db08-4ae9-9dae-98f56d51f8a4\") " pod="openshift-ingress/e2e-shared-a-w4-501738-openshift-default-54c4d5d859-bwbmq"
Nov 02 15:18:19.354298 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:19.344969    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7d14663-db08-4ae9-9dae-98f56d51f8a4-istiod-ca-cert\") pod \"e2e-shared-a-w4-501738-openshift-default-54c4d5d859-bwbmq\" (UID: \"f7d14663-db08-4ae9-9dae-98f56d51f8a4\") " pod="openshift-ingress/e2e-shared-a-w4-501738-openshift-default-54c4d5d859-bwbmq"
Nov 02 15:18:19.400182 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:19.390887    5304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w4-501738-openshift-default-dockercfg-8fbhz\""
Nov 02 15:18:19.464171 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:19.462214    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7d14663-db08-4ae9-9dae-98f56d51f8a4-istio-podinfo\") pod \"e2e-shared-a-w4-501738-openshift-default-54c4d5d859-bwbmq\" (UID: \"f7d14663-db08-4ae9-9dae-98f56d51f8a4\") " pod="openshift-ingress/e2e-shared-a-w4-501738-openshift-default-54c4d5d859-bwbmq"
Nov 02 15:18:19.464171 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:19.462277    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7d14663-db08-4ae9-9dae-98f56d51f8a4-istio-envoy\") pod \"e2e-shared-a-w4-501738-openshift-default-54c4d5d859-bwbmq\" (UID: \"f7d14663-db08-4ae9-9dae-98f56d51f8a4\") " pod="openshift-ingress/e2e-shared-a-w4-501738-openshift-default-54c4d5d859-bwbmq"
Nov 02 15:18:19.464171 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:19.462377    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nggqv\" (UniqueName: \"kubernetes.io/projected/f7d14663-db08-4ae9-9dae-98f56d51f8a4-kube-api-access-nggqv\") pod \"e2e-shared-a-w4-501738-openshift-default-54c4d5d859-bwbmq\" (UID: \"f7d14663-db08-4ae9-9dae-98f56d51f8a4\") " pod="openshift-ingress/e2e-shared-a-w4-501738-openshift-default-54c4d5d859-bwbmq"
Nov 02 15:18:19.464171 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:19.462477    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7d14663-db08-4ae9-9dae-98f56d51f8a4-credential-socket\") pod \"e2e-shared-a-w4-501738-openshift-default-54c4d5d859-bwbmq\" (UID: \"f7d14663-db08-4ae9-9dae-98f56d51f8a4\") " pod="openshift-ingress/e2e-shared-a-w4-501738-openshift-default-54c4d5d859-bwbmq"
Nov 02 15:18:19.464171 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:19.462505    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7d14663-db08-4ae9-9dae-98f56d51f8a4-istio-data\") pod \"e2e-shared-a-w4-501738-openshift-default-54c4d5d859-bwbmq\" (UID: \"f7d14663-db08-4ae9-9dae-98f56d51f8a4\") " pod="openshift-ingress/e2e-shared-a-w4-501738-openshift-default-54c4d5d859-bwbmq"
Nov 02 15:18:19.464171 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:19.462532    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7d14663-db08-4ae9-9dae-98f56d51f8a4-istiod-ca-cert\") pod \"e2e-shared-a-w4-501738-openshift-default-54c4d5d859-bwbmq\" (UID: \"f7d14663-db08-4ae9-9dae-98f56d51f8a4\") " pod="openshift-ingress/e2e-shared-a-w4-501738-openshift-default-54c4d5d859-bwbmq"
Nov 02 15:18:19.464171 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:19.462647    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7d14663-db08-4ae9-9dae-98f56d51f8a4-workload-socket\") pod \"e2e-shared-a-w4-501738-openshift-default-54c4d5d859-bwbmq\" (UID: \"f7d14663-db08-4ae9-9dae-98f56d51f8a4\") " pod="openshift-ingress/e2e-shared-a-w4-501738-openshift-default-54c4d5d859-bwbmq"
Nov 02 15:18:19.464171 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:19.462749    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7d14663-db08-4ae9-9dae-98f56d51f8a4-istio-token\") pod \"e2e-shared-a-w4-501738-openshift-default-54c4d5d859-bwbmq\" (UID: \"f7d14663-db08-4ae9-9dae-98f56d51f8a4\") " pod="openshift-ingress/e2e-shared-a-w4-501738-openshift-default-54c4d5d859-bwbmq"
Nov 02 15:18:19.464171 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:19.462781    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7d14663-db08-4ae9-9dae-98f56d51f8a4-workload-certs\") pod \"e2e-shared-a-w4-501738-openshift-default-54c4d5d859-bwbmq\" (UID: \"f7d14663-db08-4ae9-9dae-98f56d51f8a4\") " pod="openshift-ingress/e2e-shared-a-w4-501738-openshift-default-54c4d5d859-bwbmq"
Nov 02 15:18:19.464171 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:19.463362    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7d14663-db08-4ae9-9dae-98f56d51f8a4-workload-certs\") pod \"e2e-shared-a-w4-501738-openshift-default-54c4d5d859-bwbmq\" (UID: \"f7d14663-db08-4ae9-9dae-98f56d51f8a4\") " pod="openshift-ingress/e2e-shared-a-w4-501738-openshift-default-54c4d5d859-bwbmq"
Nov 02 15:18:19.490173 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:19.483827    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7d14663-db08-4ae9-9dae-98f56d51f8a4-istio-data\") pod \"e2e-shared-a-w4-501738-openshift-default-54c4d5d859-bwbmq\" (UID: \"f7d14663-db08-4ae9-9dae-98f56d51f8a4\") " pod="openshift-ingress/e2e-shared-a-w4-501738-openshift-default-54c4d5d859-bwbmq"
Nov 02 15:18:19.490173 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:19.487798    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7d14663-db08-4ae9-9dae-98f56d51f8a4-credential-socket\") pod \"e2e-shared-a-w4-501738-openshift-default-54c4d5d859-bwbmq\" (UID: \"f7d14663-db08-4ae9-9dae-98f56d51f8a4\") " pod="openshift-ingress/e2e-shared-a-w4-501738-openshift-default-54c4d5d859-bwbmq"
Nov 02 15:18:19.490173 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:19.488060    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7d14663-db08-4ae9-9dae-98f56d51f8a4-workload-socket\") pod \"e2e-shared-a-w4-501738-openshift-default-54c4d5d859-bwbmq\" (UID: \"f7d14663-db08-4ae9-9dae-98f56d51f8a4\") " pod="openshift-ingress/e2e-shared-a-w4-501738-openshift-default-54c4d5d859-bwbmq"
Nov 02 15:18:19.490173 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:19.488752    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7d14663-db08-4ae9-9dae-98f56d51f8a4-istiod-ca-cert\") pod \"e2e-shared-a-w4-501738-openshift-default-54c4d5d859-bwbmq\" (UID: \"f7d14663-db08-4ae9-9dae-98f56d51f8a4\") " pod="openshift-ingress/e2e-shared-a-w4-501738-openshift-default-54c4d5d859-bwbmq"
Nov 02 15:18:19.521361 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:19.520424    5304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-xtxns"
Nov 02 15:18:19.575341 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:19.528235    5304 patch_prober.go:29] interesting pod/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-xtxns container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body=
Nov 02 15:18:19.575341 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:19.528296    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-xtxns" podUID="abefd6e8-6072-45e0-9585-7c16ae3b6b35" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused"
Nov 02 15:18:19.575341 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:19.574271    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7d14663-db08-4ae9-9dae-98f56d51f8a4-istio-token\") pod \"e2e-shared-a-w4-501738-openshift-default-54c4d5d859-bwbmq\" (UID: \"f7d14663-db08-4ae9-9dae-98f56d51f8a4\") " pod="openshift-ingress/e2e-shared-a-w4-501738-openshift-default-54c4d5d859-bwbmq"
Nov 02 15:18:19.708962 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:19.703441    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nggqv\" (UniqueName: \"kubernetes.io/projected/f7d14663-db08-4ae9-9dae-98f56d51f8a4-kube-api-access-nggqv\") pod \"e2e-shared-a-w4-501738-openshift-default-54c4d5d859-bwbmq\" (UID: \"f7d14663-db08-4ae9-9dae-98f56d51f8a4\") " pod="openshift-ingress/e2e-shared-a-w4-501738-openshift-default-54c4d5d859-bwbmq"
Nov 02 15:18:19.708962 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:19.704201    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7d14663-db08-4ae9-9dae-98f56d51f8a4-istio-podinfo\") pod \"e2e-shared-a-w4-501738-openshift-default-54c4d5d859-bwbmq\" (UID: \"f7d14663-db08-4ae9-9dae-98f56d51f8a4\") " pod="openshift-ingress/e2e-shared-a-w4-501738-openshift-default-54c4d5d859-bwbmq"
Nov 02 15:18:19.708962 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:19.706771    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7d14663-db08-4ae9-9dae-98f56d51f8a4-istio-envoy\") pod \"e2e-shared-a-w4-501738-openshift-default-54c4d5d859-bwbmq\" (UID: \"f7d14663-db08-4ae9-9dae-98f56d51f8a4\") " pod="openshift-ingress/e2e-shared-a-w4-501738-openshift-default-54c4d5d859-bwbmq"
Nov 02 15:18:19.957150 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:19.956510    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-501738-openshift-default-54c4d5d859-bwbmq"
Nov 02 15:18:20.430479 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:20.430404    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-501738-openshift-default-54c4d5d859-bwbmq"]
Nov 02 15:18:20.466665 ip-10-0-135-138 kubenswrapper[5304]: W1102 15:18:20.466634    5304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7d14663_db08_4ae9_9dae_98f56d51f8a4.slice/crio-9a4b4006637ecde4d11d07bb13d66af2bca14171b007f21f8388a6a44efd38c2 WatchSource:0}: Error finding container 9a4b4006637ecde4d11d07bb13d66af2bca14171b007f21f8388a6a44efd38c2: Status 404 returned error can't find the container with id 9a4b4006637ecde4d11d07bb13d66af2bca14171b007f21f8388a6a44efd38c2
Nov 02 15:18:20.469268 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:20.469225    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:18:20.469367 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:20.469316    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:18:20.469367 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:20.469353    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:18:20.500824 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:20.500793    5304 patch_prober.go:29] interesting pod/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-xtxns container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body=
Nov 02 15:18:20.500941 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:20.500854    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-xtxns" podUID="abefd6e8-6072-45e0-9585-7c16ae3b6b35" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused"
Nov 02 15:18:21.162837 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:21.162804    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-501738-openshift-default-54c4d5d859-bwbmq" event={"ID":"f7d14663-db08-4ae9-9dae-98f56d51f8a4","Type":"ContainerStarted","Data":"26297c35ae35a96a888e684c429bbe4883e98d41daecfe97074a203d088eb6db"}
Nov 02 15:18:21.163016 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:21.162860    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-501738-openshift-default-54c4d5d859-bwbmq" event={"ID":"f7d14663-db08-4ae9-9dae-98f56d51f8a4","Type":"ContainerStarted","Data":"9a4b4006637ecde4d11d07bb13d66af2bca14171b007f21f8388a6a44efd38c2"}
Nov 02 15:18:21.221416 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:21.221358    5304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-501738-openshift-default-54c4d5d859-bwbmq" podStartSLOduration=2.221342969 podStartE2EDuration="2.221342969s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:18:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:18:21.219358626 +0000 UTC m=+1301.213183036" watchObservedRunningTime="2026-11-02 15:18:21.221342969 +0000 UTC m=+1301.215167378"
Nov 02 15:18:21.501255 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:21.501173    5304 patch_prober.go:29] interesting pod/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-xtxns container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body=
Nov 02 15:18:21.501255 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:21.501244    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-xtxns" podUID="abefd6e8-6072-45e0-9585-7c16ae3b6b35" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused"
Nov 02 15:18:21.957102 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:21.957064    5304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-501738-openshift-default-54c4d5d859-bwbmq"
Nov 02 15:18:21.962896 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:21.962870    5304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-501738-openshift-default-54c4d5d859-bwbmq"
Nov 02 15:18:22.172258 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:22.172225    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-501738-openshift-default-54c4d5d859-bwbmq"
Nov 02 15:18:22.173205 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:22.173176    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-501738-openshift-default-54c4d5d859-bwbmq"
Nov 02 15:18:22.501073 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:22.501039    5304 patch_prober.go:29] interesting pod/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-xtxns container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body=
Nov 02 15:18:22.501272 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:22.501095    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-xtxns" podUID="abefd6e8-6072-45e0-9585-7c16ae3b6b35" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused"
Nov 02 15:18:23.500776 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:23.500733    5304 patch_prober.go:29] interesting pod/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-xtxns container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body=
Nov 02 15:18:23.500961 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:23.500795    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-xtxns" podUID="abefd6e8-6072-45e0-9585-7c16ae3b6b35" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused"
Nov 02 15:18:24.500864 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:24.500823    5304 patch_prober.go:29] interesting pod/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-xtxns container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body=
Nov 02 15:18:24.501321 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:24.500882    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-xtxns" podUID="abefd6e8-6072-45e0-9585-7c16ae3b6b35" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused"
Nov 02 15:18:24.799743 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:18:24.799653    5304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k6fjk" containerName="extractor"
Nov 02 15:18:25.505982 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:25.505951    5304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-xtxns"
Nov 02 15:18:25.506461 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:25.506406    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-xtxns"
Nov 02 15:18:25.507044 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:25.507021    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-xtxns"
Nov 02 15:18:25.781508 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:25.781430    5304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-501738-767988665-nwzcr"]
Nov 02 15:18:25.785680 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:25.785657    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-501738-767988665-nwzcr"
Nov 02 15:18:25.801924 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:25.801888    5304 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w4-501738\""
Nov 02 15:18:25.802150 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:25.802066    5304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w4-501738-dockercfg-6cp75\""
Nov 02 15:18:25.804742 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:25.804715    5304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-501738-864d5f868c-2ssc8"]
Nov 02 15:18:25.808974 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:25.808946    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-501738-864d5f868c-2ssc8"
Nov 02 15:18:25.809100 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:25.809059    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-501738-767988665-nwzcr"]
Nov 02 15:18:25.831211 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:25.831182    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-501738-864d5f868c-2ssc8"]
Nov 02 15:18:25.838335 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:25.838306    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ae97c11-e62c-4022-ace2-03a398ff3854-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-501738-864d5f868c-2ssc8\" (UID: \"5ae97c11-e62c-4022-ace2-03a398ff3854\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-501738-864d5f868c-2ssc8"
Nov 02 15:18:25.838633 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:25.838608    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pr5p5\" (UniqueName: \"kubernetes.io/projected/6eab7ac5-dbb2-4b5d-adac-86e3c425c011-kube-api-access-pr5p5\") pod \"payload-processing-e2e-shared-a-w4-501738-767988665-nwzcr\" (UID: \"6eab7ac5-dbb2-4b5d-adac-86e3c425c011\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-501738-767988665-nwzcr"
Nov 02 15:18:25.838696 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:25.838669    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6eab7ac5-dbb2-4b5d-adac-86e3c425c011-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-501738-767988665-nwzcr\" (UID: \"6eab7ac5-dbb2-4b5d-adac-86e3c425c011\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-501738-767988665-nwzcr"
Nov 02 15:18:25.838860 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:25.838837    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f7snt\" (UniqueName: \"kubernetes.io/projected/5ae97c11-e62c-4022-ace2-03a398ff3854-kube-api-access-f7snt\") pod \"payload-pre-processing-e2e-shared-a-w4-501738-864d5f868c-2ssc8\" (UID: \"5ae97c11-e62c-4022-ace2-03a398ff3854\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-501738-864d5f868c-2ssc8"
Nov 02 15:18:25.944426 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:25.944374    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f7snt\" (UniqueName: \"kubernetes.io/projected/5ae97c11-e62c-4022-ace2-03a398ff3854-kube-api-access-f7snt\") pod \"payload-pre-processing-e2e-shared-a-w4-501738-864d5f868c-2ssc8\" (UID: \"5ae97c11-e62c-4022-ace2-03a398ff3854\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-501738-864d5f868c-2ssc8"
Nov 02 15:18:25.944670 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:25.944492    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ae97c11-e62c-4022-ace2-03a398ff3854-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-501738-864d5f868c-2ssc8\" (UID: \"5ae97c11-e62c-4022-ace2-03a398ff3854\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-501738-864d5f868c-2ssc8"
Nov 02 15:18:25.944842 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:25.944798    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pr5p5\" (UniqueName: \"kubernetes.io/projected/6eab7ac5-dbb2-4b5d-adac-86e3c425c011-kube-api-access-pr5p5\") pod \"payload-processing-e2e-shared-a-w4-501738-767988665-nwzcr\" (UID: \"6eab7ac5-dbb2-4b5d-adac-86e3c425c011\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-501738-767988665-nwzcr"
Nov 02 15:18:25.944936 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:25.944867    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6eab7ac5-dbb2-4b5d-adac-86e3c425c011-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-501738-767988665-nwzcr\" (UID: \"6eab7ac5-dbb2-4b5d-adac-86e3c425c011\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-501738-767988665-nwzcr"
Nov 02 15:18:25.946120 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:25.945978    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6eab7ac5-dbb2-4b5d-adac-86e3c425c011-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-501738-767988665-nwzcr\" (UID: \"6eab7ac5-dbb2-4b5d-adac-86e3c425c011\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-501738-767988665-nwzcr"
Nov 02 15:18:25.950391 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:25.950361    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ae97c11-e62c-4022-ace2-03a398ff3854-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-501738-864d5f868c-2ssc8\" (UID: \"5ae97c11-e62c-4022-ace2-03a398ff3854\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-501738-864d5f868c-2ssc8"
Nov 02 15:18:25.997871 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:25.997835    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f7snt\" (UniqueName: \"kubernetes.io/projected/5ae97c11-e62c-4022-ace2-03a398ff3854-kube-api-access-f7snt\") pod \"payload-pre-processing-e2e-shared-a-w4-501738-864d5f868c-2ssc8\" (UID: \"5ae97c11-e62c-4022-ace2-03a398ff3854\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-501738-864d5f868c-2ssc8"
Nov 02 15:18:26.006839 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:26.006808    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pr5p5\" (UniqueName: \"kubernetes.io/projected/6eab7ac5-dbb2-4b5d-adac-86e3c425c011-kube-api-access-pr5p5\") pod \"payload-processing-e2e-shared-a-w4-501738-767988665-nwzcr\" (UID: \"6eab7ac5-dbb2-4b5d-adac-86e3c425c011\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-501738-767988665-nwzcr"
Nov 02 15:18:26.101178 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:26.099415    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-501738-767988665-nwzcr"
Nov 02 15:18:26.124585 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:26.123909    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-501738-864d5f868c-2ssc8"
Nov 02 15:18:26.469453 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:26.469416    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-501738-767988665-nwzcr"]
Nov 02 15:18:26.471657 ip-10-0-135-138 kubenswrapper[5304]: W1102 15:18:26.471626    5304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6eab7ac5_dbb2_4b5d_adac_86e3c425c011.slice/crio-91e2cceeb6e844436163f10441ebafad7ef1d48d9392a83338923083eecb24d0 WatchSource:0}: Error finding container 91e2cceeb6e844436163f10441ebafad7ef1d48d9392a83338923083eecb24d0: Status 404 returned error can't find the container with id 91e2cceeb6e844436163f10441ebafad7ef1d48d9392a83338923083eecb24d0
Nov 02 15:18:26.484902 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:26.484873    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-501738-864d5f868c-2ssc8"]
Nov 02 15:18:26.489564 ip-10-0-135-138 kubenswrapper[5304]: W1102 15:18:26.489532    5304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ae97c11_e62c_4022_ace2_03a398ff3854.slice/crio-75421e895d5d49ac29e3e0ff39b3c066b7619e0815f3c1407b97e4192fe20788 WatchSource:0}: Error finding container 75421e895d5d49ac29e3e0ff39b3c066b7619e0815f3c1407b97e4192fe20788: Status 404 returned error can't find the container with id 75421e895d5d49ac29e3e0ff39b3c066b7619e0815f3c1407b97e4192fe20788
Nov 02 15:18:27.280086 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:27.280051    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-501738-864d5f868c-2ssc8" event={"ID":"5ae97c11-e62c-4022-ace2-03a398ff3854","Type":"ContainerStarted","Data":"4c4d983a44d0823251a3d5f5eb2c8b42b6389365b1627bf268c2d393ebcac1f7"}
Nov 02 15:18:27.280582 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:27.280113    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-501738-864d5f868c-2ssc8" event={"ID":"5ae97c11-e62c-4022-ace2-03a398ff3854","Type":"ContainerStarted","Data":"75421e895d5d49ac29e3e0ff39b3c066b7619e0815f3c1407b97e4192fe20788"}
Nov 02 15:18:27.280582 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:27.280267    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-501738-864d5f868c-2ssc8"
Nov 02 15:18:27.281849 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:27.281818    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-501738-767988665-nwzcr" event={"ID":"6eab7ac5-dbb2-4b5d-adac-86e3c425c011","Type":"ContainerStarted","Data":"b4997936ebf8111dd7fcd710eee1fdb7909ab32fd4aeb56f3760f614f25f6616"}
Nov 02 15:18:27.281999 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:27.281869    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-501738-767988665-nwzcr" event={"ID":"6eab7ac5-dbb2-4b5d-adac-86e3c425c011","Type":"ContainerStarted","Data":"91e2cceeb6e844436163f10441ebafad7ef1d48d9392a83338923083eecb24d0"}
Nov 02 15:18:27.281999 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:27.281964    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-501738-767988665-nwzcr"
Nov 02 15:18:27.328418 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:27.328356    5304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-501738-864d5f868c-2ssc8" podStartSLOduration=2.328341542 podStartE2EDuration="2.328341542s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:18:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:18:27.32579342 +0000 UTC m=+1307.319617826" watchObservedRunningTime="2026-11-02 15:18:27.328341542 +0000 UTC m=+1307.322165951"
Nov 02 15:18:27.375019 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:27.374960    5304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-501738-767988665-nwzcr" podStartSLOduration=2.374941002 podStartE2EDuration="2.374941002s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:18:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:18:27.374464849 +0000 UTC m=+1307.368289257" watchObservedRunningTime="2026-11-02 15:18:27.374941002 +0000 UTC m=+1307.368765413"
Nov 02 15:18:31.066822 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:31.066787    5304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-5f9166-769d4fdb6c-529mc"]
Nov 02 15:18:31.071342 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:31.071313    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5f9166-769d4fdb6c-529mc"
Nov 02 15:18:31.099307 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:31.099281    5304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-5f9166-serving-cert\""
Nov 02 15:18:31.121782 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:31.121746    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-5f9166-769d4fdb6c-529mc"]
Nov 02 15:18:31.143618 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:31.143580    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fe8598be-5e22-47f6-bf0b-3a279ea72ebb-maas-api-tls\") pod \"maas-api-e2e-api-b-5f9166-769d4fdb6c-529mc\" (UID: \"fe8598be-5e22-47f6-bf0b-3a279ea72ebb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5f9166-769d4fdb6c-529mc"
Nov 02 15:18:31.143820 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:31.143710    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe8598be-5e22-47f6-bf0b-3a279ea72ebb-openshift-service-ca\") pod \"maas-api-e2e-api-b-5f9166-769d4fdb6c-529mc\" (UID: \"fe8598be-5e22-47f6-bf0b-3a279ea72ebb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5f9166-769d4fdb6c-529mc"
Nov 02 15:18:31.143820 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:31.143805    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4jrkv\" (UniqueName: \"kubernetes.io/projected/fe8598be-5e22-47f6-bf0b-3a279ea72ebb-kube-api-access-4jrkv\") pod \"maas-api-e2e-api-b-5f9166-769d4fdb6c-529mc\" (UID: \"fe8598be-5e22-47f6-bf0b-3a279ea72ebb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5f9166-769d4fdb6c-529mc"
Nov 02 15:18:31.143955 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:31.143860    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fe8598be-5e22-47f6-bf0b-3a279ea72ebb-metrics-certs\") pod \"maas-api-e2e-api-b-5f9166-769d4fdb6c-529mc\" (UID: \"fe8598be-5e22-47f6-bf0b-3a279ea72ebb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5f9166-769d4fdb6c-529mc"
Nov 02 15:18:31.245459 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:31.245416    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe8598be-5e22-47f6-bf0b-3a279ea72ebb-openshift-service-ca\") pod \"maas-api-e2e-api-b-5f9166-769d4fdb6c-529mc\" (UID: \"fe8598be-5e22-47f6-bf0b-3a279ea72ebb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5f9166-769d4fdb6c-529mc"
Nov 02 15:18:31.245964 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:31.245937    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4jrkv\" (UniqueName: \"kubernetes.io/projected/fe8598be-5e22-47f6-bf0b-3a279ea72ebb-kube-api-access-4jrkv\") pod \"maas-api-e2e-api-b-5f9166-769d4fdb6c-529mc\" (UID: \"fe8598be-5e22-47f6-bf0b-3a279ea72ebb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5f9166-769d4fdb6c-529mc"
Nov 02 15:18:31.246253 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:31.246233    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fe8598be-5e22-47f6-bf0b-3a279ea72ebb-metrics-certs\") pod \"maas-api-e2e-api-b-5f9166-769d4fdb6c-529mc\" (UID: \"fe8598be-5e22-47f6-bf0b-3a279ea72ebb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5f9166-769d4fdb6c-529mc"
Nov 02 15:18:31.246560 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:31.246538    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fe8598be-5e22-47f6-bf0b-3a279ea72ebb-maas-api-tls\") pod \"maas-api-e2e-api-b-5f9166-769d4fdb6c-529mc\" (UID: \"fe8598be-5e22-47f6-bf0b-3a279ea72ebb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5f9166-769d4fdb6c-529mc"
Nov 02 15:18:31.255469 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:31.249618    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe8598be-5e22-47f6-bf0b-3a279ea72ebb-openshift-service-ca\") pod \"maas-api-e2e-api-b-5f9166-769d4fdb6c-529mc\" (UID: \"fe8598be-5e22-47f6-bf0b-3a279ea72ebb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5f9166-769d4fdb6c-529mc"
Nov 02 15:18:31.255469 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:18:31.251206    5304 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-5f9166-serving-cert: secret "maas-api-e2e-api-b-5f9166-serving-cert" not found
Nov 02 15:18:31.255469 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:18:31.251332    5304 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fe8598be-5e22-47f6-bf0b-3a279ea72ebb-maas-api-tls podName:fe8598be-5e22-47f6-bf0b-3a279ea72ebb nodeName:}" failed. No retries permitted until 2026-11-02 15:18:31.751310365 +0000 UTC m=+1311.745134768 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/fe8598be-5e22-47f6-bf0b-3a279ea72ebb-maas-api-tls") pod "maas-api-e2e-api-b-5f9166-769d4fdb6c-529mc" (UID: "fe8598be-5e22-47f6-bf0b-3a279ea72ebb") : secret "maas-api-e2e-api-b-5f9166-serving-cert" not found
Nov 02 15:18:31.256059 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:31.255940    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fe8598be-5e22-47f6-bf0b-3a279ea72ebb-metrics-certs\") pod \"maas-api-e2e-api-b-5f9166-769d4fdb6c-529mc\" (UID: \"fe8598be-5e22-47f6-bf0b-3a279ea72ebb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5f9166-769d4fdb6c-529mc"
Nov 02 15:18:31.276816 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:31.276749    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4jrkv\" (UniqueName: \"kubernetes.io/projected/fe8598be-5e22-47f6-bf0b-3a279ea72ebb-kube-api-access-4jrkv\") pod \"maas-api-e2e-api-b-5f9166-769d4fdb6c-529mc\" (UID: \"fe8598be-5e22-47f6-bf0b-3a279ea72ebb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5f9166-769d4fdb6c-529mc"
Nov 02 15:18:31.754670 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:31.754639    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fe8598be-5e22-47f6-bf0b-3a279ea72ebb-maas-api-tls\") pod \"maas-api-e2e-api-b-5f9166-769d4fdb6c-529mc\" (UID: \"fe8598be-5e22-47f6-bf0b-3a279ea72ebb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5f9166-769d4fdb6c-529mc"
Nov 02 15:18:31.757541 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:31.757514    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fe8598be-5e22-47f6-bf0b-3a279ea72ebb-maas-api-tls\") pod \"maas-api-e2e-api-b-5f9166-769d4fdb6c-529mc\" (UID: \"fe8598be-5e22-47f6-bf0b-3a279ea72ebb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5f9166-769d4fdb6c-529mc"
Nov 02 15:18:31.986306 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:31.986245    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5f9166-769d4fdb6c-529mc"
Nov 02 15:18:32.397309 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:32.397276    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-5f9166-769d4fdb6c-529mc"]
Nov 02 15:18:32.425896 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:32.425868    5304 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 02 15:18:33.392072 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:33.392030    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5f9166-769d4fdb6c-529mc" event={"ID":"fe8598be-5e22-47f6-bf0b-3a279ea72ebb","Type":"ContainerStarted","Data":"f54f927b07c6bbe2e19ea5ceeb88bccf197ad723c41d6a6e5f3a6a4268e91439"}
Nov 02 15:18:33.392327 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:33.392098    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5f9166-769d4fdb6c-529mc" event={"ID":"fe8598be-5e22-47f6-bf0b-3a279ea72ebb","Type":"ContainerStarted","Data":"fcd6c62212bd7b14cb16e711195147eed256e2bf69a8d5b733e9f44289114227"}
Nov 02 15:18:33.392327 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:33.392157    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5f9166-769d4fdb6c-529mc"
Nov 02 15:18:33.443648 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:33.443581    5304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5f9166-769d4fdb6c-529mc" podStartSLOduration=2.048262934 podStartE2EDuration="2.44356677s" totalImagesPullingTime="395.303836ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:18:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:18:33.438680397 +0000 UTC m=+1313.432504808" watchObservedRunningTime="2026-11-02 15:18:33.44356677 +0000 UTC m=+1313.437391179"
Nov 02 15:18:34.751933 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:34.751900    5304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-a713aa-openshift-default-7554947cc9-trs9t"]
Nov 02 15:18:34.756769 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:34.756734    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-a713aa-openshift-default-7554947cc9-trs9t"
Nov 02 15:18:34.787185 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:34.781509    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-a713aa-openshift-default-7554947cc9-trs9t"]
Nov 02 15:18:34.788233 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:34.788210    5304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-a713aa-openshift-default-dockercfg-vrt2d\""
Nov 02 15:18:34.809143 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:34.809072    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b33eaba-53ea-47b6-bdfe-3d3d9ffd8a36-workload-certs\") pod \"e2e-inf-a-a713aa-openshift-default-7554947cc9-trs9t\" (UID: \"9b33eaba-53ea-47b6-bdfe-3d3d9ffd8a36\") " pod="openshift-ingress/e2e-inf-a-a713aa-openshift-default-7554947cc9-trs9t"
Nov 02 15:18:34.809308 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:34.809231    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b33eaba-53ea-47b6-bdfe-3d3d9ffd8a36-istio-envoy\") pod \"e2e-inf-a-a713aa-openshift-default-7554947cc9-trs9t\" (UID: \"9b33eaba-53ea-47b6-bdfe-3d3d9ffd8a36\") " pod="openshift-ingress/e2e-inf-a-a713aa-openshift-default-7554947cc9-trs9t"
Nov 02 15:18:34.809411 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:34.809392    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b33eaba-53ea-47b6-bdfe-3d3d9ffd8a36-istio-podinfo\") pod \"e2e-inf-a-a713aa-openshift-default-7554947cc9-trs9t\" (UID: \"9b33eaba-53ea-47b6-bdfe-3d3d9ffd8a36\") " pod="openshift-ingress/e2e-inf-a-a713aa-openshift-default-7554947cc9-trs9t"
Nov 02 15:18:34.809514 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:34.809488    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fttkn\" (UniqueName: \"kubernetes.io/projected/9b33eaba-53ea-47b6-bdfe-3d3d9ffd8a36-kube-api-access-fttkn\") pod \"e2e-inf-a-a713aa-openshift-default-7554947cc9-trs9t\" (UID: \"9b33eaba-53ea-47b6-bdfe-3d3d9ffd8a36\") " pod="openshift-ingress/e2e-inf-a-a713aa-openshift-default-7554947cc9-trs9t"
Nov 02 15:18:34.809677 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:34.809658    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b33eaba-53ea-47b6-bdfe-3d3d9ffd8a36-workload-socket\") pod \"e2e-inf-a-a713aa-openshift-default-7554947cc9-trs9t\" (UID: \"9b33eaba-53ea-47b6-bdfe-3d3d9ffd8a36\") " pod="openshift-ingress/e2e-inf-a-a713aa-openshift-default-7554947cc9-trs9t"
Nov 02 15:18:34.809778 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:34.809754    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b33eaba-53ea-47b6-bdfe-3d3d9ffd8a36-istio-data\") pod \"e2e-inf-a-a713aa-openshift-default-7554947cc9-trs9t\" (UID: \"9b33eaba-53ea-47b6-bdfe-3d3d9ffd8a36\") " pod="openshift-ingress/e2e-inf-a-a713aa-openshift-default-7554947cc9-trs9t"
Nov 02 15:18:34.809831 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:34.809815    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b33eaba-53ea-47b6-bdfe-3d3d9ffd8a36-istio-token\") pod \"e2e-inf-a-a713aa-openshift-default-7554947cc9-trs9t\" (UID: \"9b33eaba-53ea-47b6-bdfe-3d3d9ffd8a36\") " pod="openshift-ingress/e2e-inf-a-a713aa-openshift-default-7554947cc9-trs9t"
Nov 02 15:18:34.809908 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:34.809889    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b33eaba-53ea-47b6-bdfe-3d3d9ffd8a36-credential-socket\") pod \"e2e-inf-a-a713aa-openshift-default-7554947cc9-trs9t\" (UID: \"9b33eaba-53ea-47b6-bdfe-3d3d9ffd8a36\") " pod="openshift-ingress/e2e-inf-a-a713aa-openshift-default-7554947cc9-trs9t"
Nov 02 15:18:34.809956 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:34.809942    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b33eaba-53ea-47b6-bdfe-3d3d9ffd8a36-istiod-ca-cert\") pod \"e2e-inf-a-a713aa-openshift-default-7554947cc9-trs9t\" (UID: \"9b33eaba-53ea-47b6-bdfe-3d3d9ffd8a36\") " pod="openshift-ingress/e2e-inf-a-a713aa-openshift-default-7554947cc9-trs9t"
Nov 02 15:18:34.911910 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:34.911871    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b33eaba-53ea-47b6-bdfe-3d3d9ffd8a36-istio-podinfo\") pod \"e2e-inf-a-a713aa-openshift-default-7554947cc9-trs9t\" (UID: \"9b33eaba-53ea-47b6-bdfe-3d3d9ffd8a36\") " pod="openshift-ingress/e2e-inf-a-a713aa-openshift-default-7554947cc9-trs9t"
Nov 02 15:18:34.912165 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:34.911951    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fttkn\" (UniqueName: \"kubernetes.io/projected/9b33eaba-53ea-47b6-bdfe-3d3d9ffd8a36-kube-api-access-fttkn\") pod \"e2e-inf-a-a713aa-openshift-default-7554947cc9-trs9t\" (UID: \"9b33eaba-53ea-47b6-bdfe-3d3d9ffd8a36\") " pod="openshift-ingress/e2e-inf-a-a713aa-openshift-default-7554947cc9-trs9t"
Nov 02 15:18:34.912165 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:34.912032    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b33eaba-53ea-47b6-bdfe-3d3d9ffd8a36-workload-socket\") pod \"e2e-inf-a-a713aa-openshift-default-7554947cc9-trs9t\" (UID: \"9b33eaba-53ea-47b6-bdfe-3d3d9ffd8a36\") " pod="openshift-ingress/e2e-inf-a-a713aa-openshift-default-7554947cc9-trs9t"
Nov 02 15:18:34.912165 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:34.912069    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b33eaba-53ea-47b6-bdfe-3d3d9ffd8a36-istio-data\") pod \"e2e-inf-a-a713aa-openshift-default-7554947cc9-trs9t\" (UID: \"9b33eaba-53ea-47b6-bdfe-3d3d9ffd8a36\") " pod="openshift-ingress/e2e-inf-a-a713aa-openshift-default-7554947cc9-trs9t"
Nov 02 15:18:34.912387 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:34.912177    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b33eaba-53ea-47b6-bdfe-3d3d9ffd8a36-istio-token\") pod \"e2e-inf-a-a713aa-openshift-default-7554947cc9-trs9t\" (UID: \"9b33eaba-53ea-47b6-bdfe-3d3d9ffd8a36\") " pod="openshift-ingress/e2e-inf-a-a713aa-openshift-default-7554947cc9-trs9t"
Nov 02 15:18:34.912387 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:34.912281    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b33eaba-53ea-47b6-bdfe-3d3d9ffd8a36-credential-socket\") pod \"e2e-inf-a-a713aa-openshift-default-7554947cc9-trs9t\" (UID: \"9b33eaba-53ea-47b6-bdfe-3d3d9ffd8a36\") " pod="openshift-ingress/e2e-inf-a-a713aa-openshift-default-7554947cc9-trs9t"
Nov 02 15:18:34.912387 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:34.912331    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b33eaba-53ea-47b6-bdfe-3d3d9ffd8a36-istiod-ca-cert\") pod \"e2e-inf-a-a713aa-openshift-default-7554947cc9-trs9t\" (UID: \"9b33eaba-53ea-47b6-bdfe-3d3d9ffd8a36\") " pod="openshift-ingress/e2e-inf-a-a713aa-openshift-default-7554947cc9-trs9t"
Nov 02 15:18:34.912561 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:34.912497    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b33eaba-53ea-47b6-bdfe-3d3d9ffd8a36-workload-certs\") pod \"e2e-inf-a-a713aa-openshift-default-7554947cc9-trs9t\" (UID: \"9b33eaba-53ea-47b6-bdfe-3d3d9ffd8a36\") " pod="openshift-ingress/e2e-inf-a-a713aa-openshift-default-7554947cc9-trs9t"
Nov 02 15:18:34.912561 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:34.912540    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b33eaba-53ea-47b6-bdfe-3d3d9ffd8a36-istio-envoy\") pod \"e2e-inf-a-a713aa-openshift-default-7554947cc9-trs9t\" (UID: \"9b33eaba-53ea-47b6-bdfe-3d3d9ffd8a36\") " pod="openshift-ingress/e2e-inf-a-a713aa-openshift-default-7554947cc9-trs9t"
Nov 02 15:18:34.912823 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:34.912798    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b33eaba-53ea-47b6-bdfe-3d3d9ffd8a36-credential-socket\") pod \"e2e-inf-a-a713aa-openshift-default-7554947cc9-trs9t\" (UID: \"9b33eaba-53ea-47b6-bdfe-3d3d9ffd8a36\") " pod="openshift-ingress/e2e-inf-a-a713aa-openshift-default-7554947cc9-trs9t"
Nov 02 15:18:34.913043 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:34.912868    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b33eaba-53ea-47b6-bdfe-3d3d9ffd8a36-istio-data\") pod \"e2e-inf-a-a713aa-openshift-default-7554947cc9-trs9t\" (UID: \"9b33eaba-53ea-47b6-bdfe-3d3d9ffd8a36\") " pod="openshift-ingress/e2e-inf-a-a713aa-openshift-default-7554947cc9-trs9t"
Nov 02 15:18:34.913333 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:34.913167    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b33eaba-53ea-47b6-bdfe-3d3d9ffd8a36-workload-socket\") pod \"e2e-inf-a-a713aa-openshift-default-7554947cc9-trs9t\" (UID: \"9b33eaba-53ea-47b6-bdfe-3d3d9ffd8a36\") " pod="openshift-ingress/e2e-inf-a-a713aa-openshift-default-7554947cc9-trs9t"
Nov 02 15:18:34.913445 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:34.913423    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b33eaba-53ea-47b6-bdfe-3d3d9ffd8a36-istiod-ca-cert\") pod \"e2e-inf-a-a713aa-openshift-default-7554947cc9-trs9t\" (UID: \"9b33eaba-53ea-47b6-bdfe-3d3d9ffd8a36\") " pod="openshift-ingress/e2e-inf-a-a713aa-openshift-default-7554947cc9-trs9t"
Nov 02 15:18:34.913675 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:34.913652    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b33eaba-53ea-47b6-bdfe-3d3d9ffd8a36-workload-certs\") pod \"e2e-inf-a-a713aa-openshift-default-7554947cc9-trs9t\" (UID: \"9b33eaba-53ea-47b6-bdfe-3d3d9ffd8a36\") " pod="openshift-ingress/e2e-inf-a-a713aa-openshift-default-7554947cc9-trs9t"
Nov 02 15:18:34.914979 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:34.914953    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b33eaba-53ea-47b6-bdfe-3d3d9ffd8a36-istio-podinfo\") pod \"e2e-inf-a-a713aa-openshift-default-7554947cc9-trs9t\" (UID: \"9b33eaba-53ea-47b6-bdfe-3d3d9ffd8a36\") " pod="openshift-ingress/e2e-inf-a-a713aa-openshift-default-7554947cc9-trs9t"
Nov 02 15:18:34.915485 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:34.915461    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b33eaba-53ea-47b6-bdfe-3d3d9ffd8a36-istio-envoy\") pod \"e2e-inf-a-a713aa-openshift-default-7554947cc9-trs9t\" (UID: \"9b33eaba-53ea-47b6-bdfe-3d3d9ffd8a36\") " pod="openshift-ingress/e2e-inf-a-a713aa-openshift-default-7554947cc9-trs9t"
Nov 02 15:18:34.937527 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:34.937503    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b33eaba-53ea-47b6-bdfe-3d3d9ffd8a36-istio-token\") pod \"e2e-inf-a-a713aa-openshift-default-7554947cc9-trs9t\" (UID: \"9b33eaba-53ea-47b6-bdfe-3d3d9ffd8a36\") " pod="openshift-ingress/e2e-inf-a-a713aa-openshift-default-7554947cc9-trs9t"
Nov 02 15:18:34.937735 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:34.937709    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fttkn\" (UniqueName: \"kubernetes.io/projected/9b33eaba-53ea-47b6-bdfe-3d3d9ffd8a36-kube-api-access-fttkn\") pod \"e2e-inf-a-a713aa-openshift-default-7554947cc9-trs9t\" (UID: \"9b33eaba-53ea-47b6-bdfe-3d3d9ffd8a36\") " pod="openshift-ingress/e2e-inf-a-a713aa-openshift-default-7554947cc9-trs9t"
Nov 02 15:18:35.111479 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:35.111438    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-a713aa-openshift-default-7554947cc9-trs9t"
Nov 02 15:18:35.505951 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:35.505924    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-a713aa-openshift-default-7554947cc9-trs9t"]
Nov 02 15:18:35.507855 ip-10-0-135-138 kubenswrapper[5304]: W1102 15:18:35.507825    5304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b33eaba_53ea_47b6_bdfe_3d3d9ffd8a36.slice/crio-e7a44dcb202db0de44db94d4b90f50351ba9bffbbb9b461beb4c6bdb5405831f WatchSource:0}: Error finding container e7a44dcb202db0de44db94d4b90f50351ba9bffbbb9b461beb4c6bdb5405831f: Status 404 returned error can't find the container with id e7a44dcb202db0de44db94d4b90f50351ba9bffbbb9b461beb4c6bdb5405831f
Nov 02 15:18:35.510802 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:35.510767    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:18:35.510870 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:35.510834    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:18:35.510870 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:35.510865    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:18:36.415961 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:36.415924    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a713aa-openshift-default-7554947cc9-trs9t" event={"ID":"9b33eaba-53ea-47b6-bdfe-3d3d9ffd8a36","Type":"ContainerStarted","Data":"f48f93ebb35e8494ee80b0db35e6df19cf522c1d5e7a8601b57ef28149a900bb"}
Nov 02 15:18:36.416537 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:36.415997    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a713aa-openshift-default-7554947cc9-trs9t" event={"ID":"9b33eaba-53ea-47b6-bdfe-3d3d9ffd8a36","Type":"ContainerStarted","Data":"e7a44dcb202db0de44db94d4b90f50351ba9bffbbb9b461beb4c6bdb5405831f"}
Nov 02 15:18:36.484085 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:36.484027    5304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-a713aa-openshift-default-7554947cc9-trs9t" podStartSLOduration=2.484011543 podStartE2EDuration="2.484011543s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:18:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:18:36.473386875 +0000 UTC m=+1316.467211288" watchObservedRunningTime="2026-11-02 15:18:36.484011543 +0000 UTC m=+1316.477835952"
Nov 02 15:18:37.112288 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:37.112252    5304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-a713aa-openshift-default-7554947cc9-trs9t"
Nov 02 15:18:37.118543 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:37.118517    5304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-a713aa-openshift-default-7554947cc9-trs9t"
Nov 02 15:18:37.422681 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:37.422597    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-a713aa-openshift-default-7554947cc9-trs9t"
Nov 02 15:18:37.423655 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:37.423635    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-a713aa-openshift-default-7554947cc9-trs9t"
Nov 02 15:18:38.293558 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:38.293517    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-501738-767988665-nwzcr"
Nov 02 15:18:38.293737 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:38.293613    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-501738-864d5f868c-2ssc8"
Nov 02 15:18:39.409392 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:39.409359    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5f9166-769d4fdb6c-529mc"
Nov 02 15:18:41.328025 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:41.327784    5304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-a713aa-6cd9667b9-dk2qh"]
Nov 02 15:18:41.364228 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:41.345465    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-a713aa-6cd9667b9-dk2qh"]
Nov 02 15:18:41.364228 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:41.345622    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-a713aa-6cd9667b9-dk2qh"
Nov 02 15:18:41.364228 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:41.363684    5304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-a713aa-dockercfg-d5cws\""
Nov 02 15:18:41.385119 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:41.382816    5304 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-a713aa\""
Nov 02 15:18:41.397149 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:41.388088    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/864c95a0-3f74-41bc-8d00-2dd60f1d5cdc-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-a713aa-6cd9667b9-dk2qh\" (UID: \"864c95a0-3f74-41bc-8d00-2dd60f1d5cdc\") " pod="openshift-ingress/payload-processing-e2e-inf-a-a713aa-6cd9667b9-dk2qh"
Nov 02 15:18:41.397149 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:41.388152    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-szc9m\" (UniqueName: \"kubernetes.io/projected/864c95a0-3f74-41bc-8d00-2dd60f1d5cdc-kube-api-access-szc9m\") pod \"payload-processing-e2e-inf-a-a713aa-6cd9667b9-dk2qh\" (UID: \"864c95a0-3f74-41bc-8d00-2dd60f1d5cdc\") " pod="openshift-ingress/payload-processing-e2e-inf-a-a713aa-6cd9667b9-dk2qh"
Nov 02 15:18:41.489713 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:41.489680    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/864c95a0-3f74-41bc-8d00-2dd60f1d5cdc-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-a713aa-6cd9667b9-dk2qh\" (UID: \"864c95a0-3f74-41bc-8d00-2dd60f1d5cdc\") " pod="openshift-ingress/payload-processing-e2e-inf-a-a713aa-6cd9667b9-dk2qh"
Nov 02 15:18:41.489903 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:41.489731    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-szc9m\" (UniqueName: \"kubernetes.io/projected/864c95a0-3f74-41bc-8d00-2dd60f1d5cdc-kube-api-access-szc9m\") pod \"payload-processing-e2e-inf-a-a713aa-6cd9667b9-dk2qh\" (UID: \"864c95a0-3f74-41bc-8d00-2dd60f1d5cdc\") " pod="openshift-ingress/payload-processing-e2e-inf-a-a713aa-6cd9667b9-dk2qh"
Nov 02 15:18:41.490428 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:41.490407    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/864c95a0-3f74-41bc-8d00-2dd60f1d5cdc-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-a713aa-6cd9667b9-dk2qh\" (UID: \"864c95a0-3f74-41bc-8d00-2dd60f1d5cdc\") " pod="openshift-ingress/payload-processing-e2e-inf-a-a713aa-6cd9667b9-dk2qh"
Nov 02 15:18:41.513262 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:41.513232    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-szc9m\" (UniqueName: \"kubernetes.io/projected/864c95a0-3f74-41bc-8d00-2dd60f1d5cdc-kube-api-access-szc9m\") pod \"payload-processing-e2e-inf-a-a713aa-6cd9667b9-dk2qh\" (UID: \"864c95a0-3f74-41bc-8d00-2dd60f1d5cdc\") " pod="openshift-ingress/payload-processing-e2e-inf-a-a713aa-6cd9667b9-dk2qh"
Nov 02 15:18:41.674432 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:41.674338    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-a713aa-6cd9667b9-dk2qh"
Nov 02 15:18:41.987796 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:41.987765    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-a713aa-6cd9667b9-dk2qh"]
Nov 02 15:18:41.989790 ip-10-0-135-138 kubenswrapper[5304]: W1102 15:18:41.989761    5304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod864c95a0_3f74_41bc_8d00_2dd60f1d5cdc.slice/crio-b1f65333cf2d4fc7bdf5d423ab16455988dd6264ebe913c83ddb787b9881d1a7 WatchSource:0}: Error finding container b1f65333cf2d4fc7bdf5d423ab16455988dd6264ebe913c83ddb787b9881d1a7: Status 404 returned error can't find the container with id b1f65333cf2d4fc7bdf5d423ab16455988dd6264ebe913c83ddb787b9881d1a7
Nov 02 15:18:42.723604 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:42.723409    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-a713aa-6cd9667b9-dk2qh" event={"ID":"864c95a0-3f74-41bc-8d00-2dd60f1d5cdc","Type":"ContainerStarted","Data":"9b27d8ffdacf02d5ca788b9ab3d2a7a55cbf3e203851d0eca1a9694aca6816b6"}
Nov 02 15:18:42.723604 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:42.723501    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-a713aa-6cd9667b9-dk2qh" event={"ID":"864c95a0-3f74-41bc-8d00-2dd60f1d5cdc","Type":"ContainerStarted","Data":"b1f65333cf2d4fc7bdf5d423ab16455988dd6264ebe913c83ddb787b9881d1a7"}
Nov 02 15:18:42.724299 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:42.724055    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-a713aa-6cd9667b9-dk2qh"
Nov 02 15:18:42.776546 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:42.776493    5304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-a713aa-6cd9667b9-dk2qh" podStartSLOduration=1.7764763540000001 podStartE2EDuration="1.776476354s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:18:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:18:42.77345525 +0000 UTC m=+1322.767279656" watchObservedRunningTime="2026-11-02 15:18:42.776476354 +0000 UTC m=+1322.770300767"
Nov 02 15:18:53.735410 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:53.735374    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-a713aa-6cd9667b9-dk2qh"
Nov 02 15:18:54.800852 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:18:54.800808    5304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k6fjk" containerName="extractor"
Nov 02 15:18:55.391706 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:55.391669    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-25flx"]
Nov 02 15:18:55.391989 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:55.391943    5304 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-25flx" podUID="6f80b62a-49ba-4922-8f3e-804d6bdcc46f" containerName="istio-proxy" containerID="cri-o://bb9feef4520f31371456ece67144a6dd30fc5fb819a8f39c85e7bb9b83206635" gracePeriod=30
Nov 02 15:18:55.404348 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:55.404275    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-9gswz"]
Nov 02 15:18:55.404669 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:55.404617    5304 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-9gswz" podUID="45f29144-431b-4948-abfa-a1fc5a6847ef" containerName="istio-proxy" containerID="cri-o://24ded508397143b4d07251767075b3096cfcb6d5ce566922551c62edbdb45520" gracePeriod=30
Nov 02 15:18:55.643637 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:55.643554    5304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-501738-b94d75b7d-s94zq"]
Nov 02 15:18:55.651233 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:55.651189    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-501738-b94d75b7d-s94zq"
Nov 02 15:18:55.657796 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:55.657768    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-501738-b94d75b7d-s94zq"]
Nov 02 15:18:55.670932 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:55.670909    5304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w4-501738-serving-cert\""
Nov 02 15:18:55.739023 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:55.738991    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-lphrz"]
Nov 02 15:18:55.739246 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:55.739222    5304 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-lphrz" podUID="474f879d-3a52-4e47-8eb0-95a4fcfbe399" containerName="istio-proxy" containerID="cri-o://43cb8a845d15be46fe6bb789d7bb2ed4ae2799cb345e07749961896e872f4875" gracePeriod=30
Nov 02 15:18:55.779556 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:55.779522    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c55a8850-8fb6-4059-851e-47519d695e04-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-501738-b94d75b7d-s94zq\" (UID: \"c55a8850-8fb6-4059-851e-47519d695e04\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-501738-b94d75b7d-s94zq"
Nov 02 15:18:55.779744 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:55.779712    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c55a8850-8fb6-4059-851e-47519d695e04-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-501738-b94d75b7d-s94zq\" (UID: \"c55a8850-8fb6-4059-851e-47519d695e04\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-501738-b94d75b7d-s94zq"
Nov 02 15:18:55.780326 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:55.779794    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c55a8850-8fb6-4059-851e-47519d695e04-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-501738-b94d75b7d-s94zq\" (UID: \"c55a8850-8fb6-4059-851e-47519d695e04\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-501738-b94d75b7d-s94zq"
Nov 02 15:18:55.780326 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:55.779842    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4gqjc\" (UniqueName: \"kubernetes.io/projected/c55a8850-8fb6-4059-851e-47519d695e04-kube-api-access-4gqjc\") pod \"maas-api-e2e-shared-a-w4-501738-b94d75b7d-s94zq\" (UID: \"c55a8850-8fb6-4059-851e-47519d695e04\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-501738-b94d75b7d-s94zq"
Nov 02 15:18:55.893154 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:55.888146    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4gqjc\" (UniqueName: \"kubernetes.io/projected/c55a8850-8fb6-4059-851e-47519d695e04-kube-api-access-4gqjc\") pod \"maas-api-e2e-shared-a-w4-501738-b94d75b7d-s94zq\" (UID: \"c55a8850-8fb6-4059-851e-47519d695e04\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-501738-b94d75b7d-s94zq"
Nov 02 15:18:55.893154 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:55.888771    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c55a8850-8fb6-4059-851e-47519d695e04-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-501738-b94d75b7d-s94zq\" (UID: \"c55a8850-8fb6-4059-851e-47519d695e04\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-501738-b94d75b7d-s94zq"
Nov 02 15:18:55.893154 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:55.888967    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c55a8850-8fb6-4059-851e-47519d695e04-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-501738-b94d75b7d-s94zq\" (UID: \"c55a8850-8fb6-4059-851e-47519d695e04\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-501738-b94d75b7d-s94zq"
Nov 02 15:18:55.893154 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:55.889053    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c55a8850-8fb6-4059-851e-47519d695e04-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-501738-b94d75b7d-s94zq\" (UID: \"c55a8850-8fb6-4059-851e-47519d695e04\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-501738-b94d75b7d-s94zq"
Nov 02 15:18:55.893154 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:55.891289    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c55a8850-8fb6-4059-851e-47519d695e04-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-501738-b94d75b7d-s94zq\" (UID: \"c55a8850-8fb6-4059-851e-47519d695e04\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-501738-b94d75b7d-s94zq"
Nov 02 15:18:55.925202 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:55.918504    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c55a8850-8fb6-4059-851e-47519d695e04-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-501738-b94d75b7d-s94zq\" (UID: \"c55a8850-8fb6-4059-851e-47519d695e04\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-501738-b94d75b7d-s94zq"
Nov 02 15:18:55.929301 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:55.927772    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c55a8850-8fb6-4059-851e-47519d695e04-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-501738-b94d75b7d-s94zq\" (UID: \"c55a8850-8fb6-4059-851e-47519d695e04\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-501738-b94d75b7d-s94zq"
Nov 02 15:18:55.933454 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:55.930711    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4gqjc\" (UniqueName: \"kubernetes.io/projected/c55a8850-8fb6-4059-851e-47519d695e04-kube-api-access-4gqjc\") pod \"maas-api-e2e-shared-a-w4-501738-b94d75b7d-s94zq\" (UID: \"c55a8850-8fb6-4059-851e-47519d695e04\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-501738-b94d75b7d-s94zq"
Nov 02 15:18:55.971392 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:55.971357    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-501738-b94d75b7d-s94zq"
Nov 02 15:18:56.159388 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:56.159355    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-xcfbs"]
Nov 02 15:18:56.159607 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:56.159583    5304 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-xcfbs" podUID="a395ec29-6257-43f1-a2f0-1ec07bf215ad" containerName="istio-proxy" containerID="cri-o://a075c313dd476af7b38cb47ac46b708c29575c832eae7c3eca5e503e0f1927ee" gracePeriod=30
Nov 02 15:18:56.312483 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:56.312450    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-501738-b94d75b7d-s94zq"]
Nov 02 15:18:56.313774 ip-10-0-135-138 kubenswrapper[5304]: W1102 15:18:56.313749    5304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc55a8850_8fb6_4059_851e_47519d695e04.slice/crio-335cc3e3c0010d97c80ac89f76d433c9d3ec2cd240838736de3b5d30c89b7743 WatchSource:0}: Error finding container 335cc3e3c0010d97c80ac89f76d433c9d3ec2cd240838736de3b5d30c89b7743: Status 404 returned error can't find the container with id 335cc3e3c0010d97c80ac89f76d433c9d3ec2cd240838736de3b5d30c89b7743
Nov 02 15:18:56.834024 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:56.833988    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-501738-b94d75b7d-s94zq" event={"ID":"c55a8850-8fb6-4059-851e-47519d695e04","Type":"ContainerStarted","Data":"d0fd7613960afb9ab1525da859024bfa6e5e421d172d641faa20c5ba286c4189"}
Nov 02 15:18:56.834219 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:56.834057    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-501738-b94d75b7d-s94zq"
Nov 02 15:18:56.834219 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:56.834074    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-501738-b94d75b7d-s94zq" event={"ID":"c55a8850-8fb6-4059-851e-47519d695e04","Type":"ContainerStarted","Data":"335cc3e3c0010d97c80ac89f76d433c9d3ec2cd240838736de3b5d30c89b7743"}
Nov 02 15:18:56.878681 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:18:56.878617    5304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-501738-b94d75b7d-s94zq" podStartSLOduration=1.535587239 podStartE2EDuration="1.878601141s" totalImagesPullingTime="343.013902ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:18:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:18:56.875988056 +0000 UTC m=+1336.869812466" watchObservedRunningTime="2026-11-02 15:18:56.878601141 +0000 UTC m=+1336.872425594"
Nov 02 15:19:00.925258 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:19:00.925025    5304 generic.go:356] "Generic (PLEG): container finished" podID="474f879d-3a52-4e47-8eb0-95a4fcfbe399" containerID="43cb8a845d15be46fe6bb789d7bb2ed4ae2799cb345e07749961896e872f4875" exitCode=0
Nov 02 15:19:00.925258 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:19:00.925203    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-lphrz" event={"ID":"474f879d-3a52-4e47-8eb0-95a4fcfbe399","Type":"ContainerDied","Data":"43cb8a845d15be46fe6bb789d7bb2ed4ae2799cb345e07749961896e872f4875"}
Nov 02 15:19:00.926121 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:19:00.925395    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-25flx"
Nov 02 15:19:00.927310 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:19:00.927222    5304 generic.go:356] "Generic (PLEG): container finished" podID="45f29144-431b-4948-abfa-a1fc5a6847ef" containerID="24ded508397143b4d07251767075b3096cfcb6d5ce566922551c62edbdb45520" exitCode=0
Nov 02 15:19:00.927481 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:19:00.927405    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-9gswz" event={"ID":"45f29144-431b-4948-abfa-a1fc5a6847ef","Type":"ContainerDied","Data":"24ded508397143b4d07251767075b3096cfcb6d5ce566922551c62edbdb45520"}
Nov 02 15:19:00.929363 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:19:00.928639    5304 generic.go:356] "Generic (PLEG): container finished" podID="6f80b62a-49ba-4922-8f3e-804d6bdcc46f" containerID="bb9feef4520f31371456ece67144a6dd30fc5fb819a8f39c85e7bb9b83206635" exitCode=0
Nov 02 15:19:00.929363 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:19:00.928741    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-25flx" event={"ID":"6f80b62a-49ba-4922-8f3e-804d6bdcc46f","Type":"ContainerDied","Data":"bb9feef4520f31371456ece67144a6dd30fc5fb819a8f39c85e7bb9b83206635"}
Nov 02 15:19:00.929363 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:19:00.928770    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-25flx" event={"ID":"6f80b62a-49ba-4922-8f3e-804d6bdcc46f","Type":"ContainerDied","Data":"962ed1c22a2df2872c45e5df6e5223119ad3f40b6a85e995a11141f9cebc6b97"}
Nov 02 15:19:00.929363 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:19:00.928792    5304 scope.go:122] "RemoveContainer" containerID="bb9feef4520f31371456ece67144a6dd30fc5fb819a8f39c85e7bb9b83206635"
Nov 02 15:19:00.929363 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:19:00.928920    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-25flx"
Nov 02 15:19:00.962236 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:19:00.961605    5304 scope.go:122] "RemoveContainer" containerID="bb9feef4520f31371456ece67144a6dd30fc5fb819a8f39c85e7bb9b83206635"
Nov 02 15:19:00.963492 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:19:00.963409    5304 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb9feef4520f31371456ece67144a6dd30fc5fb819a8f39c85e7bb9b83206635\": container with ID starting with bb9feef4520f31371456ece67144a6dd30fc5fb819a8f39c85e7bb9b83206635 not found: ID does not exist" containerID="bb9feef4520f31371456ece67144a6dd30fc5fb819a8f39c85e7bb9b83206635"
Nov 02 15:19:00.963492 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:19:00.963454    5304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb9feef4520f31371456ece67144a6dd30fc5fb819a8f39c85e7bb9b83206635"} err="failed to get container status \"bb9feef4520f31371456ece67144a6dd30fc5fb819a8f39c85e7bb9b83206635\": rpc error: code = NotFound desc = could not find container \"bb9feef4520f31371456ece67144a6dd30fc5fb819a8f39c85e7bb9b83206635\": container with ID starting with bb9feef4520f31371456ece67144a6dd30fc5fb819a8f39c85e7bb9b83206635 not found: ID does not exist"
Nov 02 15:19:01.055099 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:19:01.054535    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f80b62a-49ba-4922-8f3e-804d6bdcc46f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f80b62a-49ba-4922-8f3e-804d6bdcc46f-credential-socket\") pod \"6f80b62a-49ba-4922-8f3e-804d6bdcc46f\" (UID: \"6f80b62a-49ba-4922-8f3e-804d6bdcc46f\") "
Nov 02 15:19:01.055099 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:19:01.054669    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f80b62a-49ba-4922-8f3e-804d6bdcc46f-kube-api-access-h4p8n\" (UniqueName: \"kubernetes.io/projected/6f80b62a-49ba-4922-8f3e-804d6bdcc46f-kube-api-access-h4p8n\") pod \"6f80b62a-49ba-4922-8f3e-804d6bdcc46f\" (UID: \"6f80b62a-49ba-4922-8f3e-804d6bdcc46f\") "
Nov 02 15:19:01.055099 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:19:01.054742    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f80b62a-49ba-4922-8f3e-804d6bdcc46f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f80b62a-49ba-4922-8f3e-804d6bdcc46f-workload-certs\") pod \"6f80b62a-49ba-4922-8f3e-804d6bdcc46f\" (UID: \"6f80b62a-49ba-4922-8f3e-804d6bdcc46f\") "
Nov 02 15:19:01.055099 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:19:01.054807    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f80b62a-49ba-4922-8f3e-804d6bdcc46f-istio-token\" (UniqueName: \"kubernetes.io/projected/6f80b62a-49ba-4922-8f3e-804d6bdcc46f-istio-token\") pod \"6f80b62a-49ba-4922-8f3e-804d6bdcc46f\" (UID: \"6f80b62a-49ba-4922-8f3e-804d6bdcc46f\") "
Nov 02 15:19:01.055099 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:19:01.054917    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f80b62a-49ba-4922-8f3e-804d6bdcc46f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f80b62a-49ba-4922-8f3e-804d6bdcc46f-workload-socket\") pod \"6f80b62a-49ba-4922-8f3e-804d6bdcc46f\" (UID: \"6f80b62a-49ba-4922-8f3e-804d6bdcc46f\") "
Nov 02 15:19:01.055510 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:19:01.055405    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-9gswz"
Nov 02 15:19:01.056386 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:19:01.056207    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f80b62a-49ba-4922-8f3e-804d6bdcc46f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f80b62a-49ba-4922-8f3e-804d6bdcc46f-istio-data\") pod \"6f80b62a-49ba-4922-8f3e-804d6bdcc46f\" (UID: \"6f80b62a-49ba-4922-8f3e-804d6bdcc46f\") "
Nov 02 15:19:01.056386 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:19:01.056298    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6f80b62a-49ba-4922-8f3e-804d6bdcc46f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f80b62a-49ba-4922-8f3e-804d6bdcc46f-istio-podinfo\") pod \"6f80b62a-49ba-4922-8f3e-804d6bdcc46f\" (UID: \"6f80b62a-49ba-4922-8f3e-804d6bdcc46f\") "
Nov 02 15:19:01.056577 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:19:01.056445    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f80b62a-49ba-4922-8f3e-804d6bdcc46f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f80b62a-49ba-4922-8f3e-804d6bdcc46f-istio-envoy\") pod \"6f80b62a-49ba-4922-8f3e-804d6bdcc46f\" (UID: \"6f80b62a-49ba-4922-8f3e-804d6bdcc46f\") "
Nov 02 15:19:01.056886 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:19:01.056621    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6f80b62a-49ba-4922-8f3e-804d6bdcc46f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f80b62a-49ba-4922-8f3e-804d6bdcc46f-istiod-ca-cert\") pod \"6f80b62a-49ba-4922-8f3e-804d6bdcc46f\" (UID: \"6f80b62a-49ba-4922-8f3e-804d6bdcc46f\") "
Nov 02 15:19:01.056886 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:19:01.056681    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f80b62a-49ba-4922-8f3e-804d6bdcc46f-workload-certs" pod "6f80b62a-49ba-4922-8f3e-804d6bdcc46f" (UID: "6f80b62a-49ba-4922-8f3e-804d6bdcc46f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:19:01.057366 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:19:01.057166    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f80b62a-49ba-4922-8f3e-804d6bdcc46f-credential-socket" pod "6f80b62a-49ba-4922-8f3e-804d6bdcc46f" (UID: "6f80b62a-49ba-4922-8f3e-804d6bdcc46f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:19:01.057887 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:19:01.057498    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f80b62a-49ba-4922-8f3e-804d6bdcc46f-istio-data" pod "6f80b62a-49ba-4922-8f3e-804d6bdcc46f" (UID: "6f80b62a-49ba-4922-8f3e-804d6bdcc46f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:19:01.060336 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:19:01.060309    5304 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f80b62a-49ba-4922-8f3e-804d6bdcc46f-credential-socket\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:19:01.060448 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:19:01.060341    5304 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f80b62a-49ba-4922-8f3e-804d6bdcc46f-workload-certs\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:19:01.060448 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:19:01.060357    5304 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f80b62a-49ba-4922-8f3e-804d6bdcc46f-istio-data\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:19:01.060682 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:19:01.060659    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f80b62a-49ba-4922-8f3e-804d6bdcc46f-istiod-ca-cert" pod "6f80b62a-49ba-4922-8f3e-804d6bdcc46f" (UID: "6f80b62a-49ba-4922-8f3e-804d6bdcc46f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:19:01.061502 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:19:01.061476    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f80b62a-49ba-4922-8f3e-804d6bdcc46f-workload-socket" pod "6f80b62a-49ba-4922-8f3e-804d6bdcc46f" (UID: "6f80b62a-49ba-4922-8f3e-804d6bdcc46f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:19:01.065157 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:19:01.064651    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f80b62a-49ba-4922-8f3e-804d6bdcc46f-istio-token" pod "6f80b62a-49ba-4922-8f3e-804d6bdcc46f" (UID: "6f80b62a-49ba-4922-8f3e-804d6bdcc46f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:19:01.067293 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:19:01.067249    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6f80b62a-49ba-4922-8f3e-804d6bdcc46f-istio-podinfo" pod "6f80b62a-49ba-4922-8f3e-804d6bdcc46f" (UID: "6f80b62a-49ba-4922-8f3e-804d6bdcc46f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 15:19:01.069563 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:19:01.069539    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f80b62a-49ba-4922-8f3e-804d6bdcc46f-istio-envoy" pod "6f80b62a-49ba-4922-8f3e-804d6bdcc46f" (UID: "6f80b62a-49ba-4922-8f3e-804d6bdcc46f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:19:01.072341 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:19:01.072311    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f80b62a-49ba-4922-8f3e-804d6bdcc46f-kube-api-access-h4p8n" pod "6f80b62a-49ba-4922-8f3e-804d6bdcc46f" (UID: "6f80b62a-49ba-4922-8f3e-804d6bdcc46f"). InnerVolumeSpecName "kube-api-access-h4p8n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:19:01.156584 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:19:01.156552    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-lphrz"
Nov 02 15:19:01.160965 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:19:01.160939    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/45f29144-431b-4948-abfa-a1fc5a6847ef-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45f29144-431b-4948-abfa-a1fc5a6847ef-istiod-ca-cert\") pod \"45f29144-431b-4948-abfa-a1fc5a6847ef\" (UID: \"45f29144-431b-4948-abfa-a1fc5a6847ef\") "
Nov 02 15:19:01.161320 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:19:01.161292    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45f29144-431b-4948-abfa-a1fc5a6847ef-istiod-ca-cert" pod "45f29144-431b-4948-abfa-a1fc5a6847ef" (UID: "45f29144-431b-4948-abfa-a1fc5a6847ef"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:19:01.161469 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:19:01.161448    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/45f29144-431b-4948-abfa-a1fc5a6847ef-istio-token\" (UniqueName: \"kubernetes.io/projected/45f29144-431b-4948-abfa-a1fc5a6847ef-istio-token\") pod \"45f29144-431b-4948-abfa-a1fc5a6847ef\" (UID: \"45f29144-431b-4948-abfa-a1fc5a6847ef\") "
Nov 02 15:19:01.161532 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:19:01.161491    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/45f29144-431b-4948-abfa-a1fc5a6847ef-kube-api-access-lztbm\" (UniqueName: \"kubernetes.io/projected/45f29144-431b-4948-abfa-a1fc5a6847ef-kube-api-access-lztbm\") pod \"45f29144-431b-4948-abfa-a1fc5a6847ef\" (UID: \"45f29144-431b-4948-abfa-a1fc5a6847ef\") "
Nov 02 15:19:01.161632 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:19:01.161545    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45f29144-431b-4948-abfa-a1fc5a6847ef-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45f29144-431b-4948-abfa-a1fc5a6847ef-credential-socket\") pod \"45f29144-431b-4948-abfa-a1fc5a6847ef\" (UID: \"45f29144-431b-4948-abfa-a1fc5a6847ef\") "
Nov 02 15:19:01.161688 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:19:01.161632    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45f29144-431b-4948-abfa-a1fc5a6847ef-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45f29144-431b-4948-abfa-a1fc5a6847ef-istio-envoy\") pod \"45f29144-431b-4948-abfa-a1fc5a6847ef\" (UID: \"45f29144-431b-4948-abfa-a1fc5a6847ef\") "
Nov 02 15:19:01.161797 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:19:01.161778    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45f29144-431b-4948-abfa-a1fc5a6847ef-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45f29144-431b-4948-abfa-a1fc5a6847ef-workload-certs\") pod \"45f29144-431b-4948-abfa-a1fc5a6847ef\" (UID: \"45f29144-431b-4948-abfa-a1fc5a6847ef\") "
Nov 02 15:19:01.161874 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:19:01.161846    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45f29144-431b-4948-abfa-a1fc5a6847ef-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45f29144-431b-4948-abfa-a1fc5a6847ef-istio-data\") pod \"45f29144-431b-4948-abfa-a1fc5a6847ef\" (UID: \"45f29144-431b-4948-abfa-a1fc5a6847ef\") "
Nov 02 15:19:01.161948 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:19:01.161916    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45f29144-431b-4948-abfa-a1fc5a6847ef-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45f29144-431b-4948-abfa-a1fc5a6847ef-workload-socket\") pod \"45f29144-431b-4948-abfa-a1fc5a6847ef\" (UID: \"45f29144-431b-4948-abfa-a1fc5a6847ef\") "
Nov 02 15:19:01.162044 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:19:01.162031    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/45f29144-431b-4948-abfa-a1fc5a6847ef-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45f29144-431b-4948-abfa-a1fc5a6847ef-istio-podinfo\") pod \"45f29144-431b-4948-abfa-a1fc5a6847ef\" (UID: \"45f29144-431b-4948-abfa-a1fc5a6847ef\") "
Nov 02 15:19:01.162786 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:19:01.162761    5304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h4p8n\" (UniqueName: \"kubernetes.io/projected/6f80b62a-49ba-4922-8f3e-804d6bdcc46f-kube-api-access-h4p8n\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:19:01.162890 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:19:01.162788    5304 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f80b62a-49ba-4922-8f3e-804d6bdcc46f-istio-token\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:19:01.162890 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:19:01.162802    5304 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45f29144-431b-4948-abfa-a1fc5a6847ef-istiod-ca-cert\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:19:01.162890 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:19:01.162814    5304 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f80b62a-49ba-4922-8f3e-804d6bdcc46f-workload-socket\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:19:01.162890 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:19:01.162828    5304 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f80b62a-49ba-4922-8f3e-804d6bdcc46f-istio-podinfo\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:19:01.162890 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:19:01.162840    5304 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f80b62a-49ba-4922-8f3e-804d6bdcc46f-istio-envoy\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:19:01.162890 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:19:01.162854    5304 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f80b62a-49ba-4922-8f3e-804d6bdcc46f-istiod-ca-cert\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:19:01.163743 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:19:01.163720    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45f29144-431b-4948-abfa-a1fc5a6847ef-istio-data" pod "45f29144-431b-4948-abfa-a1fc5a6847ef" (UID: "45f29144-431b-4948-abfa-a1fc5a6847ef"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:19:01.164981 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:19:01.164031    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45f29144-431b-4948-abfa-a1fc5a6847ef-credential-socket" pod "45f29144-431b-4948-abfa-a1fc5a6847ef" (UID: "45f29144-431b-4948-abfa-a1fc5a6847ef"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:19:01.164981 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:19:01.164336    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45f29144-431b-4948-abfa-a1fc5a6847ef-workload-socket" pod "45f29144-431b-4948-abfa-a1fc5a6847ef" (UID: "45f29144-431b-4948-abfa-a1fc5a6847ef"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:19:01.165530 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:19:01.165207    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45f29144-431b-4948-abfa-a1fc5a6847ef-workload-certs" pod "45f29144-431b-4948-abfa-a1fc5a6847ef" (UID: "45f29144-431b-4948-abfa-a1fc5a6847ef"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:19:01.171892 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:19:01.171859    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/45f29144-431b-4948-abfa-a1fc5a6847ef-istio-podinfo" pod "45f29144-431b-4948-abfa-a1fc5a6847ef" (UID: "45f29144-431b-4948-abfa-a1fc5a6847ef"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 15:19:01.172247 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:19:01.172217    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45f29144-431b-4948-abfa-a1fc5a6847ef-istio-envoy" pod "45f29144-431b-4948-abfa-a1fc5a6847ef" (UID: "45f29144-431b-4948-abfa-a1fc5a6847ef"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:19:01.172492 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:19:01.172476    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45f29144-431b-4948-abfa-a1fc5a6847ef-istio-token" pod "45f29144-431b-4948-abfa-a1fc5a6847ef" (UID: "45f29144-431b-4948-abfa-a1fc5a6847ef"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:19:01.175660 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:19:01.175628    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45f29144-431b-4948-abfa-a1fc5a6847ef-kube-api-access-lztbm" pod "45f29144-431b-4948-abfa-a1fc5a6847ef" (UID: "45f29144-431b-4948-abfa-a1fc5a6847ef"). InnerVolumeSpecName "kube-api-access-lztbm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:19:01.264153 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:19:01.264098    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/474f879d-3a52-4e47-8eb0-95a4fcfbe399-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/474f879d-3a52-4e47-8eb0-95a4fcfbe399-workload-socket\") pod \"474f879d-3a52-4e47-8eb0-95a4fcfbe399\" (UID: \"474f879d-3a52-4e47-8eb0-95a4fcfbe399\") "
Nov 02 15:19:01.264481 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:19:01.264439    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/474f879d-3a52-4e47-8eb0-95a4fcfbe399-workload-socket" pod "474f879d-3a52-4e47-8eb0-95a4fcfbe399" (UID: "474f879d-3a52-4e47-8eb0-95a4fcfbe399"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:19:01.264656 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:19:01.264634    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/474f879d-3a52-4e47-8eb0-95a4fcfbe399-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/474f879d-3a52-4e47-8eb0-95a4fcfbe399-istio-podinfo\") pod \"474f879d-3a52-4e47-8eb0-95a4fcfbe399\" (UID: \"474f879d-3a52-4e47-8eb0-95a4fcfbe399\") "
Nov 02 15:19:01.264844 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:19:01.264823    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/474f879d-3a52-4e47-8eb0-95a4fcfbe399-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/474f879d-3a52-4e47-8eb0-95a4fcfbe399-istiod-ca-cert\") pod \"474f879d-3a52-4e47-8eb0-95a4fcfbe399\" (UID: \"474f879d-3a52-4e47-8eb0-95a4fcfbe399\") "
Nov 02 15:19:01.264986 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:19:01.264971    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/474f879d-3a52-4e47-8eb0-95a4fcfbe399-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/474f879d-3a52-4e47-8eb0-95a4fcfbe399-istio-data\") pod \"474f879d-3a52-4e47-8eb0-95a4fcfbe399\" (UID: \"474f879d-3a52-4e47-8eb0-95a4fcfbe399\") "
Nov 02 15:19:01.265190 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:19:01.265173    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/474f879d-3a52-4e47-8eb0-95a4fcfbe399-kube-api-access-ncrxc\" (UniqueName: \"kubernetes.io/projected/474f879d-3a52-4e47-8eb0-95a4fcfbe399-kube-api-access-ncrxc\") pod \"474f879d-3a52-4e47-8eb0-95a4fcfbe399\" (UID: \"474f879d-3a52-4e47-8eb0-95a4fcfbe399\") "
Nov 02 15:19:01.265381 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:19:01.265348    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/474f879d-3a52-4e47-8eb0-95a4fcfbe399-istio-data" pod "474f879d-3a52-4e47-8eb0-95a4fcfbe399" (UID: "474f879d-3a52-4e47-8eb0-95a4fcfbe399"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:19:01.265463 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:19:01.265395    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/474f879d-3a52-4e47-8eb0-95a4fcfbe399-istiod-ca-cert" pod "474f879d-3a52-4e47-8eb0-95a4fcfbe399" (UID: "474f879d-3a52-4e47-8eb0-95a4fcfbe399"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:19:01.265595 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:19:01.265576    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/474f879d-3a52-4e47-8eb0-95a4fcfbe399-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/474f879d-3a52-4e47-8eb0-95a4fcfbe399-workload-certs\") pod \"474f879d-3a52-4e47-8eb0-95a4fcfbe399\" (UID: \"474f879d-3a52-4e47-8eb0-95a4fcfbe399\") "
Nov 02 15:19:01.265767 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:19:01.265736    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/474f879d-3a52-4e47-8eb0-95a4fcfbe399-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/474f879d-3a52-4e47-8eb0-95a4fcfbe399-credential-socket\") pod \"474f879d-3a52-4e47-8eb0-95a4fcfbe399\" (UID: \"474f879d-3a52-4e47-8eb0-95a4fcfbe399\") "
Nov 02 15:19:01.265939 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:19:01.265922    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/474f879d-3a52-4e47-8eb0-95a4fcfbe399-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/474f879d-3a52-4e47-8eb0-95a4fcfbe399-istio-envoy\") pod \"474f879d-3a52-4e47-8eb0-95a4fcfbe399\" (UID: \"474f879d-3a52-4e47-8eb0-95a4fcfbe399\") "
Nov 02 15:19:01.266083 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:19:01.266068    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/474f879d-3a52-4e47-8eb0-95a4fcfbe399-istio-token\" (UniqueName: \"kubernetes.io/projected/474f879d-3a52-4e47-8eb0-95a4fcfbe399-istio-token\") pod \"474f879d-3a52-4e47-8eb0-95a4fcfbe399\" (UID: \"474f879d-3a52-4e47-8eb0-95a4fcfbe399\") "
Nov 02 15:19:01.266283 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:19:01.265832    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/474f879d-3a52-4e47-8eb0-95a4fcfbe399-workload-certs" pod "474f879d-3a52-4e47-8eb0-95a4fcfbe399" (UID: "474f879d-3a52-4e47-8eb0-95a4fcfbe399"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:19:01.266356 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:19:01.266246    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/474f879d-3a52-4e47-8eb0-95a4fcfbe399-credential-socket" pod "474f879d-3a52-4e47-8eb0-95a4fcfbe399" (UID: "474f879d-3a52-4e47-8eb0-95a4fcfbe399"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:19:01.267205 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:19:01.267179    5304 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45f29144-431b-4948-abfa-a1fc5a6847ef-istio-podinfo\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:19:01.267338 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:19:01.267323    5304 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/474f879d-3a52-4e47-8eb0-95a4fcfbe399-istiod-ca-cert\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:19:01.267477 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:19:01.267463    5304 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/474f879d-3a52-4e47-8eb0-95a4fcfbe399-istio-data\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:19:01.267584 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:19:01.267571    5304 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45f29144-431b-4948-abfa-a1fc5a6847ef-istio-token\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:19:01.267711 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:19:01.267697    5304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lztbm\" (UniqueName: \"kubernetes.io/projected/45f29144-431b-4948-abfa-a1fc5a6847ef-kube-api-access-lztbm\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:19:01.267804 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:19:01.267790    5304 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45f29144-431b-4948-abfa-a1fc5a6847ef-credential-socket\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:19:01.267896 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:19:01.267884    5304 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45f29144-431b-4948-abfa-a1fc5a6847ef-istio-envoy\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:19:01.267980 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:19:01.267969    5304 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/474f879d-3a52-4e47-8eb0-95a4fcfbe399-workload-certs\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:19:01.268070 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:19:01.268056    5304 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/474f879d-3a52-4e47-8eb0-95a4fcfbe399-credential-socket\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:19:01.269070 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:19:01.269026    5304 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45f29144-431b-4948-abfa-a1fc5a6847ef-workload-certs\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:19:01.269223 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:19:01.269204    5304 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45f29144-431b-4948-abfa-a1fc5a6847ef-istio-data\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:19:01.269223 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:19:01.269225    5304 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/474f879d-3a52-4e47-8eb0-95a4fcfbe399-workload-socket\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:19:01.269406 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:19:01.269240    5304 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45f29144-431b-4948-abfa-a1fc5a6847ef-workload-socket\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:19:01.269406 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:19:01.268086    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/474f879d-3a52-4e47-8eb0-95a4fcfbe399-istio-podinfo" pod "474f879d-3a52-4e47-8eb0-95a4fcfbe399" (UID: "474f879d-3a52-4e47-8eb0-95a4fcfbe399"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 15:19:01.269580 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:19:01.269539    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/474f879d-3a52-4e47-8eb0-95a4fcfbe399-istio-envoy" pod "474f879d-3a52-4e47-8eb0-95a4fcfbe399" (UID: "474f879d-3a52-4e47-8eb0-95a4fcfbe399"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:19:01.269686 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:19:01.269666    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/474f879d-3a52-4e47-8eb0-95a4fcfbe399-istio-token" pod "474f879d-3a52-4e47-8eb0-95a4fcfbe399" (UID: "474f879d-3a52-4e47-8eb0-95a4fcfbe399"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:19:01.270872 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:19:01.270849    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/474f879d-3a52-4e47-8eb0-95a4fcfbe399-kube-api-access-ncrxc" pod "474f879d-3a52-4e47-8eb0-95a4fcfbe399" (UID: "474f879d-3a52-4e47-8eb0-95a4fcfbe399"). InnerVolumeSpecName "kube-api-access-ncrxc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:19:01.281616 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:19:01.281589    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-25flx"]
Nov 02 15:19:01.285578 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:19:01.285539    5304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-25flx"]
Nov 02 15:19:01.373323 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:19:01.370732    5304 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/474f879d-3a52-4e47-8eb0-95a4fcfbe399-istio-podinfo\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:19:01.373323 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:19:01.370757    5304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ncrxc\" (UniqueName: \"kubernetes.io/projected/474f879d-3a52-4e47-8eb0-95a4fcfbe399-kube-api-access-ncrxc\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:19:01.373323 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:19:01.370768    5304 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/474f879d-3a52-4e47-8eb0-95a4fcfbe399-istio-envoy\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:19:01.373323 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:19:01.370777    5304 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/474f879d-3a52-4e47-8eb0-95a4fcfbe399-istio-token\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:19:01.598457 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:19:01.598427    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-xcfbs"
Nov 02 15:19:01.683534 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:19:01.683445    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a395ec29-6257-43f1-a2f0-1ec07bf215ad-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a395ec29-6257-43f1-a2f0-1ec07bf215ad-istio-data\") pod \"a395ec29-6257-43f1-a2f0-1ec07bf215ad\" (UID: \"a395ec29-6257-43f1-a2f0-1ec07bf215ad\") "
Nov 02 15:19:01.683709 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:19:01.683582    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a395ec29-6257-43f1-a2f0-1ec07bf215ad-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a395ec29-6257-43f1-a2f0-1ec07bf215ad-credential-socket\") pod \"a395ec29-6257-43f1-a2f0-1ec07bf215ad\" (UID: \"a395ec29-6257-43f1-a2f0-1ec07bf215ad\") "
Nov 02 15:19:01.683709 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:19:01.683675    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a395ec29-6257-43f1-a2f0-1ec07bf215ad-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a395ec29-6257-43f1-a2f0-1ec07bf215ad-istio-podinfo\") pod \"a395ec29-6257-43f1-a2f0-1ec07bf215ad\" (UID: \"a395ec29-6257-43f1-a2f0-1ec07bf215ad\") "
Nov 02 15:19:01.684182 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:19:01.683747    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a395ec29-6257-43f1-a2f0-1ec07bf215ad-istio-data" pod "a395ec29-6257-43f1-a2f0-1ec07bf215ad" (UID: "a395ec29-6257-43f1-a2f0-1ec07bf215ad"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:19:01.684182 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:19:01.683916    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a395ec29-6257-43f1-a2f0-1ec07bf215ad-credential-socket" pod "a395ec29-6257-43f1-a2f0-1ec07bf215ad" (UID: "a395ec29-6257-43f1-a2f0-1ec07bf215ad"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:19:01.684182 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:19:01.683954    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a395ec29-6257-43f1-a2f0-1ec07bf215ad-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a395ec29-6257-43f1-a2f0-1ec07bf215ad-workload-socket\") pod \"a395ec29-6257-43f1-a2f0-1ec07bf215ad\" (UID: \"a395ec29-6257-43f1-a2f0-1ec07bf215ad\") "
Nov 02 15:19:01.684182 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:19:01.684106    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a395ec29-6257-43f1-a2f0-1ec07bf215ad-istio-token\" (UniqueName: \"kubernetes.io/projected/a395ec29-6257-43f1-a2f0-1ec07bf215ad-istio-token\") pod \"a395ec29-6257-43f1-a2f0-1ec07bf215ad\" (UID: \"a395ec29-6257-43f1-a2f0-1ec07bf215ad\") "
Nov 02 15:19:01.684182 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:19:01.684157    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a395ec29-6257-43f1-a2f0-1ec07bf215ad-kube-api-access-zs497\" (UniqueName: \"kubernetes.io/projected/a395ec29-6257-43f1-a2f0-1ec07bf215ad-kube-api-access-zs497\") pod \"a395ec29-6257-43f1-a2f0-1ec07bf215ad\" (UID: \"a395ec29-6257-43f1-a2f0-1ec07bf215ad\") "
Nov 02 15:19:01.684551 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:19:01.684240    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a395ec29-6257-43f1-a2f0-1ec07bf215ad-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a395ec29-6257-43f1-a2f0-1ec07bf215ad-istio-envoy\") pod \"a395ec29-6257-43f1-a2f0-1ec07bf215ad\" (UID: \"a395ec29-6257-43f1-a2f0-1ec07bf215ad\") "
Nov 02 15:19:01.684551 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:19:01.684361    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a395ec29-6257-43f1-a2f0-1ec07bf215ad-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a395ec29-6257-43f1-a2f0-1ec07bf215ad-istiod-ca-cert\") pod \"a395ec29-6257-43f1-a2f0-1ec07bf215ad\" (UID: \"a395ec29-6257-43f1-a2f0-1ec07bf215ad\") "
Nov 02 15:19:01.684551 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:19:01.684406    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a395ec29-6257-43f1-a2f0-1ec07bf215ad-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a395ec29-6257-43f1-a2f0-1ec07bf215ad-workload-certs\") pod \"a395ec29-6257-43f1-a2f0-1ec07bf215ad\" (UID: \"a395ec29-6257-43f1-a2f0-1ec07bf215ad\") "
Nov 02 15:19:01.684722 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:19:01.684558    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a395ec29-6257-43f1-a2f0-1ec07bf215ad-workload-socket" pod "a395ec29-6257-43f1-a2f0-1ec07bf215ad" (UID: "a395ec29-6257-43f1-a2f0-1ec07bf215ad"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:19:01.684975 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:19:01.684955    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a395ec29-6257-43f1-a2f0-1ec07bf215ad-istiod-ca-cert" pod "a395ec29-6257-43f1-a2f0-1ec07bf215ad" (UID: "a395ec29-6257-43f1-a2f0-1ec07bf215ad"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:19:01.685087 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:19:01.685028    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a395ec29-6257-43f1-a2f0-1ec07bf215ad-workload-certs" pod "a395ec29-6257-43f1-a2f0-1ec07bf215ad" (UID: "a395ec29-6257-43f1-a2f0-1ec07bf215ad"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:19:01.685513 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:19:01.685485    5304 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a395ec29-6257-43f1-a2f0-1ec07bf215ad-credential-socket\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:19:01.685579 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:19:01.685517    5304 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a395ec29-6257-43f1-a2f0-1ec07bf215ad-workload-socket\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:19:01.685579 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:19:01.685532    5304 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a395ec29-6257-43f1-a2f0-1ec07bf215ad-istiod-ca-cert\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:19:01.685579 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:19:01.685547    5304 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a395ec29-6257-43f1-a2f0-1ec07bf215ad-workload-certs\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:19:01.685579 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:19:01.685560    5304 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a395ec29-6257-43f1-a2f0-1ec07bf215ad-istio-data\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:19:01.687537 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:19:01.687503    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a395ec29-6257-43f1-a2f0-1ec07bf215ad-istio-token" pod "a395ec29-6257-43f1-a2f0-1ec07bf215ad" (UID: "a395ec29-6257-43f1-a2f0-1ec07bf215ad"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:19:01.687651 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:19:01.687561    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a395ec29-6257-43f1-a2f0-1ec07bf215ad-istio-podinfo" pod "a395ec29-6257-43f1-a2f0-1ec07bf215ad" (UID: "a395ec29-6257-43f1-a2f0-1ec07bf215ad"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 15:19:01.689422 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:19:01.689392    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a395ec29-6257-43f1-a2f0-1ec07bf215ad-kube-api-access-zs497" pod "a395ec29-6257-43f1-a2f0-1ec07bf215ad" (UID: "a395ec29-6257-43f1-a2f0-1ec07bf215ad"). InnerVolumeSpecName "kube-api-access-zs497". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:19:01.689758 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:19:01.689735    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a395ec29-6257-43f1-a2f0-1ec07bf215ad-istio-envoy" pod "a395ec29-6257-43f1-a2f0-1ec07bf215ad" (UID: "a395ec29-6257-43f1-a2f0-1ec07bf215ad"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:19:01.787049 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:19:01.787016    5304 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a395ec29-6257-43f1-a2f0-1ec07bf215ad-istio-podinfo\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:19:01.787049 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:19:01.787046    5304 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a395ec29-6257-43f1-a2f0-1ec07bf215ad-istio-token\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:19:01.787287 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:19:01.787059    5304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zs497\" (UniqueName: \"kubernetes.io/projected/a395ec29-6257-43f1-a2f0-1ec07bf215ad-kube-api-access-zs497\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:19:01.787287 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:19:01.787069    5304 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a395ec29-6257-43f1-a2f0-1ec07bf215ad-istio-envoy\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:19:01.939481 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:19:01.939400    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-lphrz"
Nov 02 15:19:01.939481 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:19:01.939432    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-lphrz" event={"ID":"474f879d-3a52-4e47-8eb0-95a4fcfbe399","Type":"ContainerDied","Data":"006118828f472196afdf4a95e8db0498abb6b757ec29eaa3655abb58ee27e317"}
Nov 02 15:19:01.939975 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:19:01.939502    5304 scope.go:122] "RemoveContainer" containerID="43cb8a845d15be46fe6bb789d7bb2ed4ae2799cb345e07749961896e872f4875"
Nov 02 15:19:01.941009 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:19:01.940983    5304 generic.go:356] "Generic (PLEG): container finished" podID="a395ec29-6257-43f1-a2f0-1ec07bf215ad" containerID="a075c313dd476af7b38cb47ac46b708c29575c832eae7c3eca5e503e0f1927ee" exitCode=0
Nov 02 15:19:01.941168 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:19:01.941035    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-xcfbs"
Nov 02 15:19:01.941168 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:19:01.941049    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-xcfbs" event={"ID":"a395ec29-6257-43f1-a2f0-1ec07bf215ad","Type":"ContainerDied","Data":"a075c313dd476af7b38cb47ac46b708c29575c832eae7c3eca5e503e0f1927ee"}
Nov 02 15:19:01.941168 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:19:01.941107    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-xcfbs" event={"ID":"a395ec29-6257-43f1-a2f0-1ec07bf215ad","Type":"ContainerDied","Data":"ad18961e6c5af7315acd59e55cc42053649a1abb7945dba189403e58cfbe2af5"}
Nov 02 15:19:01.942858 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:19:01.942834    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-9gswz" event={"ID":"45f29144-431b-4948-abfa-a1fc5a6847ef","Type":"ContainerDied","Data":"6d6813ca8ce5380ebe01f8933dc97d4a813a676a5124524623f22eda8ff0046f"}
Nov 02 15:19:01.942964 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:19:01.942885    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-9gswz"
Nov 02 15:19:01.955243 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:19:01.955227    5304 scope.go:122] "RemoveContainer" containerID="a075c313dd476af7b38cb47ac46b708c29575c832eae7c3eca5e503e0f1927ee"
Nov 02 15:19:01.966190 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:19:01.966172    5304 scope.go:122] "RemoveContainer" containerID="a075c313dd476af7b38cb47ac46b708c29575c832eae7c3eca5e503e0f1927ee"
Nov 02 15:19:01.966524 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:19:01.966497    5304 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a075c313dd476af7b38cb47ac46b708c29575c832eae7c3eca5e503e0f1927ee\": container with ID starting with a075c313dd476af7b38cb47ac46b708c29575c832eae7c3eca5e503e0f1927ee not found: ID does not exist" containerID="a075c313dd476af7b38cb47ac46b708c29575c832eae7c3eca5e503e0f1927ee"
Nov 02 15:19:01.966587 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:19:01.966538    5304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a075c313dd476af7b38cb47ac46b708c29575c832eae7c3eca5e503e0f1927ee"} err="failed to get container status \"a075c313dd476af7b38cb47ac46b708c29575c832eae7c3eca5e503e0f1927ee\": rpc error: code = NotFound desc = could not find container \"a075c313dd476af7b38cb47ac46b708c29575c832eae7c3eca5e503e0f1927ee\": container with ID starting with a075c313dd476af7b38cb47ac46b708c29575c832eae7c3eca5e503e0f1927ee not found: ID does not exist"
Nov 02 15:19:01.966587 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:19:01.966564    5304 scope.go:122] "RemoveContainer" containerID="24ded508397143b4d07251767075b3096cfcb6d5ce566922551c62edbdb45520"
Nov 02 15:19:02.070154 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:19:02.069345    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-lphrz"]
Nov 02 15:19:02.076330 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:19:02.076298    5304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-lphrz"]
Nov 02 15:19:02.198324 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:19:02.198151    5304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="474f879d-3a52-4e47-8eb0-95a4fcfbe399" path="/var/lib/kubelet/pods/474f879d-3a52-4e47-8eb0-95a4fcfbe399/volumes"
Nov 02 15:19:02.199531 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:19:02.199505    5304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6f80b62a-49ba-4922-8f3e-804d6bdcc46f" path="/var/lib/kubelet/pods/6f80b62a-49ba-4922-8f3e-804d6bdcc46f/volumes"
Nov 02 15:19:02.200610 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:19:02.200588    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-9gswz"]
Nov 02 15:19:02.200854 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:19:02.200839    5304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-9gswz"]
Nov 02 15:19:02.201007 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:19:02.200995    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-xcfbs"]
Nov 02 15:19:02.209783 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:19:02.209738    5304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-xcfbs"]
Nov 02 15:19:02.850413 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:19:02.850381    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-501738-b94d75b7d-s94zq"
Nov 02 15:19:04.120249 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:19:04.120217    5304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="45f29144-431b-4948-abfa-a1fc5a6847ef" path="/var/lib/kubelet/pods/45f29144-431b-4948-abfa-a1fc5a6847ef/volumes"
Nov 02 15:19:04.120860 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:19:04.120843    5304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a395ec29-6257-43f1-a2f0-1ec07bf215ad" path="/var/lib/kubelet/pods/a395ec29-6257-43f1-a2f0-1ec07bf215ad/volumes"
Nov 02 15:19:17.076571 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:19:17.076530    5304 generic.go:356] "Generic (PLEG): container finished" podID="c34a9870-db4c-462a-827f-95abdfc1f68b" containerID="5c8aa0912102f7f4905ec45c8ba28cc09a31ea54fa487ffa50530f601472732a" exitCode=0
Nov 02 15:19:17.077044 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:19:17.076604    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-5f9166-openshift-default-5d4b56b869-skdww" event={"ID":"c34a9870-db4c-462a-827f-95abdfc1f68b","Type":"ContainerDied","Data":"5c8aa0912102f7f4905ec45c8ba28cc09a31ea54fa487ffa50530f601472732a"}
Nov 02 15:19:17.077087 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:19:17.077077    5304 scope.go:122] "RemoveContainer" containerID="5c8aa0912102f7f4905ec45c8ba28cc09a31ea54fa487ffa50530f601472732a"
Nov 02 15:19:17.078927 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:19:17.078893    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:19:17.079049 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:19:17.078983    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:19:17.079049 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:19:17.079035    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:19:18.089919 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:19:18.089881    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-5f9166-openshift-default-5d4b56b869-skdww" event={"ID":"c34a9870-db4c-462a-827f-95abdfc1f68b","Type":"ContainerStarted","Data":"7553fcc74706c6f700066b6fa868972f26203f2905a05f1a775b6ebb40fa53cf"}
Nov 02 15:19:18.421019 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:19:18.420931    5304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-5f9166-openshift-default-5d4b56b869-skdww"
Nov 02 15:19:18.421198 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:19:18.421113    5304 patch_prober.go:29] interesting pod/e2e-api-b-5f9166-openshift-default-5d4b56b869-skdww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body=
Nov 02 15:19:18.421198 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:19:18.421182    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-5f9166-openshift-default-5d4b56b869-skdww" podUID="c34a9870-db4c-462a-827f-95abdfc1f68b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused"
Nov 02 15:19:19.420726 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:19:19.420695    5304 patch_prober.go:29] interesting pod/e2e-api-b-5f9166-openshift-default-5d4b56b869-skdww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body=
Nov 02 15:19:19.421115 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:19:19.420765    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-5f9166-openshift-default-5d4b56b869-skdww" podUID="c34a9870-db4c-462a-827f-95abdfc1f68b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused"
Nov 02 15:19:20.421558 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:19:20.421526    5304 patch_prober.go:29] interesting pod/e2e-api-b-5f9166-openshift-default-5d4b56b869-skdww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body=
Nov 02 15:19:20.421981 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:19:20.421576    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-5f9166-openshift-default-5d4b56b869-skdww" podUID="c34a9870-db4c-462a-827f-95abdfc1f68b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused"
Nov 02 15:19:21.421749 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:19:21.421665    5304 patch_prober.go:29] interesting pod/e2e-api-b-5f9166-openshift-default-5d4b56b869-skdww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body=
Nov 02 15:19:21.421749 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:19:21.421736    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-5f9166-openshift-default-5d4b56b869-skdww" podUID="c34a9870-db4c-462a-827f-95abdfc1f68b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused"
Nov 02 15:19:22.420627 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:19:22.420593    5304 patch_prober.go:29] interesting pod/e2e-api-b-5f9166-openshift-default-5d4b56b869-skdww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body=
Nov 02 15:19:22.420812 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:19:22.420655    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-5f9166-openshift-default-5d4b56b869-skdww" podUID="c34a9870-db4c-462a-827f-95abdfc1f68b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused"
Nov 02 15:19:23.290765 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:19:23.290723    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-ad276f-76945ff5-9gg4w"]
Nov 02 15:19:23.325667 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:19:23.325629    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-ad276f-64f66d7895-vxlcf"]
Nov 02 15:19:23.421422 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:19:23.421389    5304 patch_prober.go:29] interesting pod/e2e-api-b-5f9166-openshift-default-5d4b56b869-skdww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body=
Nov 02 15:19:23.421603 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:19:23.421450    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-5f9166-openshift-default-5d4b56b869-skdww" podUID="c34a9870-db4c-462a-827f-95abdfc1f68b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused"
Nov 02 15:19:24.441153 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:19:24.436721    5304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-5f9166-openshift-default-5d4b56b869-skdww"
Nov 02 15:19:24.441153 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:19:24.438091    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-5f9166-openshift-default-5d4b56b869-skdww"
Nov 02 15:19:24.441153 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:19:24.440766    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-5f9166-openshift-default-5d4b56b869-skdww"
Nov 02 15:19:24.824849 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:19:24.824745    5304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k6fjk" containerName="extractor"
Nov 02 15:19:28.300157 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:19:28.300098    5304 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-ad276f-76945ff5-9gg4w" podUID="ca337066-9ffb-4c9e-beee-fe339577a0b7" containerName="payload-processing" containerID="cri-o://c2a3ecee95ed737f278b090569e9855674675b0f9f3a0bb5251097ad09e029c3" gracePeriod=25
Nov 02 15:19:28.326712 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:19:28.326668    5304 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ad276f-64f66d7895-vxlcf" podUID="4f4d49b5-6c2a-4869-a30a-155e3afe755f" containerName="payload-pre-processing" containerID="cri-o://4de0394a7602536646e88be3e3ec39e6c2be0b60d021c9c80d111f576d2e57e1" gracePeriod=25
Nov 02 15:19:28.508521 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:19:28.508448    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1cc1b2/distinct-w5-1cc1b2-kserve-5b7459bc6d-zl8xr" podUID="211ad15d-9234-4f57-b1b9-181dd2d74208" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded"
Nov 02 15:19:28.519730 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:19:28.519665    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5dd8e9/premium-w1-5dd8e9-kserve-67588456fb-j6lph" podUID="4ccc607a-14e4-43ca-8999-78100d4b799f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:19:28.520844 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:19:28.520797    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6af50f/distinct-w0-6af50f-kserve-c6d994647-hbj9z" podUID="3dca5bf3-8e08-47a0-936e-205670d59ede" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:19:28.548080 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:19:28.546548    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-624254/distinct-w3-624254-kserve-5465dc4765-dmq67" podUID="1a5aa910-a6de-417b-8638-81e4ec5c1825" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded"
Nov 02 15:19:28.593166 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:19:28.591266    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nkbrk" podUID="1f68d837-fff2-418b-84d7-c2d7eb21a57e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.24:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:19:28.593166 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:19:28.591705    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-739bf1/distinct-2-w2-739bf1-kserve-5c58479c48-qgfkh" podUID="9746bf8a-91bd-4715-a02e-ae2e651ca54d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:19:28.621150 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:19:28.618707    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5dd8e9/simulator-w1-5dd8e9-kserve-6f9f755d6f-gnw5d" podUID="66b75e64-cb24-40be-9076-4984c132e208" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:19:28.621150 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:19:28.619107    5304 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-mt-3d7df16e/e2e-lifecycle-model-3d7df16e-kserve-6c967d4fb5-sh4rx" podUID="a786ec8c-9093-4152-8e99-4d334bd5928a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.91:8000/health\": context deadline exceeded"
Nov 02 15:19:28.654319 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:19:28.641246    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-624254/simulator-w3-624254-kserve-56866b7699-z47sd" podUID="345c710f-b414-4e62-86c7-d51ff4e5be15" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:19:28.682151 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:19:28.681292    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1cc1b2/premium-w5-1cc1b2-kserve-58db46f97c-cgdqg" podUID="210f620a-8ae7-4e4c-acc3-46f75ddc6105" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:19:28.859220 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:19:28.859104    5304 generic.go:356] "Generic (PLEG): container finished" podID="4f4d49b5-6c2a-4869-a30a-155e3afe755f" containerID="4de0394a7602536646e88be3e3ec39e6c2be0b60d021c9c80d111f576d2e57e1" exitCode=0
Nov 02 15:19:28.859411 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:19:28.859341    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ad276f-64f66d7895-vxlcf" event={"ID":"4f4d49b5-6c2a-4869-a30a-155e3afe755f","Type":"ContainerDied","Data":"4de0394a7602536646e88be3e3ec39e6c2be0b60d021c9c80d111f576d2e57e1"}
Nov 02 15:19:28.884147 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:19:28.861813    5304 generic.go:356] "Generic (PLEG): container finished" podID="ca337066-9ffb-4c9e-beee-fe339577a0b7" containerID="c2a3ecee95ed737f278b090569e9855674675b0f9f3a0bb5251097ad09e029c3" exitCode=0
Nov 02 15:19:28.884147 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:19:28.861871    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-ad276f-76945ff5-9gg4w" event={"ID":"ca337066-9ffb-4c9e-beee-fe339577a0b7","Type":"ContainerDied","Data":"c2a3ecee95ed737f278b090569e9855674675b0f9f3a0bb5251097ad09e029c3"}
Nov 02 15:19:28.884147 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:19:28.879577    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-3d7df16e/e2e-lifecycle-model-3d7df16e-kserve-6c967d4fb5-sh4rx" podUID="a786ec8c-9093-4152-8e99-4d334bd5928a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:19:29.428169 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:19:29.428139    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-ad276f-76945ff5-9gg4w"
Nov 02 15:19:29.460152 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:19:29.443035    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ca337066-9ffb-4c9e-beee-fe339577a0b7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca337066-9ffb-4c9e-beee-fe339577a0b7-plugins-config-volume\") pod \"ca337066-9ffb-4c9e-beee-fe339577a0b7\" (UID: \"ca337066-9ffb-4c9e-beee-fe339577a0b7\") "
Nov 02 15:19:29.474159 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:19:29.468312    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca337066-9ffb-4c9e-beee-fe339577a0b7-kube-api-access-l9ct2\" (UniqueName: \"kubernetes.io/projected/ca337066-9ffb-4c9e-beee-fe339577a0b7-kube-api-access-l9ct2\") pod \"ca337066-9ffb-4c9e-beee-fe339577a0b7\" (UID: \"ca337066-9ffb-4c9e-beee-fe339577a0b7\") "
Nov 02 15:19:29.474376 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:19:29.454552    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca337066-9ffb-4c9e-beee-fe339577a0b7-plugins-config-volume" pod "ca337066-9ffb-4c9e-beee-fe339577a0b7" (UID: "ca337066-9ffb-4c9e-beee-fe339577a0b7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:19:29.564153 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:19:29.560320    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca337066-9ffb-4c9e-beee-fe339577a0b7-kube-api-access-l9ct2" pod "ca337066-9ffb-4c9e-beee-fe339577a0b7" (UID: "ca337066-9ffb-4c9e-beee-fe339577a0b7"). InnerVolumeSpecName "kube-api-access-l9ct2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:19:29.621367 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:19:29.613992    5304 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca337066-9ffb-4c9e-beee-fe339577a0b7-plugins-config-volume\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:19:29.621367 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:19:29.614032    5304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l9ct2\" (UniqueName: \"kubernetes.io/projected/ca337066-9ffb-4c9e-beee-fe339577a0b7-kube-api-access-l9ct2\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:19:30.090195 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:19:30.082253    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-ad276f-76945ff5-9gg4w" event={"ID":"ca337066-9ffb-4c9e-beee-fe339577a0b7","Type":"ContainerDied","Data":"aeb4600865f86f39372fe139dcd026af63d6b15cd66da7686b299d2495393770"}
Nov 02 15:19:30.090195 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:19:30.082324    5304 scope.go:122] "RemoveContainer" containerID="c2a3ecee95ed737f278b090569e9855674675b0f9f3a0bb5251097ad09e029c3"
Nov 02 15:19:30.090195 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:19:30.082438    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-ad276f-76945ff5-9gg4w"
Nov 02 15:19:30.204526 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:19:30.201919    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-ad276f-76945ff5-9gg4w"]
Nov 02 15:19:30.210737 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:19:30.210709    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ad276f-64f66d7895-vxlcf"
Nov 02 15:19:30.212904 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:19:30.212877    5304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-ad276f-76945ff5-9gg4w"]
Nov 02 15:19:30.361699 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:19:30.361615    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4f4d49b5-6c2a-4869-a30a-155e3afe755f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f4d49b5-6c2a-4869-a30a-155e3afe755f-plugins-config-volume\") pod \"4f4d49b5-6c2a-4869-a30a-155e3afe755f\" (UID: \"4f4d49b5-6c2a-4869-a30a-155e3afe755f\") "
Nov 02 15:19:30.361931 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:19:30.361909    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f4d49b5-6c2a-4869-a30a-155e3afe755f-kube-api-access-2gsp4\" (UniqueName: \"kubernetes.io/projected/4f4d49b5-6c2a-4869-a30a-155e3afe755f-kube-api-access-2gsp4\") pod \"4f4d49b5-6c2a-4869-a30a-155e3afe755f\" (UID: \"4f4d49b5-6c2a-4869-a30a-155e3afe755f\") "
Nov 02 15:19:30.362081 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:19:30.361987    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f4d49b5-6c2a-4869-a30a-155e3afe755f-plugins-config-volume" pod "4f4d49b5-6c2a-4869-a30a-155e3afe755f" (UID: "4f4d49b5-6c2a-4869-a30a-155e3afe755f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:19:30.362394 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:19:30.362377    5304 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f4d49b5-6c2a-4869-a30a-155e3afe755f-plugins-config-volume\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:19:30.363982 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:19:30.363960    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f4d49b5-6c2a-4869-a30a-155e3afe755f-kube-api-access-2gsp4" pod "4f4d49b5-6c2a-4869-a30a-155e3afe755f" (UID: "4f4d49b5-6c2a-4869-a30a-155e3afe755f"). InnerVolumeSpecName "kube-api-access-2gsp4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:19:30.466573 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:19:30.466530    5304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2gsp4\" (UniqueName: \"kubernetes.io/projected/4f4d49b5-6c2a-4869-a30a-155e3afe755f-kube-api-access-2gsp4\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:19:31.090725 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:19:31.090698    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ad276f-64f66d7895-vxlcf"
Nov 02 15:19:31.090930 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:19:31.090736    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ad276f-64f66d7895-vxlcf" event={"ID":"4f4d49b5-6c2a-4869-a30a-155e3afe755f","Type":"ContainerDied","Data":"0ab37f5595190ae92d547bb683a08a1354ae28afb84fe6a1b9759a6d87b111a3"}
Nov 02 15:19:31.090930 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:19:31.090802    5304 scope.go:122] "RemoveContainer" containerID="4de0394a7602536646e88be3e3ec39e6c2be0b60d021c9c80d111f576d2e57e1"
Nov 02 15:19:31.163055 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:19:31.163020    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-ad276f-64f66d7895-vxlcf"]
Nov 02 15:19:31.172471 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:19:31.172441    5304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-ad276f-64f66d7895-vxlcf"]
Nov 02 15:19:32.255293 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:19:32.245502    5304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4f4d49b5-6c2a-4869-a30a-155e3afe755f" path="/var/lib/kubelet/pods/4f4d49b5-6c2a-4869-a30a-155e3afe755f/volumes"
Nov 02 15:19:32.255293 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:19:32.246416    5304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ca337066-9ffb-4c9e-beee-fe339577a0b7" path="/var/lib/kubelet/pods/ca337066-9ffb-4c9e-beee-fe339577a0b7/volumes"
Nov 02 15:19:37.174727 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:19:37.174692    5304 patch_prober.go:29] interesting pod/e2e-shared-a-w4-501738-openshift-default-54c4d5d859-bwbmq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body=
Nov 02 15:19:37.175225 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:19:37.174757    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w4-501738-openshift-default-54c4d5d859-bwbmq" podUID="f7d14663-db08-4ae9-9dae-98f56d51f8a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused"
Nov 02 15:19:37.277555 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:19:37.277523    5304 generic.go:356] "Generic (PLEG): container finished" podID="f7d14663-db08-4ae9-9dae-98f56d51f8a4" containerID="26297c35ae35a96a888e684c429bbe4883e98d41daecfe97074a203d088eb6db" exitCode=0
Nov 02 15:19:37.277729 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:19:37.277564    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-501738-openshift-default-54c4d5d859-bwbmq" event={"ID":"f7d14663-db08-4ae9-9dae-98f56d51f8a4","Type":"ContainerDied","Data":"26297c35ae35a96a888e684c429bbe4883e98d41daecfe97074a203d088eb6db"}
Nov 02 15:19:37.278011 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:19:37.277997    5304 scope.go:122] "RemoveContainer" containerID="26297c35ae35a96a888e684c429bbe4883e98d41daecfe97074a203d088eb6db"
Nov 02 15:19:37.279792 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:19:37.279754    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:19:37.279922 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:19:37.279845    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:19:37.279922 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:19:37.279895    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:19:38.286564 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:19:38.286524    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-501738-openshift-default-54c4d5d859-bwbmq" event={"ID":"f7d14663-db08-4ae9-9dae-98f56d51f8a4","Type":"ContainerStarted","Data":"0a34c6823625ecbc1ed41f24dee0a5cd0cd06159cecd9bba65d71e90fbcc34a8"}
Nov 02 15:19:38.957100 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:19:38.957064    5304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-501738-openshift-default-54c4d5d859-bwbmq"
Nov 02 15:19:38.957418 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:19:38.957392    5304 patch_prober.go:29] interesting pod/e2e-shared-a-w4-501738-openshift-default-54c4d5d859-bwbmq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body=
Nov 02 15:19:38.957544 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:19:38.957450    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-501738-openshift-default-54c4d5d859-bwbmq" podUID="f7d14663-db08-4ae9-9dae-98f56d51f8a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused"
Nov 02 15:19:39.957783 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:19:39.957745    5304 patch_prober.go:29] interesting pod/e2e-shared-a-w4-501738-openshift-default-54c4d5d859-bwbmq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body=
Nov 02 15:19:39.958251 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:19:39.957805    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-501738-openshift-default-54c4d5d859-bwbmq" podUID="f7d14663-db08-4ae9-9dae-98f56d51f8a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused"
Nov 02 15:19:40.957701 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:19:40.957664    5304 patch_prober.go:29] interesting pod/e2e-shared-a-w4-501738-openshift-default-54c4d5d859-bwbmq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body=
Nov 02 15:19:40.957888 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:19:40.957729    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-501738-openshift-default-54c4d5d859-bwbmq" podUID="f7d14663-db08-4ae9-9dae-98f56d51f8a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused"
Nov 02 15:19:41.113385 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:19:41.113355    5304 scope.go:122] "RemoveContainer" containerID="156ba9c562d45aac63e59d03266fcd448db031c105d41582bc99e393c277ae45"
Nov 02 15:19:41.958249 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:19:41.958215    5304 patch_prober.go:29] interesting pod/e2e-shared-a-w4-501738-openshift-default-54c4d5d859-bwbmq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body=
Nov 02 15:19:41.958717 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:19:41.958289    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-501738-openshift-default-54c4d5d859-bwbmq" podUID="f7d14663-db08-4ae9-9dae-98f56d51f8a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused"
Nov 02 15:19:42.957383 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:19:42.957340    5304 patch_prober.go:29] interesting pod/e2e-shared-a-w4-501738-openshift-default-54c4d5d859-bwbmq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body=
Nov 02 15:19:42.957714 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:19:42.957416    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-501738-openshift-default-54c4d5d859-bwbmq" podUID="f7d14663-db08-4ae9-9dae-98f56d51f8a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused"
Nov 02 15:19:43.957636 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:19:43.957594    5304 patch_prober.go:29] interesting pod/e2e-shared-a-w4-501738-openshift-default-54c4d5d859-bwbmq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body=
Nov 02 15:19:43.958174 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:19:43.957657    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-501738-openshift-default-54c4d5d859-bwbmq" podUID="f7d14663-db08-4ae9-9dae-98f56d51f8a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused"
Nov 02 15:19:44.962379 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:19:44.962346    5304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-501738-openshift-default-54c4d5d859-bwbmq"
Nov 02 15:19:44.962819 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:19:44.962670    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-501738-openshift-default-54c4d5d859-bwbmq"
Nov 02 15:19:44.963454 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:19:44.963432    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-501738-openshift-default-54c4d5d859-bwbmq"
Nov 02 15:19:54.802295 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:19:54.802255    5304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k6fjk" containerName="extractor"
Nov 02 15:19:57.553860 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:19:57.553825    5304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-55e4c3-578b56f96d-lmkft"]
Nov 02 15:19:57.554483 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:19:57.554469    5304 cpu_manager.go:414] "containerMap: removing container" podUID="45f29144-431b-4948-abfa-a1fc5a6847ef" containerName="istio-proxy"
Nov 02 15:19:57.554536 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:19:57.554484    5304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="45f29144-431b-4948-abfa-a1fc5a6847ef" containerName="istio-proxy"
Nov 02 15:19:57.554536 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:19:57.554499    5304 cpu_manager.go:414] "containerMap: removing container" podUID="6f80b62a-49ba-4922-8f3e-804d6bdcc46f" containerName="istio-proxy"
Nov 02 15:19:57.554536 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:19:57.554506    5304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f80b62a-49ba-4922-8f3e-804d6bdcc46f" containerName="istio-proxy"
Nov 02 15:19:57.554536 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:19:57.554519    5304 cpu_manager.go:414] "containerMap: removing container" podUID="a395ec29-6257-43f1-a2f0-1ec07bf215ad" containerName="istio-proxy"
Nov 02 15:19:57.554536 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:19:57.554524    5304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a395ec29-6257-43f1-a2f0-1ec07bf215ad" containerName="istio-proxy"
Nov 02 15:19:57.554684 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:19:57.554552    5304 cpu_manager.go:414] "containerMap: removing container" podUID="ca337066-9ffb-4c9e-beee-fe339577a0b7" containerName="payload-processing"
Nov 02 15:19:57.554684 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:19:57.554557    5304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca337066-9ffb-4c9e-beee-fe339577a0b7" containerName="payload-processing"
Nov 02 15:19:57.554684 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:19:57.554567    5304 cpu_manager.go:414] "containerMap: removing container" podUID="474f879d-3a52-4e47-8eb0-95a4fcfbe399" containerName="istio-proxy"
Nov 02 15:19:57.554684 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:19:57.554572    5304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="474f879d-3a52-4e47-8eb0-95a4fcfbe399" containerName="istio-proxy"
Nov 02 15:19:57.554684 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:19:57.554579    5304 cpu_manager.go:414] "containerMap: removing container" podUID="4f4d49b5-6c2a-4869-a30a-155e3afe755f" containerName="payload-pre-processing"
Nov 02 15:19:57.554684 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:19:57.554583    5304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f4d49b5-6c2a-4869-a30a-155e3afe755f" containerName="payload-pre-processing"
Nov 02 15:19:57.554684 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:19:57.554681    5304 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca337066-9ffb-4c9e-beee-fe339577a0b7" containerName="payload-processing"
Nov 02 15:19:57.554895 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:19:57.554690    5304 memory_manager.go:365] "RemoveStaleState removing state" podUID="474f879d-3a52-4e47-8eb0-95a4fcfbe399" containerName="istio-proxy"
Nov 02 15:19:57.554895 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:19:57.554696    5304 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f4d49b5-6c2a-4869-a30a-155e3afe755f" containerName="payload-pre-processing"
Nov 02 15:19:57.554895 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:19:57.554705    5304 memory_manager.go:365] "RemoveStaleState removing state" podUID="45f29144-431b-4948-abfa-a1fc5a6847ef" containerName="istio-proxy"
Nov 02 15:19:57.554895 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:19:57.554712    5304 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f80b62a-49ba-4922-8f3e-804d6bdcc46f" containerName="istio-proxy"
Nov 02 15:19:57.554895 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:19:57.554717    5304 memory_manager.go:365] "RemoveStaleState removing state" podUID="a395ec29-6257-43f1-a2f0-1ec07bf215ad" containerName="istio-proxy"
Nov 02 15:19:57.558499 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:19:57.558471    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-55e4c3-578b56f96d-lmkft"
Nov 02 15:19:57.577286 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:19:57.577257    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-55e4c3-578b56f96d-lmkft"]
Nov 02 15:19:57.593957 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:19:57.584286    5304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-55e4c3-serving-cert\""
Nov 02 15:19:57.616094 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:19:57.616060    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hbsnz\" (UniqueName: \"kubernetes.io/projected/494bd9cf-9a4d-4448-8850-f746c50b5c8e-kube-api-access-hbsnz\") pod \"maas-api-e2e-api-a-55e4c3-578b56f96d-lmkft\" (UID: \"494bd9cf-9a4d-4448-8850-f746c50b5c8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-55e4c3-578b56f96d-lmkft"
Nov 02 15:19:57.616312 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:19:57.616187    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/494bd9cf-9a4d-4448-8850-f746c50b5c8e-metrics-certs\") pod \"maas-api-e2e-api-a-55e4c3-578b56f96d-lmkft\" (UID: \"494bd9cf-9a4d-4448-8850-f746c50b5c8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-55e4c3-578b56f96d-lmkft"
Nov 02 15:19:57.616384 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:19:57.616365    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/494bd9cf-9a4d-4448-8850-f746c50b5c8e-openshift-service-ca\") pod \"maas-api-e2e-api-a-55e4c3-578b56f96d-lmkft\" (UID: \"494bd9cf-9a4d-4448-8850-f746c50b5c8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-55e4c3-578b56f96d-lmkft"
Nov 02 15:19:57.616460 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:19:57.616389    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/494bd9cf-9a4d-4448-8850-f746c50b5c8e-maas-api-tls\") pod \"maas-api-e2e-api-a-55e4c3-578b56f96d-lmkft\" (UID: \"494bd9cf-9a4d-4448-8850-f746c50b5c8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-55e4c3-578b56f96d-lmkft"
Nov 02 15:19:57.718370 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:19:57.718336    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hbsnz\" (UniqueName: \"kubernetes.io/projected/494bd9cf-9a4d-4448-8850-f746c50b5c8e-kube-api-access-hbsnz\") pod \"maas-api-e2e-api-a-55e4c3-578b56f96d-lmkft\" (UID: \"494bd9cf-9a4d-4448-8850-f746c50b5c8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-55e4c3-578b56f96d-lmkft"
Nov 02 15:19:57.718558 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:19:57.718467    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/494bd9cf-9a4d-4448-8850-f746c50b5c8e-metrics-certs\") pod \"maas-api-e2e-api-a-55e4c3-578b56f96d-lmkft\" (UID: \"494bd9cf-9a4d-4448-8850-f746c50b5c8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-55e4c3-578b56f96d-lmkft"
Nov 02 15:19:57.718628 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:19:57.718610    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/494bd9cf-9a4d-4448-8850-f746c50b5c8e-openshift-service-ca\") pod \"maas-api-e2e-api-a-55e4c3-578b56f96d-lmkft\" (UID: \"494bd9cf-9a4d-4448-8850-f746c50b5c8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-55e4c3-578b56f96d-lmkft"
Nov 02 15:19:57.718684 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:19:57.718632    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/494bd9cf-9a4d-4448-8850-f746c50b5c8e-maas-api-tls\") pod \"maas-api-e2e-api-a-55e4c3-578b56f96d-lmkft\" (UID: \"494bd9cf-9a4d-4448-8850-f746c50b5c8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-55e4c3-578b56f96d-lmkft"
Nov 02 15:19:57.719538 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:19:57.719505    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/494bd9cf-9a4d-4448-8850-f746c50b5c8e-openshift-service-ca\") pod \"maas-api-e2e-api-a-55e4c3-578b56f96d-lmkft\" (UID: \"494bd9cf-9a4d-4448-8850-f746c50b5c8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-55e4c3-578b56f96d-lmkft"
Nov 02 15:19:57.721059 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:19:57.721031    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/494bd9cf-9a4d-4448-8850-f746c50b5c8e-metrics-certs\") pod \"maas-api-e2e-api-a-55e4c3-578b56f96d-lmkft\" (UID: \"494bd9cf-9a4d-4448-8850-f746c50b5c8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-55e4c3-578b56f96d-lmkft"
Nov 02 15:19:57.721275 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:19:57.721039    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/494bd9cf-9a4d-4448-8850-f746c50b5c8e-maas-api-tls\") pod \"maas-api-e2e-api-a-55e4c3-578b56f96d-lmkft\" (UID: \"494bd9cf-9a4d-4448-8850-f746c50b5c8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-55e4c3-578b56f96d-lmkft"
Nov 02 15:19:57.745883 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:19:57.745845    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hbsnz\" (UniqueName: \"kubernetes.io/projected/494bd9cf-9a4d-4448-8850-f746c50b5c8e-kube-api-access-hbsnz\") pod \"maas-api-e2e-api-a-55e4c3-578b56f96d-lmkft\" (UID: \"494bd9cf-9a4d-4448-8850-f746c50b5c8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-55e4c3-578b56f96d-lmkft"
Nov 02 15:19:57.878293 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:19:57.877740    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-55e4c3-578b56f96d-lmkft"
Nov 02 15:19:58.066654 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:19:58.066608    5304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-2bbcb5-67694d7bc-wzlkw"]
Nov 02 15:19:58.071656 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:19:58.071622    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2bbcb5-67694d7bc-wzlkw"
Nov 02 15:19:58.087178 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:19:58.086702    5304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-2bbcb5-dockercfg-vvtzx\""
Nov 02 15:19:58.087178 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:19:58.086798    5304 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-2bbcb5\""
Nov 02 15:19:58.090577 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:19:58.090543    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-2bbcb5-67694d7bc-wzlkw"]
Nov 02 15:19:58.140784 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:19:58.140056    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6bxn5\" (UniqueName: \"kubernetes.io/projected/3567913b-db5f-4a11-9911-9d0fb8dfe57a-kube-api-access-6bxn5\") pod \"payload-pre-processing-e2e-inf-b-2bbcb5-67694d7bc-wzlkw\" (UID: \"3567913b-db5f-4a11-9911-9d0fb8dfe57a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2bbcb5-67694d7bc-wzlkw"
Nov 02 15:19:58.140784 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:19:58.140370    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3567913b-db5f-4a11-9911-9d0fb8dfe57a-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-2bbcb5-67694d7bc-wzlkw\" (UID: \"3567913b-db5f-4a11-9911-9d0fb8dfe57a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2bbcb5-67694d7bc-wzlkw"
Nov 02 15:19:58.266154 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:19:58.264164    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3567913b-db5f-4a11-9911-9d0fb8dfe57a-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-2bbcb5-67694d7bc-wzlkw\" (UID: \"3567913b-db5f-4a11-9911-9d0fb8dfe57a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2bbcb5-67694d7bc-wzlkw"
Nov 02 15:19:58.266154 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:19:58.264455    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6bxn5\" (UniqueName: \"kubernetes.io/projected/3567913b-db5f-4a11-9911-9d0fb8dfe57a-kube-api-access-6bxn5\") pod \"payload-pre-processing-e2e-inf-b-2bbcb5-67694d7bc-wzlkw\" (UID: \"3567913b-db5f-4a11-9911-9d0fb8dfe57a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2bbcb5-67694d7bc-wzlkw"
Nov 02 15:19:58.294197 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:19:58.291030    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3567913b-db5f-4a11-9911-9d0fb8dfe57a-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-2bbcb5-67694d7bc-wzlkw\" (UID: \"3567913b-db5f-4a11-9911-9d0fb8dfe57a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2bbcb5-67694d7bc-wzlkw"
Nov 02 15:19:58.383201 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:19:58.383167    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6bxn5\" (UniqueName: \"kubernetes.io/projected/3567913b-db5f-4a11-9911-9d0fb8dfe57a-kube-api-access-6bxn5\") pod \"payload-pre-processing-e2e-inf-b-2bbcb5-67694d7bc-wzlkw\" (UID: \"3567913b-db5f-4a11-9911-9d0fb8dfe57a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2bbcb5-67694d7bc-wzlkw"
Nov 02 15:19:58.423192 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:19:58.422466    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2bbcb5-67694d7bc-wzlkw"
Nov 02 15:19:58.480322 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:19:58.480286    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-55e4c3-578b56f96d-lmkft"]
Nov 02 15:19:58.532049 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:19:58.531966    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-55e4c3-578b56f96d-lmkft" event={"ID":"494bd9cf-9a4d-4448-8850-f746c50b5c8e","Type":"ContainerStarted","Data":"3f822bac681f82093194c1e4bb690c7f4ce489ae2ca379572f0d8386fa96b5b3"}
Nov 02 15:19:58.803168 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:19:58.794350    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-2bbcb5-67694d7bc-wzlkw"]
Nov 02 15:19:58.803168 ip-10-0-135-138 kubenswrapper[5304]: W1102 15:19:58.795760    5304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3567913b_db5f_4a11_9911_9d0fb8dfe57a.slice/crio-f7fa0aeb6cff2864e605b03207678be4c8a4f13d060610d3203dccaa2e0b11fa WatchSource:0}: Error finding container f7fa0aeb6cff2864e605b03207678be4c8a4f13d060610d3203dccaa2e0b11fa: Status 404 returned error can't find the container with id f7fa0aeb6cff2864e605b03207678be4c8a4f13d060610d3203dccaa2e0b11fa
Nov 02 15:19:59.540833 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:19:59.540789    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-55e4c3-578b56f96d-lmkft" event={"ID":"494bd9cf-9a4d-4448-8850-f746c50b5c8e","Type":"ContainerStarted","Data":"eab6fd8ca782f752159b380bc059b5ea2125f8eb5edf3fde9493d0a11d2cb00a"}
Nov 02 15:19:59.541036 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:19:59.540886    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-55e4c3-578b56f96d-lmkft"
Nov 02 15:19:59.542702 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:19:59.542678    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2bbcb5-67694d7bc-wzlkw" event={"ID":"3567913b-db5f-4a11-9911-9d0fb8dfe57a","Type":"ContainerStarted","Data":"f35301c6ea2b8f1aaac76746e761fe2fcdbc8deaeec1dfea81a78caeebad0b0d"}
Nov 02 15:19:59.542818 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:19:59.542717    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2bbcb5-67694d7bc-wzlkw" event={"ID":"3567913b-db5f-4a11-9911-9d0fb8dfe57a","Type":"ContainerStarted","Data":"f7fa0aeb6cff2864e605b03207678be4c8a4f13d060610d3203dccaa2e0b11fa"}
Nov 02 15:19:59.542818 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:19:59.542771    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2bbcb5-67694d7bc-wzlkw"
Nov 02 15:19:59.590011 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:19:59.589957    5304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-55e4c3-578b56f96d-lmkft" podStartSLOduration=2.122646925 podStartE2EDuration="2.58994223s" totalImagesPullingTime="467.295305ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:19:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:19:59.586331358 +0000 UTC m=+1399.580155766" watchObservedRunningTime="2026-11-02 15:19:59.58994223 +0000 UTC m=+1399.583766639"
Nov 02 15:19:59.628255 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:19:59.628188    5304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2bbcb5-67694d7bc-wzlkw" podStartSLOduration=1.628168914 podStartE2EDuration="1.628168914s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:19:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:19:59.625630874 +0000 UTC m=+1399.619455278" watchObservedRunningTime="2026-11-02 15:19:59.628168914 +0000 UTC m=+1399.621993331"
Nov 02 15:20:01.569148 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:20:01.568578    5304 generic.go:356] "Generic (PLEG): container finished" podID="9b33eaba-53ea-47b6-bdfe-3d3d9ffd8a36" containerID="f48f93ebb35e8494ee80b0db35e6df19cf522c1d5e7a8601b57ef28149a900bb" exitCode=0
Nov 02 15:20:01.569148 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:20:01.568745    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a713aa-openshift-default-7554947cc9-trs9t" event={"ID":"9b33eaba-53ea-47b6-bdfe-3d3d9ffd8a36","Type":"ContainerDied","Data":"f48f93ebb35e8494ee80b0db35e6df19cf522c1d5e7a8601b57ef28149a900bb"}
Nov 02 15:20:01.569772 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:20:01.569302    5304 scope.go:122] "RemoveContainer" containerID="f48f93ebb35e8494ee80b0db35e6df19cf522c1d5e7a8601b57ef28149a900bb"
Nov 02 15:20:01.577691 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:20:01.573356    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:20:01.577691 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:20:01.573450    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:20:01.577691 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:20:01.573513    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:20:02.578838 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:20:02.578803    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a713aa-openshift-default-7554947cc9-trs9t" event={"ID":"9b33eaba-53ea-47b6-bdfe-3d3d9ffd8a36","Type":"ContainerStarted","Data":"386621ec229173080d14b76e5daaeefd5b35a1b33aeca89d3fecf66ae38a29bf"}
Nov 02 15:20:03.112901 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:20:03.112863    5304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-a713aa-openshift-default-7554947cc9-trs9t"
Nov 02 15:20:03.114020 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:20:03.113992    5304 patch_prober.go:29] interesting pod/e2e-inf-a-a713aa-openshift-default-7554947cc9-trs9t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" start-of-body=
Nov 02 15:20:03.114159 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:20:03.114046    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a713aa-openshift-default-7554947cc9-trs9t" podUID="9b33eaba-53ea-47b6-bdfe-3d3d9ffd8a36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused"
Nov 02 15:20:04.112255 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:20:04.112214    5304 patch_prober.go:29] interesting pod/e2e-inf-a-a713aa-openshift-default-7554947cc9-trs9t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" start-of-body=
Nov 02 15:20:04.112767 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:20:04.112266    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a713aa-openshift-default-7554947cc9-trs9t" podUID="9b33eaba-53ea-47b6-bdfe-3d3d9ffd8a36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused"
Nov 02 15:20:05.112693 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:20:05.112658    5304 patch_prober.go:29] interesting pod/e2e-inf-a-a713aa-openshift-default-7554947cc9-trs9t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" start-of-body=
Nov 02 15:20:05.113090 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:20:05.112711    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a713aa-openshift-default-7554947cc9-trs9t" podUID="9b33eaba-53ea-47b6-bdfe-3d3d9ffd8a36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused"
Nov 02 15:20:05.560563 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:20:05.560485    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-55e4c3-578b56f96d-lmkft"
Nov 02 15:20:06.113140 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:20:06.113081    5304 patch_prober.go:29] interesting pod/e2e-inf-a-a713aa-openshift-default-7554947cc9-trs9t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" start-of-body=
Nov 02 15:20:06.113547 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:20:06.113176    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a713aa-openshift-default-7554947cc9-trs9t" podUID="9b33eaba-53ea-47b6-bdfe-3d3d9ffd8a36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused"
Nov 02 15:20:07.113088 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:20:07.113055    5304 patch_prober.go:29] interesting pod/e2e-inf-a-a713aa-openshift-default-7554947cc9-trs9t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" start-of-body=
Nov 02 15:20:07.113299 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:20:07.113121    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a713aa-openshift-default-7554947cc9-trs9t" podUID="9b33eaba-53ea-47b6-bdfe-3d3d9ffd8a36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused"
Nov 02 15:20:07.422951 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:20:07.422872    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-a713aa-openshift-default-7554947cc9-trs9t"
Nov 02 15:20:08.120608 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:20:08.120575    5304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-a713aa-openshift-default-7554947cc9-trs9t"
Nov 02 15:20:08.121845 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:20:08.121823    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-a713aa-openshift-default-7554947cc9-trs9t"
Nov 02 15:20:10.347354 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:20:10.347317    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-ad276f-openshift-default-7cb6d54d77-rps78"]
Nov 02 15:20:10.347737 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:20:10.347535    5304 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-ad276f-openshift-default-7cb6d54d77-rps78" podUID="350b279b-88a8-4318-b49c-d5001077f702" containerName="istio-proxy" containerID="cri-o://04794ea5bf11354bb2d6fa52c5c14cc251ecd5c4fb600f33337bf644e004114b" gracePeriod=30
Nov 02 15:20:10.587159 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:20:10.579664    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2bbcb5-67694d7bc-wzlkw"
Nov 02 15:20:15.774016 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:20:15.773977    5304 generic.go:356] "Generic (PLEG): container finished" podID="350b279b-88a8-4318-b49c-d5001077f702" containerID="04794ea5bf11354bb2d6fa52c5c14cc251ecd5c4fb600f33337bf644e004114b" exitCode=0
Nov 02 15:20:15.774991 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:20:15.774964    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-ad276f-openshift-default-7cb6d54d77-rps78" event={"ID":"350b279b-88a8-4318-b49c-d5001077f702","Type":"ContainerDied","Data":"04794ea5bf11354bb2d6fa52c5c14cc251ecd5c4fb600f33337bf644e004114b"}
Nov 02 15:20:15.775219 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:20:15.775200    5304 scope.go:122] "RemoveContainer" containerID="68e340f6a7203903e085d2bee7707621c57f9f1d6b1e7b75aa965438a9764cba"
Nov 02 15:20:16.317850 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:20:16.317737    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-ad276f-openshift-default-7cb6d54d77-rps78"
Nov 02 15:20:16.415671 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:20:16.409745    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/350b279b-88a8-4318-b49c-d5001077f702-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/350b279b-88a8-4318-b49c-d5001077f702-credential-socket\") pod \"350b279b-88a8-4318-b49c-d5001077f702\" (UID: \"350b279b-88a8-4318-b49c-d5001077f702\") "
Nov 02 15:20:16.415671 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:20:16.409815    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/350b279b-88a8-4318-b49c-d5001077f702-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/350b279b-88a8-4318-b49c-d5001077f702-istio-podinfo\") pod \"350b279b-88a8-4318-b49c-d5001077f702\" (UID: \"350b279b-88a8-4318-b49c-d5001077f702\") "
Nov 02 15:20:16.415671 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:20:16.409980    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/350b279b-88a8-4318-b49c-d5001077f702-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/350b279b-88a8-4318-b49c-d5001077f702-workload-certs\") pod \"350b279b-88a8-4318-b49c-d5001077f702\" (UID: \"350b279b-88a8-4318-b49c-d5001077f702\") "
Nov 02 15:20:16.415671 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:20:16.410012    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/350b279b-88a8-4318-b49c-d5001077f702-kube-api-access-qdp4n\" (UniqueName: \"kubernetes.io/projected/350b279b-88a8-4318-b49c-d5001077f702-kube-api-access-qdp4n\") pod \"350b279b-88a8-4318-b49c-d5001077f702\" (UID: \"350b279b-88a8-4318-b49c-d5001077f702\") "
Nov 02 15:20:16.415671 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:20:16.410576    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/350b279b-88a8-4318-b49c-d5001077f702-credential-socket" pod "350b279b-88a8-4318-b49c-d5001077f702" (UID: "350b279b-88a8-4318-b49c-d5001077f702"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:20:16.415671 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:20:16.410816    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/350b279b-88a8-4318-b49c-d5001077f702-workload-certs" pod "350b279b-88a8-4318-b49c-d5001077f702" (UID: "350b279b-88a8-4318-b49c-d5001077f702"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:20:16.415671 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:20:16.410912    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/350b279b-88a8-4318-b49c-d5001077f702-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/350b279b-88a8-4318-b49c-d5001077f702-istio-data\") pod \"350b279b-88a8-4318-b49c-d5001077f702\" (UID: \"350b279b-88a8-4318-b49c-d5001077f702\") "
Nov 02 15:20:16.415671 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:20:16.410997    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/350b279b-88a8-4318-b49c-d5001077f702-istio-token\" (UniqueName: \"kubernetes.io/projected/350b279b-88a8-4318-b49c-d5001077f702-istio-token\") pod \"350b279b-88a8-4318-b49c-d5001077f702\" (UID: \"350b279b-88a8-4318-b49c-d5001077f702\") "
Nov 02 15:20:16.415671 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:20:16.411146    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/350b279b-88a8-4318-b49c-d5001077f702-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/350b279b-88a8-4318-b49c-d5001077f702-workload-socket\") pod \"350b279b-88a8-4318-b49c-d5001077f702\" (UID: \"350b279b-88a8-4318-b49c-d5001077f702\") "
Nov 02 15:20:16.415671 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:20:16.411242    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/350b279b-88a8-4318-b49c-d5001077f702-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/350b279b-88a8-4318-b49c-d5001077f702-istiod-ca-cert\") pod \"350b279b-88a8-4318-b49c-d5001077f702\" (UID: \"350b279b-88a8-4318-b49c-d5001077f702\") "
Nov 02 15:20:16.415671 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:20:16.411366    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/350b279b-88a8-4318-b49c-d5001077f702-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/350b279b-88a8-4318-b49c-d5001077f702-istio-envoy\") pod \"350b279b-88a8-4318-b49c-d5001077f702\" (UID: \"350b279b-88a8-4318-b49c-d5001077f702\") "
Nov 02 15:20:16.415671 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:20:16.412395    5304 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/350b279b-88a8-4318-b49c-d5001077f702-credential-socket\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:20:16.415671 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:20:16.412428    5304 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/350b279b-88a8-4318-b49c-d5001077f702-workload-certs\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:20:16.415671 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:20:16.413756    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/350b279b-88a8-4318-b49c-d5001077f702-istio-data" pod "350b279b-88a8-4318-b49c-d5001077f702" (UID: "350b279b-88a8-4318-b49c-d5001077f702"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:20:16.415671 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:20:16.414596    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/350b279b-88a8-4318-b49c-d5001077f702-istiod-ca-cert" pod "350b279b-88a8-4318-b49c-d5001077f702" (UID: "350b279b-88a8-4318-b49c-d5001077f702"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:20:16.415671 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:20:16.414702    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/350b279b-88a8-4318-b49c-d5001077f702-workload-socket" pod "350b279b-88a8-4318-b49c-d5001077f702" (UID: "350b279b-88a8-4318-b49c-d5001077f702"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:20:16.415671 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:20:16.415468    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/350b279b-88a8-4318-b49c-d5001077f702-kube-api-access-qdp4n" pod "350b279b-88a8-4318-b49c-d5001077f702" (UID: "350b279b-88a8-4318-b49c-d5001077f702"). InnerVolumeSpecName "kube-api-access-qdp4n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:20:16.418747 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:20:16.418716    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/350b279b-88a8-4318-b49c-d5001077f702-istio-token" pod "350b279b-88a8-4318-b49c-d5001077f702" (UID: "350b279b-88a8-4318-b49c-d5001077f702"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:20:16.419459 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:20:16.419357    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/350b279b-88a8-4318-b49c-d5001077f702-istio-podinfo" pod "350b279b-88a8-4318-b49c-d5001077f702" (UID: "350b279b-88a8-4318-b49c-d5001077f702"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 15:20:16.420612 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:20:16.420584    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/350b279b-88a8-4318-b49c-d5001077f702-istio-envoy" pod "350b279b-88a8-4318-b49c-d5001077f702" (UID: "350b279b-88a8-4318-b49c-d5001077f702"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:20:16.514294 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:20:16.514208    5304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qdp4n\" (UniqueName: \"kubernetes.io/projected/350b279b-88a8-4318-b49c-d5001077f702-kube-api-access-qdp4n\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:20:16.514294 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:20:16.514238    5304 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/350b279b-88a8-4318-b49c-d5001077f702-istio-data\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:20:16.514294 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:20:16.514248    5304 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/350b279b-88a8-4318-b49c-d5001077f702-istio-token\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:20:16.514294 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:20:16.514256    5304 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/350b279b-88a8-4318-b49c-d5001077f702-workload-socket\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:20:16.514294 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:20:16.514267    5304 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/350b279b-88a8-4318-b49c-d5001077f702-istiod-ca-cert\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:20:16.514294 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:20:16.514276    5304 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/350b279b-88a8-4318-b49c-d5001077f702-istio-envoy\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:20:16.514294 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:20:16.514283    5304 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/350b279b-88a8-4318-b49c-d5001077f702-istio-podinfo\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:20:16.811773 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:20:16.811690    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-ad276f-openshift-default-7cb6d54d77-rps78"
Nov 02 15:20:16.811773 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:20:16.811709    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-ad276f-openshift-default-7cb6d54d77-rps78" event={"ID":"350b279b-88a8-4318-b49c-d5001077f702","Type":"ContainerDied","Data":"ae449b9da32762514648fbff42087c42dd7799f09a52a50b005cffa6749ce176"}
Nov 02 15:20:16.812459 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:20:16.811792    5304 scope.go:122] "RemoveContainer" containerID="04794ea5bf11354bb2d6fa52c5c14cc251ecd5c4fb600f33337bf644e004114b"
Nov 02 15:20:16.866828 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:20:16.866792    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-ad276f-openshift-default-7cb6d54d77-rps78"]
Nov 02 15:20:16.870416 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:20:16.870387    5304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-ad276f-openshift-default-7cb6d54d77-rps78"]
Nov 02 15:20:18.120451 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:20:18.120407    5304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="350b279b-88a8-4318-b49c-d5001077f702" path="/var/lib/kubelet/pods/350b279b-88a8-4318-b49c-d5001077f702/volumes"
Nov 02 15:20:21.585327 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:20:21.585278    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-5f9166-8c7954fff-tq5g2"]
Nov 02 15:20:21.624531 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:20:21.624497    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-5f9166-85b8cf5dc-2q6d2"]
Nov 02 15:20:24.731190 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:20:24.731159    5304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-1cdbe6-6ccf98cccf-76jxh"]
Nov 02 15:20:24.731880 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:20:24.731863    5304 cpu_manager.go:414] "containerMap: removing container" podUID="350b279b-88a8-4318-b49c-d5001077f702" containerName="istio-proxy"
Nov 02 15:20:24.731945 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:20:24.731882    5304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="350b279b-88a8-4318-b49c-d5001077f702" containerName="istio-proxy"
Nov 02 15:20:24.732058 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:20:24.732048    5304 memory_manager.go:365] "RemoveStaleState removing state" podUID="350b279b-88a8-4318-b49c-d5001077f702" containerName="istio-proxy"
Nov 02 15:20:24.735895 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:20:24.735875    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-1cdbe6-6ccf98cccf-76jxh"
Nov 02 15:20:24.753820 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:20:24.753787    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-1cdbe6-6ccf98cccf-76jxh"]
Nov 02 15:20:24.761488 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:20:24.761462    5304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-1cdbe6-dockercfg-8cwcw\""
Nov 02 15:20:24.761673 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:20:24.761479    5304 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-1cdbe6\""
Nov 02 15:20:24.810473 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:20:24.810167    5304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-1cdbe6-86f44f7857-x9klg"]
Nov 02 15:20:24.839152 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:20:24.814862    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q96nx\" (UniqueName: \"kubernetes.io/projected/fc17910d-68da-47a3-95c6-441451c71378-kube-api-access-q96nx\") pod \"payload-pre-processing-e2e-api-a-1cdbe6-6ccf98cccf-76jxh\" (UID: \"fc17910d-68da-47a3-95c6-441451c71378\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-1cdbe6-6ccf98cccf-76jxh"
Nov 02 15:20:24.839152 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:20:24.816235    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc17910d-68da-47a3-95c6-441451c71378-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-1cdbe6-6ccf98cccf-76jxh\" (UID: \"fc17910d-68da-47a3-95c6-441451c71378\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-1cdbe6-6ccf98cccf-76jxh"
Nov 02 15:20:24.839152 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:20:24.817256    5304 cpu_manager.go:414] "containerMap: removing container" podUID="350b279b-88a8-4318-b49c-d5001077f702" containerName="istio-proxy"
Nov 02 15:20:24.839152 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:20:24.817277    5304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="350b279b-88a8-4318-b49c-d5001077f702" containerName="istio-proxy"
Nov 02 15:20:24.839152 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:20:24.817495    5304 memory_manager.go:365] "RemoveStaleState removing state" podUID="350b279b-88a8-4318-b49c-d5001077f702" containerName="istio-proxy"
Nov 02 15:20:24.884148 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:20:24.871871    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-1cdbe6-86f44f7857-x9klg"
Nov 02 15:20:24.936388 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:20:24.936360    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-1cdbe6-86f44f7857-x9klg"]
Nov 02 15:20:24.984834 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:20:24.966426    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lv7hr\" (UniqueName: \"kubernetes.io/projected/1c24b5d7-d4a5-46bb-94f4-22046cc2e27b-kube-api-access-lv7hr\") pod \"payload-processing-e2e-api-a-1cdbe6-86f44f7857-x9klg\" (UID: \"1c24b5d7-d4a5-46bb-94f4-22046cc2e27b\") " pod="openshift-ingress/payload-processing-e2e-api-a-1cdbe6-86f44f7857-x9klg"
Nov 02 15:20:24.984834 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:20:24.966562    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c24b5d7-d4a5-46bb-94f4-22046cc2e27b-plugins-config-volume\") pod \"payload-processing-e2e-api-a-1cdbe6-86f44f7857-x9klg\" (UID: \"1c24b5d7-d4a5-46bb-94f4-22046cc2e27b\") " pod="openshift-ingress/payload-processing-e2e-api-a-1cdbe6-86f44f7857-x9klg"
Nov 02 15:20:24.984834 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:20:24.966666    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q96nx\" (UniqueName: \"kubernetes.io/projected/fc17910d-68da-47a3-95c6-441451c71378-kube-api-access-q96nx\") pod \"payload-pre-processing-e2e-api-a-1cdbe6-6ccf98cccf-76jxh\" (UID: \"fc17910d-68da-47a3-95c6-441451c71378\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-1cdbe6-6ccf98cccf-76jxh"
Nov 02 15:20:24.984834 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:20:24.966804    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc17910d-68da-47a3-95c6-441451c71378-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-1cdbe6-6ccf98cccf-76jxh\" (UID: \"fc17910d-68da-47a3-95c6-441451c71378\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-1cdbe6-6ccf98cccf-76jxh"
Nov 02 15:20:24.984834 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:20:24.967761    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc17910d-68da-47a3-95c6-441451c71378-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-1cdbe6-6ccf98cccf-76jxh\" (UID: \"fc17910d-68da-47a3-95c6-441451c71378\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-1cdbe6-6ccf98cccf-76jxh"
Nov 02 15:20:25.037221 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:20:25.037147    5304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k6fjk" containerName="extractor"
Nov 02 15:20:25.063179 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:20:25.062877    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q96nx\" (UniqueName: \"kubernetes.io/projected/fc17910d-68da-47a3-95c6-441451c71378-kube-api-access-q96nx\") pod \"payload-pre-processing-e2e-api-a-1cdbe6-6ccf98cccf-76jxh\" (UID: \"fc17910d-68da-47a3-95c6-441451c71378\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-1cdbe6-6ccf98cccf-76jxh"
Nov 02 15:20:25.068168 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:20:25.068050    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lv7hr\" (UniqueName: \"kubernetes.io/projected/1c24b5d7-d4a5-46bb-94f4-22046cc2e27b-kube-api-access-lv7hr\") pod \"payload-processing-e2e-api-a-1cdbe6-86f44f7857-x9klg\" (UID: \"1c24b5d7-d4a5-46bb-94f4-22046cc2e27b\") " pod="openshift-ingress/payload-processing-e2e-api-a-1cdbe6-86f44f7857-x9klg"
Nov 02 15:20:25.068537 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:20:25.068501    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c24b5d7-d4a5-46bb-94f4-22046cc2e27b-plugins-config-volume\") pod \"payload-processing-e2e-api-a-1cdbe6-86f44f7857-x9klg\" (UID: \"1c24b5d7-d4a5-46bb-94f4-22046cc2e27b\") " pod="openshift-ingress/payload-processing-e2e-api-a-1cdbe6-86f44f7857-x9klg"
Nov 02 15:20:25.069841 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:20:25.069808    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c24b5d7-d4a5-46bb-94f4-22046cc2e27b-plugins-config-volume\") pod \"payload-processing-e2e-api-a-1cdbe6-86f44f7857-x9klg\" (UID: \"1c24b5d7-d4a5-46bb-94f4-22046cc2e27b\") " pod="openshift-ingress/payload-processing-e2e-api-a-1cdbe6-86f44f7857-x9klg"
Nov 02 15:20:25.101605 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:20:25.101571    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lv7hr\" (UniqueName: \"kubernetes.io/projected/1c24b5d7-d4a5-46bb-94f4-22046cc2e27b-kube-api-access-lv7hr\") pod \"payload-processing-e2e-api-a-1cdbe6-86f44f7857-x9klg\" (UID: \"1c24b5d7-d4a5-46bb-94f4-22046cc2e27b\") " pod="openshift-ingress/payload-processing-e2e-api-a-1cdbe6-86f44f7857-x9klg"
Nov 02 15:20:25.349809 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:20:25.349778    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-1cdbe6-6ccf98cccf-76jxh"
Nov 02 15:20:25.353074 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:20:25.353045    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-1cdbe6-86f44f7857-x9klg"
Nov 02 15:20:25.669220 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:20:25.669197    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-1cdbe6-6ccf98cccf-76jxh"]
Nov 02 15:20:25.671504 ip-10-0-135-138 kubenswrapper[5304]: W1102 15:20:25.671471    5304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc17910d_68da_47a3_95c6_441451c71378.slice/crio-c8e59f425821351e6d624e7eb82490fff168762995028f178e144335873c4ec7 WatchSource:0}: Error finding container c8e59f425821351e6d624e7eb82490fff168762995028f178e144335873c4ec7: Status 404 returned error can't find the container with id c8e59f425821351e6d624e7eb82490fff168762995028f178e144335873c4ec7
Nov 02 15:20:25.694489 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:20:25.694466    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-1cdbe6-86f44f7857-x9klg"]
Nov 02 15:20:25.696666 ip-10-0-135-138 kubenswrapper[5304]: W1102 15:20:25.696639    5304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c24b5d7_d4a5_46bb_94f4_22046cc2e27b.slice/crio-146a8e7dd4ab461fa1986160129f5558bec6e6085c623585a9b5586836ca7ccc WatchSource:0}: Error finding container 146a8e7dd4ab461fa1986160129f5558bec6e6085c623585a9b5586836ca7ccc: Status 404 returned error can't find the container with id 146a8e7dd4ab461fa1986160129f5558bec6e6085c623585a9b5586836ca7ccc
Nov 02 15:20:26.048818 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:20:26.048721    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-1cdbe6-86f44f7857-x9klg" event={"ID":"1c24b5d7-d4a5-46bb-94f4-22046cc2e27b","Type":"ContainerStarted","Data":"61b174fa9522bfa0df00475401a31621d6807550614d97a3f90c0fe2cebbfe88"}
Nov 02 15:20:26.048818 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:20:26.048778    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-1cdbe6-86f44f7857-x9klg" event={"ID":"1c24b5d7-d4a5-46bb-94f4-22046cc2e27b","Type":"ContainerStarted","Data":"146a8e7dd4ab461fa1986160129f5558bec6e6085c623585a9b5586836ca7ccc"}
Nov 02 15:20:26.049424 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:20:26.048817    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-1cdbe6-86f44f7857-x9klg"
Nov 02 15:20:26.050539 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:20:26.050514    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-1cdbe6-6ccf98cccf-76jxh" event={"ID":"fc17910d-68da-47a3-95c6-441451c71378","Type":"ContainerStarted","Data":"8c4168eecadd00915928f034e3fed0cf2ee9d1a89eb8f40a7df73f4675cb3d0a"}
Nov 02 15:20:26.050638 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:20:26.050563    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-1cdbe6-6ccf98cccf-76jxh"
Nov 02 15:20:26.050638 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:20:26.050584    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-1cdbe6-6ccf98cccf-76jxh" event={"ID":"fc17910d-68da-47a3-95c6-441451c71378","Type":"ContainerStarted","Data":"c8e59f425821351e6d624e7eb82490fff168762995028f178e144335873c4ec7"}
Nov 02 15:20:26.095436 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:20:26.095324    5304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-1cdbe6-86f44f7857-x9klg" podStartSLOduration=2.095309681 podStartE2EDuration="2.095309681s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:20:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:20:26.093234164 +0000 UTC m=+1426.087058573" watchObservedRunningTime="2026-11-02 15:20:26.095309681 +0000 UTC m=+1426.089134387"
Nov 02 15:20:26.136842 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:20:26.136792    5304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-1cdbe6-6ccf98cccf-76jxh" podStartSLOduration=2.1367775780000002 podStartE2EDuration="2.136777578s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:20:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:20:26.136770954 +0000 UTC m=+1426.130595364" watchObservedRunningTime="2026-11-02 15:20:26.136777578 +0000 UTC m=+1426.130601987"
Nov 02 15:20:26.586336 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:20:26.586294    5304 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-5f9166-8c7954fff-tq5g2" podUID="b3bdd3ae-9a9c-4945-b820-96d56f208aa3" containerName="payload-processing" containerID="cri-o://2078e55337110072c4297383f22c973ac713512124e9c54753772d7291891809" gracePeriod=25
Nov 02 15:20:26.625215 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:20:26.625177    5304 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-5f9166-85b8cf5dc-2q6d2" podUID="c6530de8-45d4-4be1-9c37-e4c29d8a3ace" containerName="payload-pre-processing" containerID="cri-o://d614fd9c13ea98f71be766c10b51acda536172b0177ed930a069944922f2e3c3" gracePeriod=25
Nov 02 15:20:26.922106 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:20:26.922080    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-5f9166-8c7954fff-tq5g2"
Nov 02 15:20:26.939188 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:20:26.939159    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-5f9166-85b8cf5dc-2q6d2"
Nov 02 15:20:27.000067 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:20:27.000035    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c6530de8-45d4-4be1-9c37-e4c29d8a3ace-kube-api-access-tdlhh\" (UniqueName: \"kubernetes.io/projected/c6530de8-45d4-4be1-9c37-e4c29d8a3ace-kube-api-access-tdlhh\") pod \"c6530de8-45d4-4be1-9c37-e4c29d8a3ace\" (UID: \"c6530de8-45d4-4be1-9c37-e4c29d8a3ace\") "
Nov 02 15:20:27.000319 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:20:27.000119    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b3bdd3ae-9a9c-4945-b820-96d56f208aa3-kube-api-access-glzsf\" (UniqueName: \"kubernetes.io/projected/b3bdd3ae-9a9c-4945-b820-96d56f208aa3-kube-api-access-glzsf\") pod \"b3bdd3ae-9a9c-4945-b820-96d56f208aa3\" (UID: \"b3bdd3ae-9a9c-4945-b820-96d56f208aa3\") "
Nov 02 15:20:27.000319 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:20:27.000172    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c6530de8-45d4-4be1-9c37-e4c29d8a3ace-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6530de8-45d4-4be1-9c37-e4c29d8a3ace-plugins-config-volume\") pod \"c6530de8-45d4-4be1-9c37-e4c29d8a3ace\" (UID: \"c6530de8-45d4-4be1-9c37-e4c29d8a3ace\") "
Nov 02 15:20:27.000462 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:20:27.000356    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b3bdd3ae-9a9c-4945-b820-96d56f208aa3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3bdd3ae-9a9c-4945-b820-96d56f208aa3-plugins-config-volume\") pod \"b3bdd3ae-9a9c-4945-b820-96d56f208aa3\" (UID: \"b3bdd3ae-9a9c-4945-b820-96d56f208aa3\") "
Nov 02 15:20:27.000745 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:20:27.000712    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6530de8-45d4-4be1-9c37-e4c29d8a3ace-plugins-config-volume" pod "c6530de8-45d4-4be1-9c37-e4c29d8a3ace" (UID: "c6530de8-45d4-4be1-9c37-e4c29d8a3ace"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:20:27.000859 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:20:27.000771    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3bdd3ae-9a9c-4945-b820-96d56f208aa3-plugins-config-volume" pod "b3bdd3ae-9a9c-4945-b820-96d56f208aa3" (UID: "b3bdd3ae-9a9c-4945-b820-96d56f208aa3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:20:27.001245 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:20:27.001220    5304 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6530de8-45d4-4be1-9c37-e4c29d8a3ace-plugins-config-volume\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:20:27.001357 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:20:27.001248    5304 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3bdd3ae-9a9c-4945-b820-96d56f208aa3-plugins-config-volume\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:20:27.002505 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:20:27.002473    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6530de8-45d4-4be1-9c37-e4c29d8a3ace-kube-api-access-tdlhh" pod "c6530de8-45d4-4be1-9c37-e4c29d8a3ace" (UID: "c6530de8-45d4-4be1-9c37-e4c29d8a3ace"). InnerVolumeSpecName "kube-api-access-tdlhh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:20:27.002604 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:20:27.002528    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3bdd3ae-9a9c-4945-b820-96d56f208aa3-kube-api-access-glzsf" pod "b3bdd3ae-9a9c-4945-b820-96d56f208aa3" (UID: "b3bdd3ae-9a9c-4945-b820-96d56f208aa3"). InnerVolumeSpecName "kube-api-access-glzsf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:20:27.061623 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:20:27.061586    5304 generic.go:356] "Generic (PLEG): container finished" podID="c6530de8-45d4-4be1-9c37-e4c29d8a3ace" containerID="d614fd9c13ea98f71be766c10b51acda536172b0177ed930a069944922f2e3c3" exitCode=0
Nov 02 15:20:27.062086 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:20:27.061640    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-5f9166-85b8cf5dc-2q6d2"
Nov 02 15:20:27.062086 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:20:27.061671    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-5f9166-85b8cf5dc-2q6d2" event={"ID":"c6530de8-45d4-4be1-9c37-e4c29d8a3ace","Type":"ContainerDied","Data":"d614fd9c13ea98f71be766c10b51acda536172b0177ed930a069944922f2e3c3"}
Nov 02 15:20:27.062086 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:20:27.061728    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-5f9166-85b8cf5dc-2q6d2" event={"ID":"c6530de8-45d4-4be1-9c37-e4c29d8a3ace","Type":"ContainerDied","Data":"36ff9910bebed1bcb612df23d3d335b4173999e7802778397635f7d53b2937ef"}
Nov 02 15:20:27.062086 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:20:27.061757    5304 scope.go:122] "RemoveContainer" containerID="d614fd9c13ea98f71be766c10b51acda536172b0177ed930a069944922f2e3c3"
Nov 02 15:20:27.063420 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:20:27.063397    5304 generic.go:356] "Generic (PLEG): container finished" podID="b3bdd3ae-9a9c-4945-b820-96d56f208aa3" containerID="2078e55337110072c4297383f22c973ac713512124e9c54753772d7291891809" exitCode=0
Nov 02 15:20:27.063589 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:20:27.063438    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-5f9166-8c7954fff-tq5g2"
Nov 02 15:20:27.063589 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:20:27.063465    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-5f9166-8c7954fff-tq5g2" event={"ID":"b3bdd3ae-9a9c-4945-b820-96d56f208aa3","Type":"ContainerDied","Data":"2078e55337110072c4297383f22c973ac713512124e9c54753772d7291891809"}
Nov 02 15:20:27.063589 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:20:27.063511    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-5f9166-8c7954fff-tq5g2" event={"ID":"b3bdd3ae-9a9c-4945-b820-96d56f208aa3","Type":"ContainerDied","Data":"bfac8978a576d6c7fc7042fe18d24b6dacf0c506e26f17be99abae82e26d0adb"}
Nov 02 15:20:27.088713 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:20:27.088692    5304 scope.go:122] "RemoveContainer" containerID="d614fd9c13ea98f71be766c10b51acda536172b0177ed930a069944922f2e3c3"
Nov 02 15:20:27.089039 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:20:27.089012    5304 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d614fd9c13ea98f71be766c10b51acda536172b0177ed930a069944922f2e3c3\": container with ID starting with d614fd9c13ea98f71be766c10b51acda536172b0177ed930a069944922f2e3c3 not found: ID does not exist" containerID="d614fd9c13ea98f71be766c10b51acda536172b0177ed930a069944922f2e3c3"
Nov 02 15:20:27.089088 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:20:27.089046    5304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d614fd9c13ea98f71be766c10b51acda536172b0177ed930a069944922f2e3c3"} err="failed to get container status \"d614fd9c13ea98f71be766c10b51acda536172b0177ed930a069944922f2e3c3\": rpc error: code = NotFound desc = could not find container \"d614fd9c13ea98f71be766c10b51acda536172b0177ed930a069944922f2e3c3\": container with ID starting with d614fd9c13ea98f71be766c10b51acda536172b0177ed930a069944922f2e3c3 not found: ID does not exist"
Nov 02 15:20:27.089088 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:20:27.089064    5304 scope.go:122] "RemoveContainer" containerID="2078e55337110072c4297383f22c973ac713512124e9c54753772d7291891809"
Nov 02 15:20:27.098793 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:20:27.098773    5304 scope.go:122] "RemoveContainer" containerID="2078e55337110072c4297383f22c973ac713512124e9c54753772d7291891809"
Nov 02 15:20:27.099093 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:20:27.099071    5304 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2078e55337110072c4297383f22c973ac713512124e9c54753772d7291891809\": container with ID starting with 2078e55337110072c4297383f22c973ac713512124e9c54753772d7291891809 not found: ID does not exist" containerID="2078e55337110072c4297383f22c973ac713512124e9c54753772d7291891809"
Nov 02 15:20:27.099184 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:20:27.099102    5304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2078e55337110072c4297383f22c973ac713512124e9c54753772d7291891809"} err="failed to get container status \"2078e55337110072c4297383f22c973ac713512124e9c54753772d7291891809\": rpc error: code = NotFound desc = could not find container \"2078e55337110072c4297383f22c973ac713512124e9c54753772d7291891809\": container with ID starting with 2078e55337110072c4297383f22c973ac713512124e9c54753772d7291891809 not found: ID does not exist"
Nov 02 15:20:27.103334 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:20:27.103310    5304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tdlhh\" (UniqueName: \"kubernetes.io/projected/c6530de8-45d4-4be1-9c37-e4c29d8a3ace-kube-api-access-tdlhh\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:20:27.103334 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:20:27.103333    5304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-glzsf\" (UniqueName: \"kubernetes.io/projected/b3bdd3ae-9a9c-4945-b820-96d56f208aa3-kube-api-access-glzsf\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:20:27.130537 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:20:27.130504    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-5f9166-85b8cf5dc-2q6d2"]
Nov 02 15:20:27.134735 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:20:27.134709    5304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-5f9166-85b8cf5dc-2q6d2"]
Nov 02 15:20:27.172638 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:20:27.172601    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-5f9166-8c7954fff-tq5g2"]
Nov 02 15:20:27.176662 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:20:27.176630    5304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-5f9166-8c7954fff-tq5g2"]
Nov 02 15:20:28.127075 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:20:28.127009    5304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b3bdd3ae-9a9c-4945-b820-96d56f208aa3" path="/var/lib/kubelet/pods/b3bdd3ae-9a9c-4945-b820-96d56f208aa3/volumes"
Nov 02 15:20:28.128097 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:20:28.128072    5304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c6530de8-45d4-4be1-9c37-e4c29d8a3ace" path="/var/lib/kubelet/pods/c6530de8-45d4-4be1-9c37-e4c29d8a3ace/volumes"
Nov 02 15:20:30.835709 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:20:30.835627    5304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-db75ff-openshift-default-7558cfb4df-qnbxl"]
Nov 02 15:20:30.836836 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:20:30.836700    5304 cpu_manager.go:414] "containerMap: removing container" podUID="b3bdd3ae-9a9c-4945-b820-96d56f208aa3" containerName="payload-processing"
Nov 02 15:20:30.836836 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:20:30.836736    5304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b3bdd3ae-9a9c-4945-b820-96d56f208aa3" containerName="payload-processing"
Nov 02 15:20:30.836836 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:20:30.836753    5304 cpu_manager.go:414] "containerMap: removing container" podUID="c6530de8-45d4-4be1-9c37-e4c29d8a3ace" containerName="payload-pre-processing"
Nov 02 15:20:30.836836 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:20:30.836761    5304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c6530de8-45d4-4be1-9c37-e4c29d8a3ace" containerName="payload-pre-processing"
Nov 02 15:20:30.837093 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:20:30.836987    5304 memory_manager.go:365] "RemoveStaleState removing state" podUID="b3bdd3ae-9a9c-4945-b820-96d56f208aa3" containerName="payload-processing"
Nov 02 15:20:30.837093 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:20:30.837004    5304 memory_manager.go:365] "RemoveStaleState removing state" podUID="c6530de8-45d4-4be1-9c37-e4c29d8a3ace" containerName="payload-pre-processing"
Nov 02 15:20:30.840412 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:20:30.840390    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-db75ff-openshift-default-7558cfb4df-qnbxl"
Nov 02 15:20:30.857412 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:20:30.857389    5304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-db75ff-openshift-default-dockercfg-bb6mf\""
Nov 02 15:20:30.876234 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:20:30.875265    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-db75ff-openshift-default-7558cfb4df-qnbxl"]
Nov 02 15:20:31.012147 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:20:31.011907    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba7e1e2d-fe7e-41ab-8fb0-814e76dfb08b-credential-socket\") pod \"e2e-api-a-db75ff-openshift-default-7558cfb4df-qnbxl\" (UID: \"ba7e1e2d-fe7e-41ab-8fb0-814e76dfb08b\") " pod="openshift-ingress/e2e-api-a-db75ff-openshift-default-7558cfb4df-qnbxl"
Nov 02 15:20:31.012147 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:20:31.012091    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba7e1e2d-fe7e-41ab-8fb0-814e76dfb08b-workload-socket\") pod \"e2e-api-a-db75ff-openshift-default-7558cfb4df-qnbxl\" (UID: \"ba7e1e2d-fe7e-41ab-8fb0-814e76dfb08b\") " pod="openshift-ingress/e2e-api-a-db75ff-openshift-default-7558cfb4df-qnbxl"
Nov 02 15:20:31.012423 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:20:31.012161    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba7e1e2d-fe7e-41ab-8fb0-814e76dfb08b-istiod-ca-cert\") pod \"e2e-api-a-db75ff-openshift-default-7558cfb4df-qnbxl\" (UID: \"ba7e1e2d-fe7e-41ab-8fb0-814e76dfb08b\") " pod="openshift-ingress/e2e-api-a-db75ff-openshift-default-7558cfb4df-qnbxl"
Nov 02 15:20:31.012423 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:20:31.012210    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba7e1e2d-fe7e-41ab-8fb0-814e76dfb08b-istio-data\") pod \"e2e-api-a-db75ff-openshift-default-7558cfb4df-qnbxl\" (UID: \"ba7e1e2d-fe7e-41ab-8fb0-814e76dfb08b\") " pod="openshift-ingress/e2e-api-a-db75ff-openshift-default-7558cfb4df-qnbxl"
Nov 02 15:20:31.012423 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:20:31.012237    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w2kzh\" (UniqueName: \"kubernetes.io/projected/ba7e1e2d-fe7e-41ab-8fb0-814e76dfb08b-kube-api-access-w2kzh\") pod \"e2e-api-a-db75ff-openshift-default-7558cfb4df-qnbxl\" (UID: \"ba7e1e2d-fe7e-41ab-8fb0-814e76dfb08b\") " pod="openshift-ingress/e2e-api-a-db75ff-openshift-default-7558cfb4df-qnbxl"
Nov 02 15:20:31.012423 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:20:31.012284    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba7e1e2d-fe7e-41ab-8fb0-814e76dfb08b-istio-token\") pod \"e2e-api-a-db75ff-openshift-default-7558cfb4df-qnbxl\" (UID: \"ba7e1e2d-fe7e-41ab-8fb0-814e76dfb08b\") " pod="openshift-ingress/e2e-api-a-db75ff-openshift-default-7558cfb4df-qnbxl"
Nov 02 15:20:31.012423 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:20:31.012352    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba7e1e2d-fe7e-41ab-8fb0-814e76dfb08b-istio-envoy\") pod \"e2e-api-a-db75ff-openshift-default-7558cfb4df-qnbxl\" (UID: \"ba7e1e2d-fe7e-41ab-8fb0-814e76dfb08b\") " pod="openshift-ingress/e2e-api-a-db75ff-openshift-default-7558cfb4df-qnbxl"
Nov 02 15:20:31.012699 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:20:31.012461    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba7e1e2d-fe7e-41ab-8fb0-814e76dfb08b-workload-certs\") pod \"e2e-api-a-db75ff-openshift-default-7558cfb4df-qnbxl\" (UID: \"ba7e1e2d-fe7e-41ab-8fb0-814e76dfb08b\") " pod="openshift-ingress/e2e-api-a-db75ff-openshift-default-7558cfb4df-qnbxl"
Nov 02 15:20:31.012699 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:20:31.012485    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba7e1e2d-fe7e-41ab-8fb0-814e76dfb08b-istio-podinfo\") pod \"e2e-api-a-db75ff-openshift-default-7558cfb4df-qnbxl\" (UID: \"ba7e1e2d-fe7e-41ab-8fb0-814e76dfb08b\") " pod="openshift-ingress/e2e-api-a-db75ff-openshift-default-7558cfb4df-qnbxl"
Nov 02 15:20:31.121205 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:20:31.120380    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba7e1e2d-fe7e-41ab-8fb0-814e76dfb08b-workload-certs\") pod \"e2e-api-a-db75ff-openshift-default-7558cfb4df-qnbxl\" (UID: \"ba7e1e2d-fe7e-41ab-8fb0-814e76dfb08b\") " pod="openshift-ingress/e2e-api-a-db75ff-openshift-default-7558cfb4df-qnbxl"
Nov 02 15:20:31.121205 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:20:31.120437    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba7e1e2d-fe7e-41ab-8fb0-814e76dfb08b-istio-podinfo\") pod \"e2e-api-a-db75ff-openshift-default-7558cfb4df-qnbxl\" (UID: \"ba7e1e2d-fe7e-41ab-8fb0-814e76dfb08b\") " pod="openshift-ingress/e2e-api-a-db75ff-openshift-default-7558cfb4df-qnbxl"
Nov 02 15:20:31.121205 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:20:31.120533    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba7e1e2d-fe7e-41ab-8fb0-814e76dfb08b-credential-socket\") pod \"e2e-api-a-db75ff-openshift-default-7558cfb4df-qnbxl\" (UID: \"ba7e1e2d-fe7e-41ab-8fb0-814e76dfb08b\") " pod="openshift-ingress/e2e-api-a-db75ff-openshift-default-7558cfb4df-qnbxl"
Nov 02 15:20:31.121205 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:20:31.120700    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba7e1e2d-fe7e-41ab-8fb0-814e76dfb08b-workload-socket\") pod \"e2e-api-a-db75ff-openshift-default-7558cfb4df-qnbxl\" (UID: \"ba7e1e2d-fe7e-41ab-8fb0-814e76dfb08b\") " pod="openshift-ingress/e2e-api-a-db75ff-openshift-default-7558cfb4df-qnbxl"
Nov 02 15:20:31.121205 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:20:31.120750    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba7e1e2d-fe7e-41ab-8fb0-814e76dfb08b-istiod-ca-cert\") pod \"e2e-api-a-db75ff-openshift-default-7558cfb4df-qnbxl\" (UID: \"ba7e1e2d-fe7e-41ab-8fb0-814e76dfb08b\") " pod="openshift-ingress/e2e-api-a-db75ff-openshift-default-7558cfb4df-qnbxl"
Nov 02 15:20:31.121205 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:20:31.120798    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba7e1e2d-fe7e-41ab-8fb0-814e76dfb08b-istio-data\") pod \"e2e-api-a-db75ff-openshift-default-7558cfb4df-qnbxl\" (UID: \"ba7e1e2d-fe7e-41ab-8fb0-814e76dfb08b\") " pod="openshift-ingress/e2e-api-a-db75ff-openshift-default-7558cfb4df-qnbxl"
Nov 02 15:20:31.121205 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:20:31.120825    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2kzh\" (UniqueName: \"kubernetes.io/projected/ba7e1e2d-fe7e-41ab-8fb0-814e76dfb08b-kube-api-access-w2kzh\") pod \"e2e-api-a-db75ff-openshift-default-7558cfb4df-qnbxl\" (UID: \"ba7e1e2d-fe7e-41ab-8fb0-814e76dfb08b\") " pod="openshift-ingress/e2e-api-a-db75ff-openshift-default-7558cfb4df-qnbxl"
Nov 02 15:20:31.121205 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:20:31.120869    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba7e1e2d-fe7e-41ab-8fb0-814e76dfb08b-istio-token\") pod \"e2e-api-a-db75ff-openshift-default-7558cfb4df-qnbxl\" (UID: \"ba7e1e2d-fe7e-41ab-8fb0-814e76dfb08b\") " pod="openshift-ingress/e2e-api-a-db75ff-openshift-default-7558cfb4df-qnbxl"
Nov 02 15:20:31.121205 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:20:31.120936    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba7e1e2d-fe7e-41ab-8fb0-814e76dfb08b-istio-envoy\") pod \"e2e-api-a-db75ff-openshift-default-7558cfb4df-qnbxl\" (UID: \"ba7e1e2d-fe7e-41ab-8fb0-814e76dfb08b\") " pod="openshift-ingress/e2e-api-a-db75ff-openshift-default-7558cfb4df-qnbxl"
Nov 02 15:20:31.122408 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:20:31.122325    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba7e1e2d-fe7e-41ab-8fb0-814e76dfb08b-workload-socket\") pod \"e2e-api-a-db75ff-openshift-default-7558cfb4df-qnbxl\" (UID: \"ba7e1e2d-fe7e-41ab-8fb0-814e76dfb08b\") " pod="openshift-ingress/e2e-api-a-db75ff-openshift-default-7558cfb4df-qnbxl"
Nov 02 15:20:31.122858 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:20:31.122836    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba7e1e2d-fe7e-41ab-8fb0-814e76dfb08b-workload-certs\") pod \"e2e-api-a-db75ff-openshift-default-7558cfb4df-qnbxl\" (UID: \"ba7e1e2d-fe7e-41ab-8fb0-814e76dfb08b\") " pod="openshift-ingress/e2e-api-a-db75ff-openshift-default-7558cfb4df-qnbxl"
Nov 02 15:20:31.123769 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:20:31.123747    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba7e1e2d-fe7e-41ab-8fb0-814e76dfb08b-istio-data\") pod \"e2e-api-a-db75ff-openshift-default-7558cfb4df-qnbxl\" (UID: \"ba7e1e2d-fe7e-41ab-8fb0-814e76dfb08b\") " pod="openshift-ingress/e2e-api-a-db75ff-openshift-default-7558cfb4df-qnbxl"
Nov 02 15:20:31.124610 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:20:31.124590    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba7e1e2d-fe7e-41ab-8fb0-814e76dfb08b-istiod-ca-cert\") pod \"e2e-api-a-db75ff-openshift-default-7558cfb4df-qnbxl\" (UID: \"ba7e1e2d-fe7e-41ab-8fb0-814e76dfb08b\") " pod="openshift-ingress/e2e-api-a-db75ff-openshift-default-7558cfb4df-qnbxl"
Nov 02 15:20:31.124964 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:20:31.124944    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba7e1e2d-fe7e-41ab-8fb0-814e76dfb08b-credential-socket\") pod \"e2e-api-a-db75ff-openshift-default-7558cfb4df-qnbxl\" (UID: \"ba7e1e2d-fe7e-41ab-8fb0-814e76dfb08b\") " pod="openshift-ingress/e2e-api-a-db75ff-openshift-default-7558cfb4df-qnbxl"
Nov 02 15:20:31.138727 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:20:31.138694    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba7e1e2d-fe7e-41ab-8fb0-814e76dfb08b-istio-podinfo\") pod \"e2e-api-a-db75ff-openshift-default-7558cfb4df-qnbxl\" (UID: \"ba7e1e2d-fe7e-41ab-8fb0-814e76dfb08b\") " pod="openshift-ingress/e2e-api-a-db75ff-openshift-default-7558cfb4df-qnbxl"
Nov 02 15:20:31.142912 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:20:31.142724    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba7e1e2d-fe7e-41ab-8fb0-814e76dfb08b-istio-envoy\") pod \"e2e-api-a-db75ff-openshift-default-7558cfb4df-qnbxl\" (UID: \"ba7e1e2d-fe7e-41ab-8fb0-814e76dfb08b\") " pod="openshift-ingress/e2e-api-a-db75ff-openshift-default-7558cfb4df-qnbxl"
Nov 02 15:20:31.153307 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:20:31.153278    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba7e1e2d-fe7e-41ab-8fb0-814e76dfb08b-istio-token\") pod \"e2e-api-a-db75ff-openshift-default-7558cfb4df-qnbxl\" (UID: \"ba7e1e2d-fe7e-41ab-8fb0-814e76dfb08b\") " pod="openshift-ingress/e2e-api-a-db75ff-openshift-default-7558cfb4df-qnbxl"
Nov 02 15:20:31.157165 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:20:31.156864    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w2kzh\" (UniqueName: \"kubernetes.io/projected/ba7e1e2d-fe7e-41ab-8fb0-814e76dfb08b-kube-api-access-w2kzh\") pod \"e2e-api-a-db75ff-openshift-default-7558cfb4df-qnbxl\" (UID: \"ba7e1e2d-fe7e-41ab-8fb0-814e76dfb08b\") " pod="openshift-ingress/e2e-api-a-db75ff-openshift-default-7558cfb4df-qnbxl"
Nov 02 15:20:31.456240 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:20:31.456120    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-db75ff-openshift-default-7558cfb4df-qnbxl"
Nov 02 15:20:31.801859 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:20:31.801835    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-db75ff-openshift-default-7558cfb4df-qnbxl"]
Nov 02 15:20:31.803913 ip-10-0-135-138 kubenswrapper[5304]: W1102 15:20:31.803888    5304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba7e1e2d_fe7e_41ab_8fb0_814e76dfb08b.slice/crio-895041ea777ca2e9e7ba5e60506fddbd33b270b2b64560849fb930d763223985 WatchSource:0}: Error finding container 895041ea777ca2e9e7ba5e60506fddbd33b270b2b64560849fb930d763223985: Status 404 returned error can't find the container with id 895041ea777ca2e9e7ba5e60506fddbd33b270b2b64560849fb930d763223985
Nov 02 15:20:31.806885 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:20:31.806851    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:20:31.806948 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:20:31.806921    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:20:31.806996 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:20:31.806949    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:20:32.128541 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:20:32.128503    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-db75ff-openshift-default-7558cfb4df-qnbxl" event={"ID":"ba7e1e2d-fe7e-41ab-8fb0-814e76dfb08b","Type":"ContainerStarted","Data":"8662bbc92abbc0a7254dedc4fea3f21fca49213a2da12c7d8f39939caad6a109"}
Nov 02 15:20:32.129037 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:20:32.128576    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-db75ff-openshift-default-7558cfb4df-qnbxl" event={"ID":"ba7e1e2d-fe7e-41ab-8fb0-814e76dfb08b","Type":"ContainerStarted","Data":"895041ea777ca2e9e7ba5e60506fddbd33b270b2b64560849fb930d763223985"}
Nov 02 15:20:32.191346 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:20:32.191282    5304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-db75ff-openshift-default-7558cfb4df-qnbxl" podStartSLOduration=2.191262707 podStartE2EDuration="2.191262707s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:20:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:20:32.190604631 +0000 UTC m=+1432.184429041" watchObservedRunningTime="2026-11-02 15:20:32.191262707 +0000 UTC m=+1432.185087117"
Nov 02 15:20:32.457120 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:20:32.456576    5304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-db75ff-openshift-default-7558cfb4df-qnbxl"
Nov 02 15:20:32.465180 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:20:32.465144    5304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-db75ff-openshift-default-7558cfb4df-qnbxl"
Nov 02 15:20:33.135273 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:20:33.135244    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-db75ff-openshift-default-7558cfb4df-qnbxl"
Nov 02 15:20:33.136041 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:20:33.136022    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-db75ff-openshift-default-7558cfb4df-qnbxl"
Nov 02 15:20:37.068444 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:20:37.068399    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-1cdbe6-86f44f7857-x9klg"
Nov 02 15:20:37.070659 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:20:37.070634    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-1cdbe6-6ccf98cccf-76jxh"
Nov 02 15:20:41.558213 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:20:41.558111    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-fda646-openshift-default-67f4df6b79-vb9vm"]
Nov 02 15:20:41.558618 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:20:41.558323    5304 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w1-fda646-openshift-default-67f4df6b79-vb9vm" podUID="64093f7f-6b31-43af-afe2-ca759bd8ce74" containerName="istio-proxy" containerID="cri-o://4a779a8180978752aebbfb26cbe6a646dab9dd301d41e3d8ec16c66a6267f975" gracePeriod=30
Nov 02 15:20:45.011185 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:20:45.010169    5304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-815746-openshift-default-74696d484f-9f99s"]
Nov 02 15:20:45.015841 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:20:45.015816    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-815746-openshift-default-74696d484f-9f99s"
Nov 02 15:20:45.016732 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:20:45.016703    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-815746-openshift-default-74696d484f-9f99s"]
Nov 02 15:20:45.031304 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:20:45.031281    5304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-815746-openshift-default-dockercfg-kjc4n\""
Nov 02 15:20:45.042287 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:20:45.042262    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ecb8e24-cbc7-4808-a2aa-6536d94590b5-istio-data\") pod \"e2e-api-b-815746-openshift-default-74696d484f-9f99s\" (UID: \"0ecb8e24-cbc7-4808-a2aa-6536d94590b5\") " pod="openshift-ingress/e2e-api-b-815746-openshift-default-74696d484f-9f99s"
Nov 02 15:20:45.042426 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:20:45.042294    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ecb8e24-cbc7-4808-a2aa-6536d94590b5-istio-podinfo\") pod \"e2e-api-b-815746-openshift-default-74696d484f-9f99s\" (UID: \"0ecb8e24-cbc7-4808-a2aa-6536d94590b5\") " pod="openshift-ingress/e2e-api-b-815746-openshift-default-74696d484f-9f99s"
Nov 02 15:20:45.042426 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:20:45.042342    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-snklp\" (UniqueName: \"kubernetes.io/projected/0ecb8e24-cbc7-4808-a2aa-6536d94590b5-kube-api-access-snklp\") pod \"e2e-api-b-815746-openshift-default-74696d484f-9f99s\" (UID: \"0ecb8e24-cbc7-4808-a2aa-6536d94590b5\") " pod="openshift-ingress/e2e-api-b-815746-openshift-default-74696d484f-9f99s"
Nov 02 15:20:45.042590 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:20:45.042568    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ecb8e24-cbc7-4808-a2aa-6536d94590b5-istio-token\") pod \"e2e-api-b-815746-openshift-default-74696d484f-9f99s\" (UID: \"0ecb8e24-cbc7-4808-a2aa-6536d94590b5\") " pod="openshift-ingress/e2e-api-b-815746-openshift-default-74696d484f-9f99s"
Nov 02 15:20:45.042726 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:20:45.042710    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ecb8e24-cbc7-4808-a2aa-6536d94590b5-istiod-ca-cert\") pod \"e2e-api-b-815746-openshift-default-74696d484f-9f99s\" (UID: \"0ecb8e24-cbc7-4808-a2aa-6536d94590b5\") " pod="openshift-ingress/e2e-api-b-815746-openshift-default-74696d484f-9f99s"
Nov 02 15:20:45.042814 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:20:45.042742    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ecb8e24-cbc7-4808-a2aa-6536d94590b5-workload-certs\") pod \"e2e-api-b-815746-openshift-default-74696d484f-9f99s\" (UID: \"0ecb8e24-cbc7-4808-a2aa-6536d94590b5\") " pod="openshift-ingress/e2e-api-b-815746-openshift-default-74696d484f-9f99s"
Nov 02 15:20:45.042871 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:20:45.042817    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ecb8e24-cbc7-4808-a2aa-6536d94590b5-credential-socket\") pod \"e2e-api-b-815746-openshift-default-74696d484f-9f99s\" (UID: \"0ecb8e24-cbc7-4808-a2aa-6536d94590b5\") " pod="openshift-ingress/e2e-api-b-815746-openshift-default-74696d484f-9f99s"
Nov 02 15:20:45.042871 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:20:45.042849    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ecb8e24-cbc7-4808-a2aa-6536d94590b5-istio-envoy\") pod \"e2e-api-b-815746-openshift-default-74696d484f-9f99s\" (UID: \"0ecb8e24-cbc7-4808-a2aa-6536d94590b5\") " pod="openshift-ingress/e2e-api-b-815746-openshift-default-74696d484f-9f99s"
Nov 02 15:20:45.042983 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:20:45.042970    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ecb8e24-cbc7-4808-a2aa-6536d94590b5-workload-socket\") pod \"e2e-api-b-815746-openshift-default-74696d484f-9f99s\" (UID: \"0ecb8e24-cbc7-4808-a2aa-6536d94590b5\") " pod="openshift-ingress/e2e-api-b-815746-openshift-default-74696d484f-9f99s"
Nov 02 15:20:45.205529 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:20:45.162384    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ecb8e24-cbc7-4808-a2aa-6536d94590b5-workload-socket\") pod \"e2e-api-b-815746-openshift-default-74696d484f-9f99s\" (UID: \"0ecb8e24-cbc7-4808-a2aa-6536d94590b5\") " pod="openshift-ingress/e2e-api-b-815746-openshift-default-74696d484f-9f99s"
Nov 02 15:20:45.206728 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:20:45.206689    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ecb8e24-cbc7-4808-a2aa-6536d94590b5-istio-data\") pod \"e2e-api-b-815746-openshift-default-74696d484f-9f99s\" (UID: \"0ecb8e24-cbc7-4808-a2aa-6536d94590b5\") " pod="openshift-ingress/e2e-api-b-815746-openshift-default-74696d484f-9f99s"
Nov 02 15:20:45.207016 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:20:45.206993    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ecb8e24-cbc7-4808-a2aa-6536d94590b5-istio-podinfo\") pod \"e2e-api-b-815746-openshift-default-74696d484f-9f99s\" (UID: \"0ecb8e24-cbc7-4808-a2aa-6536d94590b5\") " pod="openshift-ingress/e2e-api-b-815746-openshift-default-74696d484f-9f99s"
Nov 02 15:20:45.207384 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:20:45.207364    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-snklp\" (UniqueName: \"kubernetes.io/projected/0ecb8e24-cbc7-4808-a2aa-6536d94590b5-kube-api-access-snklp\") pod \"e2e-api-b-815746-openshift-default-74696d484f-9f99s\" (UID: \"0ecb8e24-cbc7-4808-a2aa-6536d94590b5\") " pod="openshift-ingress/e2e-api-b-815746-openshift-default-74696d484f-9f99s"
Nov 02 15:20:45.208027 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:20:45.208005    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ecb8e24-cbc7-4808-a2aa-6536d94590b5-istio-token\") pod \"e2e-api-b-815746-openshift-default-74696d484f-9f99s\" (UID: \"0ecb8e24-cbc7-4808-a2aa-6536d94590b5\") " pod="openshift-ingress/e2e-api-b-815746-openshift-default-74696d484f-9f99s"
Nov 02 15:20:45.227163 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:20:45.225227    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ecb8e24-cbc7-4808-a2aa-6536d94590b5-istiod-ca-cert\") pod \"e2e-api-b-815746-openshift-default-74696d484f-9f99s\" (UID: \"0ecb8e24-cbc7-4808-a2aa-6536d94590b5\") " pod="openshift-ingress/e2e-api-b-815746-openshift-default-74696d484f-9f99s"
Nov 02 15:20:45.227163 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:20:45.225298    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ecb8e24-cbc7-4808-a2aa-6536d94590b5-workload-certs\") pod \"e2e-api-b-815746-openshift-default-74696d484f-9f99s\" (UID: \"0ecb8e24-cbc7-4808-a2aa-6536d94590b5\") " pod="openshift-ingress/e2e-api-b-815746-openshift-default-74696d484f-9f99s"
Nov 02 15:20:45.227163 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:20:45.225345    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ecb8e24-cbc7-4808-a2aa-6536d94590b5-credential-socket\") pod \"e2e-api-b-815746-openshift-default-74696d484f-9f99s\" (UID: \"0ecb8e24-cbc7-4808-a2aa-6536d94590b5\") " pod="openshift-ingress/e2e-api-b-815746-openshift-default-74696d484f-9f99s"
Nov 02 15:20:45.227163 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:20:45.225375    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ecb8e24-cbc7-4808-a2aa-6536d94590b5-istio-envoy\") pod \"e2e-api-b-815746-openshift-default-74696d484f-9f99s\" (UID: \"0ecb8e24-cbc7-4808-a2aa-6536d94590b5\") " pod="openshift-ingress/e2e-api-b-815746-openshift-default-74696d484f-9f99s"
Nov 02 15:20:45.227163 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:20:45.210408    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ecb8e24-cbc7-4808-a2aa-6536d94590b5-workload-socket\") pod \"e2e-api-b-815746-openshift-default-74696d484f-9f99s\" (UID: \"0ecb8e24-cbc7-4808-a2aa-6536d94590b5\") " pod="openshift-ingress/e2e-api-b-815746-openshift-default-74696d484f-9f99s"
Nov 02 15:20:45.227163 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:20:45.209558    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ecb8e24-cbc7-4808-a2aa-6536d94590b5-istio-data\") pod \"e2e-api-b-815746-openshift-default-74696d484f-9f99s\" (UID: \"0ecb8e24-cbc7-4808-a2aa-6536d94590b5\") " pod="openshift-ingress/e2e-api-b-815746-openshift-default-74696d484f-9f99s"
Nov 02 15:20:45.236143 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:20:45.230213    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ecb8e24-cbc7-4808-a2aa-6536d94590b5-istio-envoy\") pod \"e2e-api-b-815746-openshift-default-74696d484f-9f99s\" (UID: \"0ecb8e24-cbc7-4808-a2aa-6536d94590b5\") " pod="openshift-ingress/e2e-api-b-815746-openshift-default-74696d484f-9f99s"
Nov 02 15:20:45.236143 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:20:45.230971    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ecb8e24-cbc7-4808-a2aa-6536d94590b5-istiod-ca-cert\") pod \"e2e-api-b-815746-openshift-default-74696d484f-9f99s\" (UID: \"0ecb8e24-cbc7-4808-a2aa-6536d94590b5\") " pod="openshift-ingress/e2e-api-b-815746-openshift-default-74696d484f-9f99s"
Nov 02 15:20:45.236143 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:20:45.231242    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ecb8e24-cbc7-4808-a2aa-6536d94590b5-workload-certs\") pod \"e2e-api-b-815746-openshift-default-74696d484f-9f99s\" (UID: \"0ecb8e24-cbc7-4808-a2aa-6536d94590b5\") " pod="openshift-ingress/e2e-api-b-815746-openshift-default-74696d484f-9f99s"
Nov 02 15:20:45.236143 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:20:45.231565    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ecb8e24-cbc7-4808-a2aa-6536d94590b5-credential-socket\") pod \"e2e-api-b-815746-openshift-default-74696d484f-9f99s\" (UID: \"0ecb8e24-cbc7-4808-a2aa-6536d94590b5\") " pod="openshift-ingress/e2e-api-b-815746-openshift-default-74696d484f-9f99s"
Nov 02 15:20:45.236143 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:20:45.235773    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-snklp\" (UniqueName: \"kubernetes.io/projected/0ecb8e24-cbc7-4808-a2aa-6536d94590b5-kube-api-access-snklp\") pod \"e2e-api-b-815746-openshift-default-74696d484f-9f99s\" (UID: \"0ecb8e24-cbc7-4808-a2aa-6536d94590b5\") " pod="openshift-ingress/e2e-api-b-815746-openshift-default-74696d484f-9f99s"
Nov 02 15:20:45.260304 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:20:45.254006    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ecb8e24-cbc7-4808-a2aa-6536d94590b5-istio-podinfo\") pod \"e2e-api-b-815746-openshift-default-74696d484f-9f99s\" (UID: \"0ecb8e24-cbc7-4808-a2aa-6536d94590b5\") " pod="openshift-ingress/e2e-api-b-815746-openshift-default-74696d484f-9f99s"
Nov 02 15:20:45.294248 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:20:45.292251    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ecb8e24-cbc7-4808-a2aa-6536d94590b5-istio-token\") pod \"e2e-api-b-815746-openshift-default-74696d484f-9f99s\" (UID: \"0ecb8e24-cbc7-4808-a2aa-6536d94590b5\") " pod="openshift-ingress/e2e-api-b-815746-openshift-default-74696d484f-9f99s"
Nov 02 15:20:45.334593 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:20:45.333790    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-815746-openshift-default-74696d484f-9f99s"
Nov 02 15:20:45.711153 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:20:45.711114    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-815746-openshift-default-74696d484f-9f99s"]
Nov 02 15:20:45.712574 ip-10-0-135-138 kubenswrapper[5304]: W1102 15:20:45.712546    5304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ecb8e24_cbc7_4808_a2aa_6536d94590b5.slice/crio-8fe73f3c1c5bca67bed0fb7d27386836f4bebac8656230c89b0d286680f1fedc WatchSource:0}: Error finding container 8fe73f3c1c5bca67bed0fb7d27386836f4bebac8656230c89b0d286680f1fedc: Status 404 returned error can't find the container with id 8fe73f3c1c5bca67bed0fb7d27386836f4bebac8656230c89b0d286680f1fedc
Nov 02 15:20:45.725084 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:20:45.725036    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:20:45.725250 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:20:45.725168    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:20:45.725250 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:20:45.725221    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:20:46.319511 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:20:46.319473    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-815746-openshift-default-74696d484f-9f99s" event={"ID":"0ecb8e24-cbc7-4808-a2aa-6536d94590b5","Type":"ContainerStarted","Data":"efc40a69c6cce5f130d98deb4f6cd2f5ba2b24a74d3f5b0c559a0bf990d84bd0"}
Nov 02 15:20:46.319929 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:20:46.319524    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-815746-openshift-default-74696d484f-9f99s" event={"ID":"0ecb8e24-cbc7-4808-a2aa-6536d94590b5","Type":"ContainerStarted","Data":"8fe73f3c1c5bca67bed0fb7d27386836f4bebac8656230c89b0d286680f1fedc"}
Nov 02 15:20:46.334339 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:20:46.334302    5304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-815746-openshift-default-74696d484f-9f99s"
Nov 02 15:20:46.340976 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:20:46.340949    5304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-815746-openshift-default-74696d484f-9f99s"
Nov 02 15:20:46.412709 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:20:46.412390    5304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-815746-openshift-default-74696d484f-9f99s" podStartSLOduration=2.412370022 podStartE2EDuration="2.412370022s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:20:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:20:46.366107945 +0000 UTC m=+1446.359932355" watchObservedRunningTime="2026-11-02 15:20:46.412370022 +0000 UTC m=+1446.406194431"
Nov 02 15:20:46.848210 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:20:46.848180    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-fda646-openshift-default-67f4df6b79-vb9vm"
Nov 02 15:20:46.958239 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:20:46.958210    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/64093f7f-6b31-43af-afe2-ca759bd8ce74-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64093f7f-6b31-43af-afe2-ca759bd8ce74-istiod-ca-cert\") pod \"64093f7f-6b31-43af-afe2-ca759bd8ce74\" (UID: \"64093f7f-6b31-43af-afe2-ca759bd8ce74\") "
Nov 02 15:20:46.958439 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:20:46.958302    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64093f7f-6b31-43af-afe2-ca759bd8ce74-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64093f7f-6b31-43af-afe2-ca759bd8ce74-workload-certs\") pod \"64093f7f-6b31-43af-afe2-ca759bd8ce74\" (UID: \"64093f7f-6b31-43af-afe2-ca759bd8ce74\") "
Nov 02 15:20:46.958439 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:20:46.958421    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64093f7f-6b31-43af-afe2-ca759bd8ce74-istio-token\" (UniqueName: \"kubernetes.io/projected/64093f7f-6b31-43af-afe2-ca759bd8ce74-istio-token\") pod \"64093f7f-6b31-43af-afe2-ca759bd8ce74\" (UID: \"64093f7f-6b31-43af-afe2-ca759bd8ce74\") "
Nov 02 15:20:46.958569 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:20:46.958455    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64093f7f-6b31-43af-afe2-ca759bd8ce74-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64093f7f-6b31-43af-afe2-ca759bd8ce74-credential-socket\") pod \"64093f7f-6b31-43af-afe2-ca759bd8ce74\" (UID: \"64093f7f-6b31-43af-afe2-ca759bd8ce74\") "
Nov 02 15:20:46.958569 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:20:46.958506    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/64093f7f-6b31-43af-afe2-ca759bd8ce74-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64093f7f-6b31-43af-afe2-ca759bd8ce74-istio-podinfo\") pod \"64093f7f-6b31-43af-afe2-ca759bd8ce74\" (UID: \"64093f7f-6b31-43af-afe2-ca759bd8ce74\") "
Nov 02 15:20:46.958569 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:20:46.958555    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64093f7f-6b31-43af-afe2-ca759bd8ce74-kube-api-access-nd5qh\" (UniqueName: \"kubernetes.io/projected/64093f7f-6b31-43af-afe2-ca759bd8ce74-kube-api-access-nd5qh\") pod \"64093f7f-6b31-43af-afe2-ca759bd8ce74\" (UID: \"64093f7f-6b31-43af-afe2-ca759bd8ce74\") "
Nov 02 15:20:46.958707 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:20:46.958564    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64093f7f-6b31-43af-afe2-ca759bd8ce74-istiod-ca-cert" pod "64093f7f-6b31-43af-afe2-ca759bd8ce74" (UID: "64093f7f-6b31-43af-afe2-ca759bd8ce74"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:20:46.958707 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:20:46.958574    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64093f7f-6b31-43af-afe2-ca759bd8ce74-workload-certs" pod "64093f7f-6b31-43af-afe2-ca759bd8ce74" (UID: "64093f7f-6b31-43af-afe2-ca759bd8ce74"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:20:46.958707 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:20:46.958621    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64093f7f-6b31-43af-afe2-ca759bd8ce74-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64093f7f-6b31-43af-afe2-ca759bd8ce74-workload-socket\") pod \"64093f7f-6b31-43af-afe2-ca759bd8ce74\" (UID: \"64093f7f-6b31-43af-afe2-ca759bd8ce74\") "
Nov 02 15:20:46.958858 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:20:46.958751    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64093f7f-6b31-43af-afe2-ca759bd8ce74-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64093f7f-6b31-43af-afe2-ca759bd8ce74-istio-data\") pod \"64093f7f-6b31-43af-afe2-ca759bd8ce74\" (UID: \"64093f7f-6b31-43af-afe2-ca759bd8ce74\") "
Nov 02 15:20:46.958858 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:20:46.958757    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64093f7f-6b31-43af-afe2-ca759bd8ce74-credential-socket" pod "64093f7f-6b31-43af-afe2-ca759bd8ce74" (UID: "64093f7f-6b31-43af-afe2-ca759bd8ce74"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:20:46.958858 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:20:46.958836    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64093f7f-6b31-43af-afe2-ca759bd8ce74-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64093f7f-6b31-43af-afe2-ca759bd8ce74-istio-envoy\") pod \"64093f7f-6b31-43af-afe2-ca759bd8ce74\" (UID: \"64093f7f-6b31-43af-afe2-ca759bd8ce74\") "
Nov 02 15:20:46.959024 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:20:46.958992    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64093f7f-6b31-43af-afe2-ca759bd8ce74-workload-socket" pod "64093f7f-6b31-43af-afe2-ca759bd8ce74" (UID: "64093f7f-6b31-43af-afe2-ca759bd8ce74"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:20:46.959089 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:20:46.959031    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64093f7f-6b31-43af-afe2-ca759bd8ce74-istio-data" pod "64093f7f-6b31-43af-afe2-ca759bd8ce74" (UID: "64093f7f-6b31-43af-afe2-ca759bd8ce74"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:20:46.959900 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:20:46.959870    5304 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64093f7f-6b31-43af-afe2-ca759bd8ce74-workload-socket\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:20:46.959900 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:20:46.959901    5304 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64093f7f-6b31-43af-afe2-ca759bd8ce74-istio-data\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:20:46.960080 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:20:46.959918    5304 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64093f7f-6b31-43af-afe2-ca759bd8ce74-istiod-ca-cert\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:20:46.960080 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:20:46.959932    5304 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64093f7f-6b31-43af-afe2-ca759bd8ce74-workload-certs\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:20:46.960080 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:20:46.959945    5304 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64093f7f-6b31-43af-afe2-ca759bd8ce74-credential-socket\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:20:46.960811 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:20:46.960786    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/64093f7f-6b31-43af-afe2-ca759bd8ce74-istio-podinfo" pod "64093f7f-6b31-43af-afe2-ca759bd8ce74" (UID: "64093f7f-6b31-43af-afe2-ca759bd8ce74"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 15:20:46.960918 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:20:46.960833    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64093f7f-6b31-43af-afe2-ca759bd8ce74-kube-api-access-nd5qh" pod "64093f7f-6b31-43af-afe2-ca759bd8ce74" (UID: "64093f7f-6b31-43af-afe2-ca759bd8ce74"). InnerVolumeSpecName "kube-api-access-nd5qh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:20:46.960918 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:20:46.960861    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64093f7f-6b31-43af-afe2-ca759bd8ce74-istio-token" pod "64093f7f-6b31-43af-afe2-ca759bd8ce74" (UID: "64093f7f-6b31-43af-afe2-ca759bd8ce74"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:20:46.961171 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:20:46.961156    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64093f7f-6b31-43af-afe2-ca759bd8ce74-istio-envoy" pod "64093f7f-6b31-43af-afe2-ca759bd8ce74" (UID: "64093f7f-6b31-43af-afe2-ca759bd8ce74"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:20:47.061632 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:20:47.061602    5304 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/64093f7f-6b31-43af-afe2-ca759bd8ce74-istio-token\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:20:47.061632 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:20:47.061630    5304 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64093f7f-6b31-43af-afe2-ca759bd8ce74-istio-podinfo\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:20:47.061838 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:20:47.061640    5304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nd5qh\" (UniqueName: \"kubernetes.io/projected/64093f7f-6b31-43af-afe2-ca759bd8ce74-kube-api-access-nd5qh\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:20:47.061838 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:20:47.061650    5304 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64093f7f-6b31-43af-afe2-ca759bd8ce74-istio-envoy\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:20:47.328198 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:20:47.328096    5304 generic.go:356] "Generic (PLEG): container finished" podID="64093f7f-6b31-43af-afe2-ca759bd8ce74" containerID="4a779a8180978752aebbfb26cbe6a646dab9dd301d41e3d8ec16c66a6267f975" exitCode=0
Nov 02 15:20:47.328198 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:20:47.328172    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-fda646-openshift-default-67f4df6b79-vb9vm"
Nov 02 15:20:47.328686 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:20:47.328191    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-fda646-openshift-default-67f4df6b79-vb9vm" event={"ID":"64093f7f-6b31-43af-afe2-ca759bd8ce74","Type":"ContainerDied","Data":"4a779a8180978752aebbfb26cbe6a646dab9dd301d41e3d8ec16c66a6267f975"}
Nov 02 15:20:47.328686 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:20:47.328240    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-fda646-openshift-default-67f4df6b79-vb9vm" event={"ID":"64093f7f-6b31-43af-afe2-ca759bd8ce74","Type":"ContainerDied","Data":"a76e4a98f6dd7f89fcd42c116f53f4741e3f141ff89c682997d46edfb61354d4"}
Nov 02 15:20:47.328686 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:20:47.328265    5304 scope.go:122] "RemoveContainer" containerID="4a779a8180978752aebbfb26cbe6a646dab9dd301d41e3d8ec16c66a6267f975"
Nov 02 15:20:47.328854 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:20:47.328581    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-815746-openshift-default-74696d484f-9f99s"
Nov 02 15:20:47.329706 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:20:47.329686    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-815746-openshift-default-74696d484f-9f99s"
Nov 02 15:20:47.340313 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:20:47.340287    5304 scope.go:122] "RemoveContainer" containerID="4a779a8180978752aebbfb26cbe6a646dab9dd301d41e3d8ec16c66a6267f975"
Nov 02 15:20:47.340595 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:20:47.340575    5304 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a779a8180978752aebbfb26cbe6a646dab9dd301d41e3d8ec16c66a6267f975\": container with ID starting with 4a779a8180978752aebbfb26cbe6a646dab9dd301d41e3d8ec16c66a6267f975 not found: ID does not exist" containerID="4a779a8180978752aebbfb26cbe6a646dab9dd301d41e3d8ec16c66a6267f975"
Nov 02 15:20:47.340648 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:20:47.340602    5304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a779a8180978752aebbfb26cbe6a646dab9dd301d41e3d8ec16c66a6267f975"} err="failed to get container status \"4a779a8180978752aebbfb26cbe6a646dab9dd301d41e3d8ec16c66a6267f975\": rpc error: code = NotFound desc = could not find container \"4a779a8180978752aebbfb26cbe6a646dab9dd301d41e3d8ec16c66a6267f975\": container with ID starting with 4a779a8180978752aebbfb26cbe6a646dab9dd301d41e3d8ec16c66a6267f975 not found: ID does not exist"
Nov 02 15:20:47.389881 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:20:47.389850    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-fda646-openshift-default-67f4df6b79-vb9vm"]
Nov 02 15:20:47.401382 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:20:47.401354    5304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-fda646-openshift-default-67f4df6b79-vb9vm"]
Nov 02 15:20:48.121077 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:20:48.121044    5304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="64093f7f-6b31-43af-afe2-ca759bd8ce74" path="/var/lib/kubelet/pods/64093f7f-6b31-43af-afe2-ca759bd8ce74/volumes"
Nov 02 15:20:50.673909 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:20:50.673872    5304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-815746-5fd777bdf5-m6gk5"]
Nov 02 15:20:50.674560 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:20:50.674543    5304 cpu_manager.go:414] "containerMap: removing container" podUID="64093f7f-6b31-43af-afe2-ca759bd8ce74" containerName="istio-proxy"
Nov 02 15:20:50.674560 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:20:50.674560    5304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="64093f7f-6b31-43af-afe2-ca759bd8ce74" containerName="istio-proxy"
Nov 02 15:20:50.674723 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:20:50.674712    5304 memory_manager.go:365] "RemoveStaleState removing state" podUID="64093f7f-6b31-43af-afe2-ca759bd8ce74" containerName="istio-proxy"
Nov 02 15:20:50.678285 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:20:50.678267    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-815746-5fd777bdf5-m6gk5"
Nov 02 15:20:50.698236 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:20:50.698204    5304 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-815746\""
Nov 02 15:20:50.698383 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:20:50.698362    5304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-815746-dockercfg-g64p5\""
Nov 02 15:20:50.714514 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:20:50.714483    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-815746-5fd777bdf5-m6gk5"]
Nov 02 15:20:50.720965 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:20:50.720927    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-79tbj\" (UniqueName: \"kubernetes.io/projected/c7305f17-8a19-42e5-b537-987df8ca3eb0-kube-api-access-79tbj\") pod \"payload-pre-processing-e2e-api-b-815746-5fd777bdf5-m6gk5\" (UID: \"c7305f17-8a19-42e5-b537-987df8ca3eb0\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-815746-5fd777bdf5-m6gk5"
Nov 02 15:20:50.721157 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:20:50.721108    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7305f17-8a19-42e5-b537-987df8ca3eb0-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-815746-5fd777bdf5-m6gk5\" (UID: \"c7305f17-8a19-42e5-b537-987df8ca3eb0\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-815746-5fd777bdf5-m6gk5"
Nov 02 15:20:50.861149 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:20:50.851444    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79tbj\" (UniqueName: \"kubernetes.io/projected/c7305f17-8a19-42e5-b537-987df8ca3eb0-kube-api-access-79tbj\") pod \"payload-pre-processing-e2e-api-b-815746-5fd777bdf5-m6gk5\" (UID: \"c7305f17-8a19-42e5-b537-987df8ca3eb0\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-815746-5fd777bdf5-m6gk5"
Nov 02 15:20:50.861149 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:20:50.851605    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7305f17-8a19-42e5-b537-987df8ca3eb0-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-815746-5fd777bdf5-m6gk5\" (UID: \"c7305f17-8a19-42e5-b537-987df8ca3eb0\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-815746-5fd777bdf5-m6gk5"
Nov 02 15:20:50.861149 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:20:50.856828    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7305f17-8a19-42e5-b537-987df8ca3eb0-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-815746-5fd777bdf5-m6gk5\" (UID: \"c7305f17-8a19-42e5-b537-987df8ca3eb0\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-815746-5fd777bdf5-m6gk5"
Nov 02 15:20:51.029233 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:20:51.021751    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-79tbj\" (UniqueName: \"kubernetes.io/projected/c7305f17-8a19-42e5-b537-987df8ca3eb0-kube-api-access-79tbj\") pod \"payload-pre-processing-e2e-api-b-815746-5fd777bdf5-m6gk5\" (UID: \"c7305f17-8a19-42e5-b537-987df8ca3eb0\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-815746-5fd777bdf5-m6gk5"
Nov 02 15:20:51.291519 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:20:51.291406    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-815746-5fd777bdf5-m6gk5"
Nov 02 15:20:51.609261 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:20:51.609226    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-815746-5fd777bdf5-m6gk5"]
Nov 02 15:20:51.611559 ip-10-0-135-138 kubenswrapper[5304]: W1102 15:20:51.611532    5304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7305f17_8a19_42e5_b537_987df8ca3eb0.slice/crio-97486d06e8f39bee4fb430a737cf7b225ac6e57c58f344e790b052ca9da88088 WatchSource:0}: Error finding container 97486d06e8f39bee4fb430a737cf7b225ac6e57c58f344e790b052ca9da88088: Status 404 returned error can't find the container with id 97486d06e8f39bee4fb430a737cf7b225ac6e57c58f344e790b052ca9da88088
Nov 02 15:20:52.386084 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:20:52.386048    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-815746-5fd777bdf5-m6gk5" event={"ID":"c7305f17-8a19-42e5-b537-987df8ca3eb0","Type":"ContainerStarted","Data":"e40d2ee9b48d8d55f5cd2a40706a74bcd08d4f6f8e046f643b933ca8b1dc088c"}
Nov 02 15:20:52.386652 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:20:52.386106    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-815746-5fd777bdf5-m6gk5" event={"ID":"c7305f17-8a19-42e5-b537-987df8ca3eb0","Type":"ContainerStarted","Data":"97486d06e8f39bee4fb430a737cf7b225ac6e57c58f344e790b052ca9da88088"}
Nov 02 15:20:52.386652 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:20:52.386177    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-815746-5fd777bdf5-m6gk5"
Nov 02 15:20:52.429825 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:20:52.429754    5304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-815746-5fd777bdf5-m6gk5" podStartSLOduration=2.4297330710000002 podStartE2EDuration="2.429733071s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:20:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:20:52.427755773 +0000 UTC m=+1452.421580182" watchObservedRunningTime="2026-11-02 15:20:52.429733071 +0000 UTC m=+1452.423557482"
Nov 02 15:20:54.801157 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:20:54.801085    5304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k6fjk" containerName="extractor"
Nov 02 15:21:03.410889 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:21:03.407304    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-815746-5fd777bdf5-m6gk5"
Nov 02 15:21:13.649747 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:21:13.649715    5304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-db75ff-5d4fbcd75-ddk6r"]
Nov 02 15:21:13.653146 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:21:13.653115    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-db75ff-5d4fbcd75-ddk6r"
Nov 02 15:21:13.705366 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:21:13.705337    5304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-db75ff-serving-cert\""
Nov 02 15:21:13.727317 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:21:13.727287    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-db75ff-5d4fbcd75-ddk6r"]
Nov 02 15:21:13.734857 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:21:13.734825    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/97e9ede2-d4ac-4a06-a126-81bec21c1280-openshift-service-ca\") pod \"maas-api-e2e-api-a-db75ff-5d4fbcd75-ddk6r\" (UID: \"97e9ede2-d4ac-4a06-a126-81bec21c1280\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-db75ff-5d4fbcd75-ddk6r"
Nov 02 15:21:13.735206 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:21:13.735184    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/97e9ede2-d4ac-4a06-a126-81bec21c1280-metrics-certs\") pod \"maas-api-e2e-api-a-db75ff-5d4fbcd75-ddk6r\" (UID: \"97e9ede2-d4ac-4a06-a126-81bec21c1280\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-db75ff-5d4fbcd75-ddk6r"
Nov 02 15:21:13.735445 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:21:13.735417    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v959h\" (UniqueName: \"kubernetes.io/projected/97e9ede2-d4ac-4a06-a126-81bec21c1280-kube-api-access-v959h\") pod \"maas-api-e2e-api-a-db75ff-5d4fbcd75-ddk6r\" (UID: \"97e9ede2-d4ac-4a06-a126-81bec21c1280\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-db75ff-5d4fbcd75-ddk6r"
Nov 02 15:21:13.735558 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:21:13.735512    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/97e9ede2-d4ac-4a06-a126-81bec21c1280-maas-api-tls\") pod \"maas-api-e2e-api-a-db75ff-5d4fbcd75-ddk6r\" (UID: \"97e9ede2-d4ac-4a06-a126-81bec21c1280\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-db75ff-5d4fbcd75-ddk6r"
Nov 02 15:21:13.839412 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:21:13.839366    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/97e9ede2-d4ac-4a06-a126-81bec21c1280-openshift-service-ca\") pod \"maas-api-e2e-api-a-db75ff-5d4fbcd75-ddk6r\" (UID: \"97e9ede2-d4ac-4a06-a126-81bec21c1280\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-db75ff-5d4fbcd75-ddk6r"
Nov 02 15:21:13.845107 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:21:13.845072    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/97e9ede2-d4ac-4a06-a126-81bec21c1280-metrics-certs\") pod \"maas-api-e2e-api-a-db75ff-5d4fbcd75-ddk6r\" (UID: \"97e9ede2-d4ac-4a06-a126-81bec21c1280\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-db75ff-5d4fbcd75-ddk6r"
Nov 02 15:21:13.845291 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:21:13.845242    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v959h\" (UniqueName: \"kubernetes.io/projected/97e9ede2-d4ac-4a06-a126-81bec21c1280-kube-api-access-v959h\") pod \"maas-api-e2e-api-a-db75ff-5d4fbcd75-ddk6r\" (UID: \"97e9ede2-d4ac-4a06-a126-81bec21c1280\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-db75ff-5d4fbcd75-ddk6r"
Nov 02 15:21:13.845343 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:21:13.845328    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/97e9ede2-d4ac-4a06-a126-81bec21c1280-maas-api-tls\") pod \"maas-api-e2e-api-a-db75ff-5d4fbcd75-ddk6r\" (UID: \"97e9ede2-d4ac-4a06-a126-81bec21c1280\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-db75ff-5d4fbcd75-ddk6r"
Nov 02 15:21:13.846067 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:21:13.844850    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/97e9ede2-d4ac-4a06-a126-81bec21c1280-openshift-service-ca\") pod \"maas-api-e2e-api-a-db75ff-5d4fbcd75-ddk6r\" (UID: \"97e9ede2-d4ac-4a06-a126-81bec21c1280\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-db75ff-5d4fbcd75-ddk6r"
Nov 02 15:21:13.848869 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:21:13.848848    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/97e9ede2-d4ac-4a06-a126-81bec21c1280-maas-api-tls\") pod \"maas-api-e2e-api-a-db75ff-5d4fbcd75-ddk6r\" (UID: \"97e9ede2-d4ac-4a06-a126-81bec21c1280\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-db75ff-5d4fbcd75-ddk6r"
Nov 02 15:21:13.849058 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:21:13.849040    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/97e9ede2-d4ac-4a06-a126-81bec21c1280-metrics-certs\") pod \"maas-api-e2e-api-a-db75ff-5d4fbcd75-ddk6r\" (UID: \"97e9ede2-d4ac-4a06-a126-81bec21c1280\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-db75ff-5d4fbcd75-ddk6r"
Nov 02 15:21:13.870363 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:21:13.870336    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v959h\" (UniqueName: \"kubernetes.io/projected/97e9ede2-d4ac-4a06-a126-81bec21c1280-kube-api-access-v959h\") pod \"maas-api-e2e-api-a-db75ff-5d4fbcd75-ddk6r\" (UID: \"97e9ede2-d4ac-4a06-a126-81bec21c1280\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-db75ff-5d4fbcd75-ddk6r"
Nov 02 15:21:13.966212 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:21:13.966117    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-db75ff-5d4fbcd75-ddk6r"
Nov 02 15:21:14.273377 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:21:14.273349    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-db75ff-5d4fbcd75-ddk6r"]
Nov 02 15:21:14.275899 ip-10-0-135-138 kubenswrapper[5304]: W1102 15:21:14.275871    5304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97e9ede2_d4ac_4a06_a126_81bec21c1280.slice/crio-2e7f64ebe52ed6fca8fd03bfd89bd14c8b9080b7db2e065a80bcb642ff5fd8d8 WatchSource:0}: Error finding container 2e7f64ebe52ed6fca8fd03bfd89bd14c8b9080b7db2e065a80bcb642ff5fd8d8: Status 404 returned error can't find the container with id 2e7f64ebe52ed6fca8fd03bfd89bd14c8b9080b7db2e065a80bcb642ff5fd8d8
Nov 02 15:21:14.627393 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:21:14.627354    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-db75ff-5d4fbcd75-ddk6r" event={"ID":"97e9ede2-d4ac-4a06-a126-81bec21c1280","Type":"ContainerStarted","Data":"2e7f64ebe52ed6fca8fd03bfd89bd14c8b9080b7db2e065a80bcb642ff5fd8d8"}
Nov 02 15:21:15.638333 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:21:15.638296    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-db75ff-5d4fbcd75-ddk6r" event={"ID":"97e9ede2-d4ac-4a06-a126-81bec21c1280","Type":"ContainerStarted","Data":"ac8961f10a2319da64f8746d626d78b260ca59204911333db8472854dd3d17ae"}
Nov 02 15:21:15.638758 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:21:15.638382    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-db75ff-5d4fbcd75-ddk6r"
Nov 02 15:21:15.696954 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:21:15.696890    5304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-db75ff-5d4fbcd75-ddk6r" podStartSLOduration=2.344123742 podStartE2EDuration="2.696876303s" totalImagesPullingTime="352.752561ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:21:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:21:15.696458353 +0000 UTC m=+1475.690282763" watchObservedRunningTime="2026-11-02 15:21:15.696876303 +0000 UTC m=+1475.690700713"
Nov 02 15:21:19.311485 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:21:19.311448    5304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-f975ed-6c55746559-gc62v"]
Nov 02 15:21:19.315334 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:21:19.315315    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f975ed-6c55746559-gc62v"
Nov 02 15:21:19.348965 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:21:19.348934    5304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-f975ed-dockercfg-btlvx\""
Nov 02 15:21:19.355577 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:21:19.353688    5304 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-f975ed\""
Nov 02 15:21:19.355577 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:21:19.354749    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-f975ed-6c55746559-gc62v"]
Nov 02 15:21:19.420155 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:21:19.418171    5304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-f975ed-fcd5fdf77-sqnvq"]
Nov 02 15:21:19.445155 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:21:19.443246    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-f975ed-fcd5fdf77-sqnvq"
Nov 02 15:21:19.492557 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:21:19.489588    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-f975ed-fcd5fdf77-sqnvq"]
Nov 02 15:21:19.522152 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:21:19.495113    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qkzhb\" (UniqueName: \"kubernetes.io/projected/f2c78d1b-1497-4fe3-bfcc-f8eecb41f621-kube-api-access-qkzhb\") pod \"payload-processing-e2e-inf-a-f975ed-fcd5fdf77-sqnvq\" (UID: \"f2c78d1b-1497-4fe3-bfcc-f8eecb41f621\") " pod="openshift-ingress/payload-processing-e2e-inf-a-f975ed-fcd5fdf77-sqnvq"
Nov 02 15:21:19.522152 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:21:19.502282    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4bqw5\" (UniqueName: \"kubernetes.io/projected/1f4f76a0-e6ea-4515-9543-e7720ae0a2c6-kube-api-access-4bqw5\") pod \"payload-pre-processing-e2e-inf-a-f975ed-6c55746559-gc62v\" (UID: \"1f4f76a0-e6ea-4515-9543-e7720ae0a2c6\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f975ed-6c55746559-gc62v"
Nov 02 15:21:19.522152 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:21:19.502346    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2c78d1b-1497-4fe3-bfcc-f8eecb41f621-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-f975ed-fcd5fdf77-sqnvq\" (UID: \"f2c78d1b-1497-4fe3-bfcc-f8eecb41f621\") " pod="openshift-ingress/payload-processing-e2e-inf-a-f975ed-fcd5fdf77-sqnvq"
Nov 02 15:21:19.522152 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:21:19.502410    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f4f76a0-e6ea-4515-9543-e7720ae0a2c6-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-f975ed-6c55746559-gc62v\" (UID: \"1f4f76a0-e6ea-4515-9543-e7720ae0a2c6\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f975ed-6c55746559-gc62v"
Nov 02 15:21:19.654714 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:21:19.654469    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4bqw5\" (UniqueName: \"kubernetes.io/projected/1f4f76a0-e6ea-4515-9543-e7720ae0a2c6-kube-api-access-4bqw5\") pod \"payload-pre-processing-e2e-inf-a-f975ed-6c55746559-gc62v\" (UID: \"1f4f76a0-e6ea-4515-9543-e7720ae0a2c6\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f975ed-6c55746559-gc62v"
Nov 02 15:21:19.654714 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:21:19.654546    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2c78d1b-1497-4fe3-bfcc-f8eecb41f621-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-f975ed-fcd5fdf77-sqnvq\" (UID: \"f2c78d1b-1497-4fe3-bfcc-f8eecb41f621\") " pod="openshift-ingress/payload-processing-e2e-inf-a-f975ed-fcd5fdf77-sqnvq"
Nov 02 15:21:19.654714 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:21:19.654612    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f4f76a0-e6ea-4515-9543-e7720ae0a2c6-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-f975ed-6c55746559-gc62v\" (UID: \"1f4f76a0-e6ea-4515-9543-e7720ae0a2c6\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f975ed-6c55746559-gc62v"
Nov 02 15:21:19.655062 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:21:19.654957    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qkzhb\" (UniqueName: \"kubernetes.io/projected/f2c78d1b-1497-4fe3-bfcc-f8eecb41f621-kube-api-access-qkzhb\") pod \"payload-processing-e2e-inf-a-f975ed-fcd5fdf77-sqnvq\" (UID: \"f2c78d1b-1497-4fe3-bfcc-f8eecb41f621\") " pod="openshift-ingress/payload-processing-e2e-inf-a-f975ed-fcd5fdf77-sqnvq"
Nov 02 15:21:19.655919 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:21:19.655894    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2c78d1b-1497-4fe3-bfcc-f8eecb41f621-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-f975ed-fcd5fdf77-sqnvq\" (UID: \"f2c78d1b-1497-4fe3-bfcc-f8eecb41f621\") " pod="openshift-ingress/payload-processing-e2e-inf-a-f975ed-fcd5fdf77-sqnvq"
Nov 02 15:21:19.656108 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:21:19.656086    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f4f76a0-e6ea-4515-9543-e7720ae0a2c6-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-f975ed-6c55746559-gc62v\" (UID: \"1f4f76a0-e6ea-4515-9543-e7720ae0a2c6\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f975ed-6c55746559-gc62v"
Nov 02 15:21:19.678618 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:21:19.678591    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4bqw5\" (UniqueName: \"kubernetes.io/projected/1f4f76a0-e6ea-4515-9543-e7720ae0a2c6-kube-api-access-4bqw5\") pod \"payload-pre-processing-e2e-inf-a-f975ed-6c55746559-gc62v\" (UID: \"1f4f76a0-e6ea-4515-9543-e7720ae0a2c6\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f975ed-6c55746559-gc62v"
Nov 02 15:21:19.679560 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:21:19.679525    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qkzhb\" (UniqueName: \"kubernetes.io/projected/f2c78d1b-1497-4fe3-bfcc-f8eecb41f621-kube-api-access-qkzhb\") pod \"payload-processing-e2e-inf-a-f975ed-fcd5fdf77-sqnvq\" (UID: \"f2c78d1b-1497-4fe3-bfcc-f8eecb41f621\") " pod="openshift-ingress/payload-processing-e2e-inf-a-f975ed-fcd5fdf77-sqnvq"
Nov 02 15:21:19.807520 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:21:19.807454    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-f975ed-fcd5fdf77-sqnvq"
Nov 02 15:21:19.931353 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:21:19.930985    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f975ed-6c55746559-gc62v"
Nov 02 15:21:20.131417 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:21:20.131386    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-f975ed-fcd5fdf77-sqnvq"]
Nov 02 15:21:20.133034 ip-10-0-135-138 kubenswrapper[5304]: W1102 15:21:20.133003    5304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2c78d1b_1497_4fe3_bfcc_f8eecb41f621.slice/crio-4439f12cb99278388cfd819be2f17e51d780167a8106ad19bb64985d216db92a WatchSource:0}: Error finding container 4439f12cb99278388cfd819be2f17e51d780167a8106ad19bb64985d216db92a: Status 404 returned error can't find the container with id 4439f12cb99278388cfd819be2f17e51d780167a8106ad19bb64985d216db92a
Nov 02 15:21:20.254187 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:21:20.254152    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-f975ed-6c55746559-gc62v"]
Nov 02 15:21:20.254661 ip-10-0-135-138 kubenswrapper[5304]: W1102 15:21:20.254272    5304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f4f76a0_e6ea_4515_9543_e7720ae0a2c6.slice/crio-67f68acd4c10d91c7def5d3ef3456c16ea78e132ac19f4dd47bfdbac8f0798cf WatchSource:0}: Error finding container 67f68acd4c10d91c7def5d3ef3456c16ea78e132ac19f4dd47bfdbac8f0798cf: Status 404 returned error can't find the container with id 67f68acd4c10d91c7def5d3ef3456c16ea78e132ac19f4dd47bfdbac8f0798cf
Nov 02 15:21:20.694007 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:21:20.693821    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-f975ed-fcd5fdf77-sqnvq" event={"ID":"f2c78d1b-1497-4fe3-bfcc-f8eecb41f621","Type":"ContainerStarted","Data":"3d1c1209af7649e3620435383e5e47b91bf5ee9cd6ed0b6b1a49c1d705ad700d"}
Nov 02 15:21:20.694007 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:21:20.693920    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-f975ed-fcd5fdf77-sqnvq"
Nov 02 15:21:20.694007 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:21:20.693950    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-f975ed-fcd5fdf77-sqnvq" event={"ID":"f2c78d1b-1497-4fe3-bfcc-f8eecb41f621","Type":"ContainerStarted","Data":"4439f12cb99278388cfd819be2f17e51d780167a8106ad19bb64985d216db92a"}
Nov 02 15:21:20.696179 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:21:20.696147    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f975ed-6c55746559-gc62v" event={"ID":"1f4f76a0-e6ea-4515-9543-e7720ae0a2c6","Type":"ContainerStarted","Data":"cc1105ad8af35c1a3d1da216f47955f5b45f10cc33915131f8a3a646393847d8"}
Nov 02 15:21:20.696319 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:21:20.696206    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f975ed-6c55746559-gc62v" event={"ID":"1f4f76a0-e6ea-4515-9543-e7720ae0a2c6","Type":"ContainerStarted","Data":"67f68acd4c10d91c7def5d3ef3456c16ea78e132ac19f4dd47bfdbac8f0798cf"}
Nov 02 15:21:20.696319 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:21:20.696251    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f975ed-6c55746559-gc62v"
Nov 02 15:21:20.736847 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:21:20.736788    5304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-f975ed-fcd5fdf77-sqnvq" podStartSLOduration=1.7367695090000002 podStartE2EDuration="1.736769509s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:21:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:21:20.735201895 +0000 UTC m=+1480.729026304" watchObservedRunningTime="2026-11-02 15:21:20.736769509 +0000 UTC m=+1480.730593918"
Nov 02 15:21:20.778449 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:21:20.778397    5304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f975ed-6c55746559-gc62v" podStartSLOduration=1.778384156 podStartE2EDuration="1.778384156s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:21:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:21:20.774162527 +0000 UTC m=+1480.767986933" watchObservedRunningTime="2026-11-02 15:21:20.778384156 +0000 UTC m=+1480.772208565"
Nov 02 15:21:21.651546 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:21:21.651516    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-db75ff-5d4fbcd75-ddk6r"
Nov 02 15:21:24.799919 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:21:24.799877    5304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k6fjk" containerName="extractor"
Nov 02 15:21:31.732978 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:21:31.728095    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-f975ed-fcd5fdf77-sqnvq"
Nov 02 15:21:31.732978 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:21:31.729703    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f975ed-6c55746559-gc62v"
Nov 02 15:21:33.093079 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:21:33.093042    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-5f9166-769d4fdb6c-529mc"]
Nov 02 15:21:33.093631 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:21:33.093330    5304 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5f9166-769d4fdb6c-529mc" podUID="fe8598be-5e22-47f6-bf0b-3a279ea72ebb" containerName="maas-api" containerID="cri-o://f54f927b07c6bbe2e19ea5ceeb88bccf197ad723c41d6a6e5f3a6a4268e91439" gracePeriod=30
Nov 02 15:21:33.545884 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:21:33.545848    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5f9166-769d4fdb6c-529mc"
Nov 02 15:21:33.602147 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:21:33.601701    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fe8598be-5e22-47f6-bf0b-3a279ea72ebb-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe8598be-5e22-47f6-bf0b-3a279ea72ebb-openshift-service-ca\") pod \"fe8598be-5e22-47f6-bf0b-3a279ea72ebb\" (UID: \"fe8598be-5e22-47f6-bf0b-3a279ea72ebb\") "
Nov 02 15:21:33.602147 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:21:33.601808    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fe8598be-5e22-47f6-bf0b-3a279ea72ebb-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fe8598be-5e22-47f6-bf0b-3a279ea72ebb-maas-api-tls\") pod \"fe8598be-5e22-47f6-bf0b-3a279ea72ebb\" (UID: \"fe8598be-5e22-47f6-bf0b-3a279ea72ebb\") "
Nov 02 15:21:33.602147 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:21:33.601882    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fe8598be-5e22-47f6-bf0b-3a279ea72ebb-metrics-certs\" (UniqueName: \"kubernetes.io/secret/fe8598be-5e22-47f6-bf0b-3a279ea72ebb-metrics-certs\") pod \"fe8598be-5e22-47f6-bf0b-3a279ea72ebb\" (UID: \"fe8598be-5e22-47f6-bf0b-3a279ea72ebb\") "
Nov 02 15:21:33.602147 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:21:33.602022    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe8598be-5e22-47f6-bf0b-3a279ea72ebb-kube-api-access-4jrkv\" (UniqueName: \"kubernetes.io/projected/fe8598be-5e22-47f6-bf0b-3a279ea72ebb-kube-api-access-4jrkv\") pod \"fe8598be-5e22-47f6-bf0b-3a279ea72ebb\" (UID: \"fe8598be-5e22-47f6-bf0b-3a279ea72ebb\") "
Nov 02 15:21:33.603640 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:21:33.603266    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe8598be-5e22-47f6-bf0b-3a279ea72ebb-openshift-service-ca" pod "fe8598be-5e22-47f6-bf0b-3a279ea72ebb" (UID: "fe8598be-5e22-47f6-bf0b-3a279ea72ebb"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:21:33.618726 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:21:33.618537    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe8598be-5e22-47f6-bf0b-3a279ea72ebb-kube-api-access-4jrkv" pod "fe8598be-5e22-47f6-bf0b-3a279ea72ebb" (UID: "fe8598be-5e22-47f6-bf0b-3a279ea72ebb"). InnerVolumeSpecName "kube-api-access-4jrkv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:21:33.618726 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:21:33.618663    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fe8598be-5e22-47f6-bf0b-3a279ea72ebb-metrics-certs" pod "fe8598be-5e22-47f6-bf0b-3a279ea72ebb" (UID: "fe8598be-5e22-47f6-bf0b-3a279ea72ebb"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:21:33.634151 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:21:33.630297    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fe8598be-5e22-47f6-bf0b-3a279ea72ebb-maas-api-tls" pod "fe8598be-5e22-47f6-bf0b-3a279ea72ebb" (UID: "fe8598be-5e22-47f6-bf0b-3a279ea72ebb"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:21:33.730482 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:21:33.724618    5304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4jrkv\" (UniqueName: \"kubernetes.io/projected/fe8598be-5e22-47f6-bf0b-3a279ea72ebb-kube-api-access-4jrkv\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:21:33.730482 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:21:33.724658    5304 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe8598be-5e22-47f6-bf0b-3a279ea72ebb-openshift-service-ca\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:21:33.730482 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:21:33.724674    5304 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fe8598be-5e22-47f6-bf0b-3a279ea72ebb-maas-api-tls\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:21:33.730482 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:21:33.724692    5304 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fe8598be-5e22-47f6-bf0b-3a279ea72ebb-metrics-certs\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:21:33.811447 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:21:33.811409    5304 generic.go:356] "Generic (PLEG): container finished" podID="fe8598be-5e22-47f6-bf0b-3a279ea72ebb" containerID="f54f927b07c6bbe2e19ea5ceeb88bccf197ad723c41d6a6e5f3a6a4268e91439" exitCode=0
Nov 02 15:21:33.811624 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:21:33.811465    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5f9166-769d4fdb6c-529mc"
Nov 02 15:21:33.811624 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:21:33.811496    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5f9166-769d4fdb6c-529mc" event={"ID":"fe8598be-5e22-47f6-bf0b-3a279ea72ebb","Type":"ContainerDied","Data":"f54f927b07c6bbe2e19ea5ceeb88bccf197ad723c41d6a6e5f3a6a4268e91439"}
Nov 02 15:21:33.811624 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:21:33.811557    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5f9166-769d4fdb6c-529mc" event={"ID":"fe8598be-5e22-47f6-bf0b-3a279ea72ebb","Type":"ContainerDied","Data":"fcd6c62212bd7b14cb16e711195147eed256e2bf69a8d5b733e9f44289114227"}
Nov 02 15:21:33.811624 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:21:33.811586    5304 scope.go:122] "RemoveContainer" containerID="f54f927b07c6bbe2e19ea5ceeb88bccf197ad723c41d6a6e5f3a6a4268e91439"
Nov 02 15:21:33.823049 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:21:33.823020    5304 scope.go:122] "RemoveContainer" containerID="f54f927b07c6bbe2e19ea5ceeb88bccf197ad723c41d6a6e5f3a6a4268e91439"
Nov 02 15:21:33.823414 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:21:33.823391    5304 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f54f927b07c6bbe2e19ea5ceeb88bccf197ad723c41d6a6e5f3a6a4268e91439\": container with ID starting with f54f927b07c6bbe2e19ea5ceeb88bccf197ad723c41d6a6e5f3a6a4268e91439 not found: ID does not exist" containerID="f54f927b07c6bbe2e19ea5ceeb88bccf197ad723c41d6a6e5f3a6a4268e91439"
Nov 02 15:21:33.823513 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:21:33.823425    5304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f54f927b07c6bbe2e19ea5ceeb88bccf197ad723c41d6a6e5f3a6a4268e91439"} err="failed to get container status \"f54f927b07c6bbe2e19ea5ceeb88bccf197ad723c41d6a6e5f3a6a4268e91439\": rpc error: code = NotFound desc = could not find container \"f54f927b07c6bbe2e19ea5ceeb88bccf197ad723c41d6a6e5f3a6a4268e91439\": container with ID starting with f54f927b07c6bbe2e19ea5ceeb88bccf197ad723c41d6a6e5f3a6a4268e91439 not found: ID does not exist"
Nov 02 15:21:33.868517 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:21:33.868486    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-5f9166-769d4fdb6c-529mc"]
Nov 02 15:21:33.872461 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:21:33.872433    5304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-5f9166-769d4fdb6c-529mc"]
Nov 02 15:21:34.121883 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:21:34.121851    5304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fe8598be-5e22-47f6-bf0b-3a279ea72ebb" path="/var/lib/kubelet/pods/fe8598be-5e22-47f6-bf0b-3a279ea72ebb/volumes"
Nov 02 15:21:41.251458 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:21:41.251418    5304 scope.go:122] "RemoveContainer" containerID="55e15ec3c0124c5b6b479575c8b7d35af16cda178e477a46abd03f6bfa3c22af"
Nov 02 15:21:41.809800 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:21:41.809761    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-5f9166-openshift-default-5d4b56b869-skdww"]
Nov 02 15:21:41.810100 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:21:41.810053    5304 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-5f9166-openshift-default-5d4b56b869-skdww" podUID="c34a9870-db4c-462a-827f-95abdfc1f68b" containerName="istio-proxy" containerID="cri-o://7553fcc74706c6f700066b6fa868972f26203f2905a05f1a775b6ebb40fa53cf" gracePeriod=30
Nov 02 15:21:43.028480 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:21:43.028440    5304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-a03e84-openshift-default-8f4cc949-4flt2"]
Nov 02 15:21:43.036364 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:21:43.031739    5304 cpu_manager.go:414] "containerMap: removing container" podUID="fe8598be-5e22-47f6-bf0b-3a279ea72ebb" containerName="maas-api"
Nov 02 15:21:43.036589 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:21:43.036567    5304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe8598be-5e22-47f6-bf0b-3a279ea72ebb" containerName="maas-api"
Nov 02 15:21:43.044154 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:21:43.036964    5304 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe8598be-5e22-47f6-bf0b-3a279ea72ebb" containerName="maas-api"
Nov 02 15:21:43.059978 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:21:43.059939    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-a03e84-openshift-default-8f4cc949-4flt2"
Nov 02 15:21:43.098173 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:21:43.097227    5304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-a03e84-openshift-default-dockercfg-fqmh5\""
Nov 02 15:21:43.115639 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:21:43.109103    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c109588b-9544-4a37-bb13-d81572067546-istio-envoy\") pod \"e2e-inf-a-a03e84-openshift-default-8f4cc949-4flt2\" (UID: \"c109588b-9544-4a37-bb13-d81572067546\") " pod="openshift-ingress/e2e-inf-a-a03e84-openshift-default-8f4cc949-4flt2"
Nov 02 15:21:43.115639 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:21:43.109196    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xs8cd\" (UniqueName: \"kubernetes.io/projected/c109588b-9544-4a37-bb13-d81572067546-kube-api-access-xs8cd\") pod \"e2e-inf-a-a03e84-openshift-default-8f4cc949-4flt2\" (UID: \"c109588b-9544-4a37-bb13-d81572067546\") " pod="openshift-ingress/e2e-inf-a-a03e84-openshift-default-8f4cc949-4flt2"
Nov 02 15:21:43.115639 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:21:43.109285    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c109588b-9544-4a37-bb13-d81572067546-istio-data\") pod \"e2e-inf-a-a03e84-openshift-default-8f4cc949-4flt2\" (UID: \"c109588b-9544-4a37-bb13-d81572067546\") " pod="openshift-ingress/e2e-inf-a-a03e84-openshift-default-8f4cc949-4flt2"
Nov 02 15:21:43.115639 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:21:43.109537    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c109588b-9544-4a37-bb13-d81572067546-istiod-ca-cert\") pod \"e2e-inf-a-a03e84-openshift-default-8f4cc949-4flt2\" (UID: \"c109588b-9544-4a37-bb13-d81572067546\") " pod="openshift-ingress/e2e-inf-a-a03e84-openshift-default-8f4cc949-4flt2"
Nov 02 15:21:43.115639 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:21:43.109587    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c109588b-9544-4a37-bb13-d81572067546-credential-socket\") pod \"e2e-inf-a-a03e84-openshift-default-8f4cc949-4flt2\" (UID: \"c109588b-9544-4a37-bb13-d81572067546\") " pod="openshift-ingress/e2e-inf-a-a03e84-openshift-default-8f4cc949-4flt2"
Nov 02 15:21:43.115639 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:21:43.109667    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c109588b-9544-4a37-bb13-d81572067546-istio-podinfo\") pod \"e2e-inf-a-a03e84-openshift-default-8f4cc949-4flt2\" (UID: \"c109588b-9544-4a37-bb13-d81572067546\") " pod="openshift-ingress/e2e-inf-a-a03e84-openshift-default-8f4cc949-4flt2"
Nov 02 15:21:43.115639 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:21:43.109703    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c109588b-9544-4a37-bb13-d81572067546-istio-token\") pod \"e2e-inf-a-a03e84-openshift-default-8f4cc949-4flt2\" (UID: \"c109588b-9544-4a37-bb13-d81572067546\") " pod="openshift-ingress/e2e-inf-a-a03e84-openshift-default-8f4cc949-4flt2"
Nov 02 15:21:43.115639 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:21:43.109781    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c109588b-9544-4a37-bb13-d81572067546-workload-certs\") pod \"e2e-inf-a-a03e84-openshift-default-8f4cc949-4flt2\" (UID: \"c109588b-9544-4a37-bb13-d81572067546\") " pod="openshift-ingress/e2e-inf-a-a03e84-openshift-default-8f4cc949-4flt2"
Nov 02 15:21:43.115639 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:21:43.109843    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c109588b-9544-4a37-bb13-d81572067546-workload-socket\") pod \"e2e-inf-a-a03e84-openshift-default-8f4cc949-4flt2\" (UID: \"c109588b-9544-4a37-bb13-d81572067546\") " pod="openshift-ingress/e2e-inf-a-a03e84-openshift-default-8f4cc949-4flt2"
Nov 02 15:21:43.140432 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:21:43.140386    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-a03e84-openshift-default-8f4cc949-4flt2"]
Nov 02 15:21:43.227150 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:21:43.224624    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c109588b-9544-4a37-bb13-d81572067546-istiod-ca-cert\") pod \"e2e-inf-a-a03e84-openshift-default-8f4cc949-4flt2\" (UID: \"c109588b-9544-4a37-bb13-d81572067546\") " pod="openshift-ingress/e2e-inf-a-a03e84-openshift-default-8f4cc949-4flt2"
Nov 02 15:21:43.227150 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:21:43.224696    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c109588b-9544-4a37-bb13-d81572067546-credential-socket\") pod \"e2e-inf-a-a03e84-openshift-default-8f4cc949-4flt2\" (UID: \"c109588b-9544-4a37-bb13-d81572067546\") " pod="openshift-ingress/e2e-inf-a-a03e84-openshift-default-8f4cc949-4flt2"
Nov 02 15:21:43.227150 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:21:43.224794    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c109588b-9544-4a37-bb13-d81572067546-istio-podinfo\") pod \"e2e-inf-a-a03e84-openshift-default-8f4cc949-4flt2\" (UID: \"c109588b-9544-4a37-bb13-d81572067546\") " pod="openshift-ingress/e2e-inf-a-a03e84-openshift-default-8f4cc949-4flt2"
Nov 02 15:21:43.227150 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:21:43.224841    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c109588b-9544-4a37-bb13-d81572067546-istio-token\") pod \"e2e-inf-a-a03e84-openshift-default-8f4cc949-4flt2\" (UID: \"c109588b-9544-4a37-bb13-d81572067546\") " pod="openshift-ingress/e2e-inf-a-a03e84-openshift-default-8f4cc949-4flt2"
Nov 02 15:21:43.227150 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:21:43.224922    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c109588b-9544-4a37-bb13-d81572067546-workload-certs\") pod \"e2e-inf-a-a03e84-openshift-default-8f4cc949-4flt2\" (UID: \"c109588b-9544-4a37-bb13-d81572067546\") " pod="openshift-ingress/e2e-inf-a-a03e84-openshift-default-8f4cc949-4flt2"
Nov 02 15:21:43.227150 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:21:43.224982    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c109588b-9544-4a37-bb13-d81572067546-workload-socket\") pod \"e2e-inf-a-a03e84-openshift-default-8f4cc949-4flt2\" (UID: \"c109588b-9544-4a37-bb13-d81572067546\") " pod="openshift-ingress/e2e-inf-a-a03e84-openshift-default-8f4cc949-4flt2"
Nov 02 15:21:43.227150 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:21:43.225020    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c109588b-9544-4a37-bb13-d81572067546-istio-envoy\") pod \"e2e-inf-a-a03e84-openshift-default-8f4cc949-4flt2\" (UID: \"c109588b-9544-4a37-bb13-d81572067546\") " pod="openshift-ingress/e2e-inf-a-a03e84-openshift-default-8f4cc949-4flt2"
Nov 02 15:21:43.227150 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:21:43.225070    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xs8cd\" (UniqueName: \"kubernetes.io/projected/c109588b-9544-4a37-bb13-d81572067546-kube-api-access-xs8cd\") pod \"e2e-inf-a-a03e84-openshift-default-8f4cc949-4flt2\" (UID: \"c109588b-9544-4a37-bb13-d81572067546\") " pod="openshift-ingress/e2e-inf-a-a03e84-openshift-default-8f4cc949-4flt2"
Nov 02 15:21:43.227150 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:21:43.225174    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c109588b-9544-4a37-bb13-d81572067546-istio-data\") pod \"e2e-inf-a-a03e84-openshift-default-8f4cc949-4flt2\" (UID: \"c109588b-9544-4a37-bb13-d81572067546\") " pod="openshift-ingress/e2e-inf-a-a03e84-openshift-default-8f4cc949-4flt2"
Nov 02 15:21:43.227150 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:21:43.226619    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c109588b-9544-4a37-bb13-d81572067546-istiod-ca-cert\") pod \"e2e-inf-a-a03e84-openshift-default-8f4cc949-4flt2\" (UID: \"c109588b-9544-4a37-bb13-d81572067546\") " pod="openshift-ingress/e2e-inf-a-a03e84-openshift-default-8f4cc949-4flt2"
Nov 02 15:21:43.227150 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:21:43.226920    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c109588b-9544-4a37-bb13-d81572067546-credential-socket\") pod \"e2e-inf-a-a03e84-openshift-default-8f4cc949-4flt2\" (UID: \"c109588b-9544-4a37-bb13-d81572067546\") " pod="openshift-ingress/e2e-inf-a-a03e84-openshift-default-8f4cc949-4flt2"
Nov 02 15:21:43.231174 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:21:43.228412    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c109588b-9544-4a37-bb13-d81572067546-workload-certs\") pod \"e2e-inf-a-a03e84-openshift-default-8f4cc949-4flt2\" (UID: \"c109588b-9544-4a37-bb13-d81572067546\") " pod="openshift-ingress/e2e-inf-a-a03e84-openshift-default-8f4cc949-4flt2"
Nov 02 15:21:43.231174 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:21:43.229765    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c109588b-9544-4a37-bb13-d81572067546-workload-socket\") pod \"e2e-inf-a-a03e84-openshift-default-8f4cc949-4flt2\" (UID: \"c109588b-9544-4a37-bb13-d81572067546\") " pod="openshift-ingress/e2e-inf-a-a03e84-openshift-default-8f4cc949-4flt2"
Nov 02 15:21:43.234375 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:21:43.233359    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c109588b-9544-4a37-bb13-d81572067546-istio-data\") pod \"e2e-inf-a-a03e84-openshift-default-8f4cc949-4flt2\" (UID: \"c109588b-9544-4a37-bb13-d81572067546\") " pod="openshift-ingress/e2e-inf-a-a03e84-openshift-default-8f4cc949-4flt2"
Nov 02 15:21:43.234375 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:21:43.233557    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c109588b-9544-4a37-bb13-d81572067546-istio-envoy\") pod \"e2e-inf-a-a03e84-openshift-default-8f4cc949-4flt2\" (UID: \"c109588b-9544-4a37-bb13-d81572067546\") " pod="openshift-ingress/e2e-inf-a-a03e84-openshift-default-8f4cc949-4flt2"
Nov 02 15:21:43.237083 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:21:43.237059    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c109588b-9544-4a37-bb13-d81572067546-istio-podinfo\") pod \"e2e-inf-a-a03e84-openshift-default-8f4cc949-4flt2\" (UID: \"c109588b-9544-4a37-bb13-d81572067546\") " pod="openshift-ingress/e2e-inf-a-a03e84-openshift-default-8f4cc949-4flt2"
Nov 02 15:21:43.256328 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:21:43.256292    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c109588b-9544-4a37-bb13-d81572067546-istio-token\") pod \"e2e-inf-a-a03e84-openshift-default-8f4cc949-4flt2\" (UID: \"c109588b-9544-4a37-bb13-d81572067546\") " pod="openshift-ingress/e2e-inf-a-a03e84-openshift-default-8f4cc949-4flt2"
Nov 02 15:21:43.256521 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:21:43.256412    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xs8cd\" (UniqueName: \"kubernetes.io/projected/c109588b-9544-4a37-bb13-d81572067546-kube-api-access-xs8cd\") pod \"e2e-inf-a-a03e84-openshift-default-8f4cc949-4flt2\" (UID: \"c109588b-9544-4a37-bb13-d81572067546\") " pod="openshift-ingress/e2e-inf-a-a03e84-openshift-default-8f4cc949-4flt2"
Nov 02 15:21:43.422090 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:21:43.422055    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-a03e84-openshift-default-8f4cc949-4flt2"
Nov 02 15:21:43.800373 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:21:43.800301    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-a03e84-openshift-default-8f4cc949-4flt2"]
Nov 02 15:21:43.804455 ip-10-0-135-138 kubenswrapper[5304]: W1102 15:21:43.804397    5304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc109588b_9544_4a37_bb13_d81572067546.slice/crio-196549cb08ecda30881e5ff5f8c990e688694927fb0ee40c480f19d0fb6821dc WatchSource:0}: Error finding container 196549cb08ecda30881e5ff5f8c990e688694927fb0ee40c480f19d0fb6821dc: Status 404 returned error can't find the container with id 196549cb08ecda30881e5ff5f8c990e688694927fb0ee40c480f19d0fb6821dc
Nov 02 15:21:43.811885 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:21:43.811830    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:21:43.812050 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:21:43.811929    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:21:43.812050 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:21:43.811978    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:21:43.908514 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:21:43.908478    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a03e84-openshift-default-8f4cc949-4flt2" event={"ID":"c109588b-9544-4a37-bb13-d81572067546","Type":"ContainerStarted","Data":"196549cb08ecda30881e5ff5f8c990e688694927fb0ee40c480f19d0fb6821dc"}
Nov 02 15:21:44.918476 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:21:44.918443    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a03e84-openshift-default-8f4cc949-4flt2" event={"ID":"c109588b-9544-4a37-bb13-d81572067546","Type":"ContainerStarted","Data":"8d68a037a516dffec694fa5bc9008369c95dbe007c3efb2c181b9dc643c4597a"}
Nov 02 15:21:45.018679 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:21:45.018625    5304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-a03e84-openshift-default-8f4cc949-4flt2" podStartSLOduration=3.018609741 podStartE2EDuration="3.018609741s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:21:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:21:45.01405505 +0000 UTC m=+1505.007879458" watchObservedRunningTime="2026-11-02 15:21:45.018609741 +0000 UTC m=+1505.012434150"
Nov 02 15:21:45.422955 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:21:45.422918    5304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-a03e84-openshift-default-8f4cc949-4flt2"
Nov 02 15:21:45.428894 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:21:45.428864    5304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-a03e84-openshift-default-8f4cc949-4flt2"
Nov 02 15:21:45.958205 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:21:45.953628    5304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-4ec02d-openshift-default-86955cc5c9-2j2bw"]
Nov 02 15:21:45.977730 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:21:45.968962    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-4ec02d-openshift-default-86955cc5c9-2j2bw"
Nov 02 15:21:45.994144 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:21:45.983441    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-a03e84-openshift-default-8f4cc949-4flt2"
Nov 02 15:21:45.994144 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:21:45.983987    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-4ec02d-openshift-default-86955cc5c9-2j2bw"]
Nov 02 15:21:45.998151 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:21:45.995755    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-a03e84-openshift-default-8f4cc949-4flt2"
Nov 02 15:21:46.074085 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:21:46.073500    5304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-4ec02d-openshift-default-dockercfg-pt862\""
Nov 02 15:21:46.396518 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:21:46.380677    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de0eb9f2-e614-454a-a294-77fc9830a610-istiod-ca-cert\") pod \"e2e-api-b-4ec02d-openshift-default-86955cc5c9-2j2bw\" (UID: \"de0eb9f2-e614-454a-a294-77fc9830a610\") " pod="openshift-ingress/e2e-api-b-4ec02d-openshift-default-86955cc5c9-2j2bw"
Nov 02 15:21:46.448637 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:21:46.448594    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de0eb9f2-e614-454a-a294-77fc9830a610-istio-data\") pod \"e2e-api-b-4ec02d-openshift-default-86955cc5c9-2j2bw\" (UID: \"de0eb9f2-e614-454a-a294-77fc9830a610\") " pod="openshift-ingress/e2e-api-b-4ec02d-openshift-default-86955cc5c9-2j2bw"
Nov 02 15:21:46.457274 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:21:46.457208    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zfj5n\" (UniqueName: \"kubernetes.io/projected/de0eb9f2-e614-454a-a294-77fc9830a610-kube-api-access-zfj5n\") pod \"e2e-api-b-4ec02d-openshift-default-86955cc5c9-2j2bw\" (UID: \"de0eb9f2-e614-454a-a294-77fc9830a610\") " pod="openshift-ingress/e2e-api-b-4ec02d-openshift-default-86955cc5c9-2j2bw"
Nov 02 15:21:46.471922 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:21:46.465286    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de0eb9f2-e614-454a-a294-77fc9830a610-istio-podinfo\") pod \"e2e-api-b-4ec02d-openshift-default-86955cc5c9-2j2bw\" (UID: \"de0eb9f2-e614-454a-a294-77fc9830a610\") " pod="openshift-ingress/e2e-api-b-4ec02d-openshift-default-86955cc5c9-2j2bw"
Nov 02 15:21:46.477521 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:21:46.476213    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de0eb9f2-e614-454a-a294-77fc9830a610-credential-socket\") pod \"e2e-api-b-4ec02d-openshift-default-86955cc5c9-2j2bw\" (UID: \"de0eb9f2-e614-454a-a294-77fc9830a610\") " pod="openshift-ingress/e2e-api-b-4ec02d-openshift-default-86955cc5c9-2j2bw"
Nov 02 15:21:46.518277 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:21:46.518240    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de0eb9f2-e614-454a-a294-77fc9830a610-istio-envoy\") pod \"e2e-api-b-4ec02d-openshift-default-86955cc5c9-2j2bw\" (UID: \"de0eb9f2-e614-454a-a294-77fc9830a610\") " pod="openshift-ingress/e2e-api-b-4ec02d-openshift-default-86955cc5c9-2j2bw"
Nov 02 15:21:46.518646 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:21:46.518627    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de0eb9f2-e614-454a-a294-77fc9830a610-istio-token\") pod \"e2e-api-b-4ec02d-openshift-default-86955cc5c9-2j2bw\" (UID: \"de0eb9f2-e614-454a-a294-77fc9830a610\") " pod="openshift-ingress/e2e-api-b-4ec02d-openshift-default-86955cc5c9-2j2bw"
Nov 02 15:21:46.518815 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:21:46.518798    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de0eb9f2-e614-454a-a294-77fc9830a610-workload-certs\") pod \"e2e-api-b-4ec02d-openshift-default-86955cc5c9-2j2bw\" (UID: \"de0eb9f2-e614-454a-a294-77fc9830a610\") " pod="openshift-ingress/e2e-api-b-4ec02d-openshift-default-86955cc5c9-2j2bw"
Nov 02 15:21:46.519053 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:21:46.519026    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de0eb9f2-e614-454a-a294-77fc9830a610-workload-socket\") pod \"e2e-api-b-4ec02d-openshift-default-86955cc5c9-2j2bw\" (UID: \"de0eb9f2-e614-454a-a294-77fc9830a610\") " pod="openshift-ingress/e2e-api-b-4ec02d-openshift-default-86955cc5c9-2j2bw"
Nov 02 15:21:46.634778 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:21:46.634738    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zfj5n\" (UniqueName: \"kubernetes.io/projected/de0eb9f2-e614-454a-a294-77fc9830a610-kube-api-access-zfj5n\") pod \"e2e-api-b-4ec02d-openshift-default-86955cc5c9-2j2bw\" (UID: \"de0eb9f2-e614-454a-a294-77fc9830a610\") " pod="openshift-ingress/e2e-api-b-4ec02d-openshift-default-86955cc5c9-2j2bw"
Nov 02 15:21:46.636047 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:21:46.636021    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de0eb9f2-e614-454a-a294-77fc9830a610-istio-podinfo\") pod \"e2e-api-b-4ec02d-openshift-default-86955cc5c9-2j2bw\" (UID: \"de0eb9f2-e614-454a-a294-77fc9830a610\") " pod="openshift-ingress/e2e-api-b-4ec02d-openshift-default-86955cc5c9-2j2bw"
Nov 02 15:21:46.636248 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:21:46.636229    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de0eb9f2-e614-454a-a294-77fc9830a610-credential-socket\") pod \"e2e-api-b-4ec02d-openshift-default-86955cc5c9-2j2bw\" (UID: \"de0eb9f2-e614-454a-a294-77fc9830a610\") " pod="openshift-ingress/e2e-api-b-4ec02d-openshift-default-86955cc5c9-2j2bw"
Nov 02 15:21:46.636396 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:21:46.636380    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de0eb9f2-e614-454a-a294-77fc9830a610-istio-envoy\") pod \"e2e-api-b-4ec02d-openshift-default-86955cc5c9-2j2bw\" (UID: \"de0eb9f2-e614-454a-a294-77fc9830a610\") " pod="openshift-ingress/e2e-api-b-4ec02d-openshift-default-86955cc5c9-2j2bw"
Nov 02 15:21:46.650201 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:21:46.636568    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de0eb9f2-e614-454a-a294-77fc9830a610-istio-token\") pod \"e2e-api-b-4ec02d-openshift-default-86955cc5c9-2j2bw\" (UID: \"de0eb9f2-e614-454a-a294-77fc9830a610\") " pod="openshift-ingress/e2e-api-b-4ec02d-openshift-default-86955cc5c9-2j2bw"
Nov 02 15:21:46.650495 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:21:46.650467    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de0eb9f2-e614-454a-a294-77fc9830a610-workload-certs\") pod \"e2e-api-b-4ec02d-openshift-default-86955cc5c9-2j2bw\" (UID: \"de0eb9f2-e614-454a-a294-77fc9830a610\") " pod="openshift-ingress/e2e-api-b-4ec02d-openshift-default-86955cc5c9-2j2bw"
Nov 02 15:21:46.650720 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:21:46.650694    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de0eb9f2-e614-454a-a294-77fc9830a610-workload-socket\") pod \"e2e-api-b-4ec02d-openshift-default-86955cc5c9-2j2bw\" (UID: \"de0eb9f2-e614-454a-a294-77fc9830a610\") " pod="openshift-ingress/e2e-api-b-4ec02d-openshift-default-86955cc5c9-2j2bw"
Nov 02 15:21:46.650968 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:21:46.650950    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de0eb9f2-e614-454a-a294-77fc9830a610-istiod-ca-cert\") pod \"e2e-api-b-4ec02d-openshift-default-86955cc5c9-2j2bw\" (UID: \"de0eb9f2-e614-454a-a294-77fc9830a610\") " pod="openshift-ingress/e2e-api-b-4ec02d-openshift-default-86955cc5c9-2j2bw"
Nov 02 15:21:46.651158 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:21:46.651140    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de0eb9f2-e614-454a-a294-77fc9830a610-istio-data\") pod \"e2e-api-b-4ec02d-openshift-default-86955cc5c9-2j2bw\" (UID: \"de0eb9f2-e614-454a-a294-77fc9830a610\") " pod="openshift-ingress/e2e-api-b-4ec02d-openshift-default-86955cc5c9-2j2bw"
Nov 02 15:21:46.652518 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:21:46.652487    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de0eb9f2-e614-454a-a294-77fc9830a610-workload-socket\") pod \"e2e-api-b-4ec02d-openshift-default-86955cc5c9-2j2bw\" (UID: \"de0eb9f2-e614-454a-a294-77fc9830a610\") " pod="openshift-ingress/e2e-api-b-4ec02d-openshift-default-86955cc5c9-2j2bw"
Nov 02 15:21:46.652649 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:21:46.648491    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de0eb9f2-e614-454a-a294-77fc9830a610-credential-socket\") pod \"e2e-api-b-4ec02d-openshift-default-86955cc5c9-2j2bw\" (UID: \"de0eb9f2-e614-454a-a294-77fc9830a610\") " pod="openshift-ingress/e2e-api-b-4ec02d-openshift-default-86955cc5c9-2j2bw"
Nov 02 15:21:46.652820 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:21:46.652792    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de0eb9f2-e614-454a-a294-77fc9830a610-workload-certs\") pod \"e2e-api-b-4ec02d-openshift-default-86955cc5c9-2j2bw\" (UID: \"de0eb9f2-e614-454a-a294-77fc9830a610\") " pod="openshift-ingress/e2e-api-b-4ec02d-openshift-default-86955cc5c9-2j2bw"
Nov 02 15:21:46.654324 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:21:46.653211    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de0eb9f2-e614-454a-a294-77fc9830a610-istio-data\") pod \"e2e-api-b-4ec02d-openshift-default-86955cc5c9-2j2bw\" (UID: \"de0eb9f2-e614-454a-a294-77fc9830a610\") " pod="openshift-ingress/e2e-api-b-4ec02d-openshift-default-86955cc5c9-2j2bw"
Nov 02 15:21:46.654963 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:21:46.654897    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de0eb9f2-e614-454a-a294-77fc9830a610-istiod-ca-cert\") pod \"e2e-api-b-4ec02d-openshift-default-86955cc5c9-2j2bw\" (UID: \"de0eb9f2-e614-454a-a294-77fc9830a610\") " pod="openshift-ingress/e2e-api-b-4ec02d-openshift-default-86955cc5c9-2j2bw"
Nov 02 15:21:46.679174 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:21:46.678652    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de0eb9f2-e614-454a-a294-77fc9830a610-istio-podinfo\") pod \"e2e-api-b-4ec02d-openshift-default-86955cc5c9-2j2bw\" (UID: \"de0eb9f2-e614-454a-a294-77fc9830a610\") " pod="openshift-ingress/e2e-api-b-4ec02d-openshift-default-86955cc5c9-2j2bw"
Nov 02 15:21:46.684191 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:21:46.682410    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de0eb9f2-e614-454a-a294-77fc9830a610-istio-envoy\") pod \"e2e-api-b-4ec02d-openshift-default-86955cc5c9-2j2bw\" (UID: \"de0eb9f2-e614-454a-a294-77fc9830a610\") " pod="openshift-ingress/e2e-api-b-4ec02d-openshift-default-86955cc5c9-2j2bw"
Nov 02 15:21:46.690162 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:21:46.687847    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de0eb9f2-e614-454a-a294-77fc9830a610-istio-token\") pod \"e2e-api-b-4ec02d-openshift-default-86955cc5c9-2j2bw\" (UID: \"de0eb9f2-e614-454a-a294-77fc9830a610\") " pod="openshift-ingress/e2e-api-b-4ec02d-openshift-default-86955cc5c9-2j2bw"
Nov 02 15:21:46.701614 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:21:46.697791    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zfj5n\" (UniqueName: \"kubernetes.io/projected/de0eb9f2-e614-454a-a294-77fc9830a610-kube-api-access-zfj5n\") pod \"e2e-api-b-4ec02d-openshift-default-86955cc5c9-2j2bw\" (UID: \"de0eb9f2-e614-454a-a294-77fc9830a610\") " pod="openshift-ingress/e2e-api-b-4ec02d-openshift-default-86955cc5c9-2j2bw"
Nov 02 15:21:46.968835 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:21:46.968465    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-4ec02d-openshift-default-86955cc5c9-2j2bw"
Nov 02 15:21:46.998898 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:21:46.998863    5304 generic.go:356] "Generic (PLEG): container finished" podID="c34a9870-db4c-462a-827f-95abdfc1f68b" containerID="7553fcc74706c6f700066b6fa868972f26203f2905a05f1a775b6ebb40fa53cf" exitCode=0
Nov 02 15:21:46.999499 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:21:46.999473    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-5f9166-openshift-default-5d4b56b869-skdww" event={"ID":"c34a9870-db4c-462a-827f-95abdfc1f68b","Type":"ContainerDied","Data":"7553fcc74706c6f700066b6fa868972f26203f2905a05f1a775b6ebb40fa53cf"}
Nov 02 15:21:46.999692 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:21:46.999676    5304 scope.go:122] "RemoveContainer" containerID="5c8aa0912102f7f4905ec45c8ba28cc09a31ea54fa487ffa50530f601472732a"
Nov 02 15:21:47.461153 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:21:47.457341    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-5f9166-openshift-default-5d4b56b869-skdww"
Nov 02 15:21:47.466583 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:21:47.466554    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-4ec02d-openshift-default-86955cc5c9-2j2bw"]
Nov 02 15:21:47.469096 ip-10-0-135-138 kubenswrapper[5304]: W1102 15:21:47.469064    5304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde0eb9f2_e614_454a_a294_77fc9830a610.slice/crio-2347819a82b76a47c127de3314c770b3dff2a928f35aaa3eb03b8bfb9d28063d WatchSource:0}: Error finding container 2347819a82b76a47c127de3314c770b3dff2a928f35aaa3eb03b8bfb9d28063d: Status 404 returned error can't find the container with id 2347819a82b76a47c127de3314c770b3dff2a928f35aaa3eb03b8bfb9d28063d
Nov 02 15:21:47.472313 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:21:47.472273    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:21:47.472438 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:21:47.472394    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:21:47.472507 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:21:47.472462    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:21:47.591171 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:21:47.591121    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c34a9870-db4c-462a-827f-95abdfc1f68b-istio-token\" (UniqueName: \"kubernetes.io/projected/c34a9870-db4c-462a-827f-95abdfc1f68b-istio-token\") pod \"c34a9870-db4c-462a-827f-95abdfc1f68b\" (UID: \"c34a9870-db4c-462a-827f-95abdfc1f68b\") "
Nov 02 15:21:47.591332 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:21:47.591315    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c34a9870-db4c-462a-827f-95abdfc1f68b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c34a9870-db4c-462a-827f-95abdfc1f68b-credential-socket\") pod \"c34a9870-db4c-462a-827f-95abdfc1f68b\" (UID: \"c34a9870-db4c-462a-827f-95abdfc1f68b\") "
Nov 02 15:21:47.591446 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:21:47.591414    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c34a9870-db4c-462a-827f-95abdfc1f68b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c34a9870-db4c-462a-827f-95abdfc1f68b-istio-envoy\") pod \"c34a9870-db4c-462a-827f-95abdfc1f68b\" (UID: \"c34a9870-db4c-462a-827f-95abdfc1f68b\") "
Nov 02 15:21:47.591603 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:21:47.591465    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c34a9870-db4c-462a-827f-95abdfc1f68b-kube-api-access-zhh54\" (UniqueName: \"kubernetes.io/projected/c34a9870-db4c-462a-827f-95abdfc1f68b-kube-api-access-zhh54\") pod \"c34a9870-db4c-462a-827f-95abdfc1f68b\" (UID: \"c34a9870-db4c-462a-827f-95abdfc1f68b\") "
Nov 02 15:21:47.591603 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:21:47.591511    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c34a9870-db4c-462a-827f-95abdfc1f68b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c34a9870-db4c-462a-827f-95abdfc1f68b-istio-podinfo\") pod \"c34a9870-db4c-462a-827f-95abdfc1f68b\" (UID: \"c34a9870-db4c-462a-827f-95abdfc1f68b\") "
Nov 02 15:21:47.591603 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:21:47.591540    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c34a9870-db4c-462a-827f-95abdfc1f68b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c34a9870-db4c-462a-827f-95abdfc1f68b-workload-certs\") pod \"c34a9870-db4c-462a-827f-95abdfc1f68b\" (UID: \"c34a9870-db4c-462a-827f-95abdfc1f68b\") "
Nov 02 15:21:47.591603 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:21:47.591580    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c34a9870-db4c-462a-827f-95abdfc1f68b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c34a9870-db4c-462a-827f-95abdfc1f68b-istiod-ca-cert\") pod \"c34a9870-db4c-462a-827f-95abdfc1f68b\" (UID: \"c34a9870-db4c-462a-827f-95abdfc1f68b\") "
Nov 02 15:21:47.591804 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:21:47.591738    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c34a9870-db4c-462a-827f-95abdfc1f68b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c34a9870-db4c-462a-827f-95abdfc1f68b-istio-data\") pod \"c34a9870-db4c-462a-827f-95abdfc1f68b\" (UID: \"c34a9870-db4c-462a-827f-95abdfc1f68b\") "
Nov 02 15:21:47.591804 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:21:47.591744    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c34a9870-db4c-462a-827f-95abdfc1f68b-credential-socket" pod "c34a9870-db4c-462a-827f-95abdfc1f68b" (UID: "c34a9870-db4c-462a-827f-95abdfc1f68b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:21:47.591804 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:21:47.591767    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c34a9870-db4c-462a-827f-95abdfc1f68b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c34a9870-db4c-462a-827f-95abdfc1f68b-workload-socket\") pod \"c34a9870-db4c-462a-827f-95abdfc1f68b\" (UID: \"c34a9870-db4c-462a-827f-95abdfc1f68b\") "
Nov 02 15:21:47.592151 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:21:47.592081    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c34a9870-db4c-462a-827f-95abdfc1f68b-istiod-ca-cert" pod "c34a9870-db4c-462a-827f-95abdfc1f68b" (UID: "c34a9870-db4c-462a-827f-95abdfc1f68b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:21:47.592250 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:21:47.592213    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c34a9870-db4c-462a-827f-95abdfc1f68b-workload-certs" pod "c34a9870-db4c-462a-827f-95abdfc1f68b" (UID: "c34a9870-db4c-462a-827f-95abdfc1f68b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:21:47.592411 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:21:47.592372    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c34a9870-db4c-462a-827f-95abdfc1f68b-istio-data" pod "c34a9870-db4c-462a-827f-95abdfc1f68b" (UID: "c34a9870-db4c-462a-827f-95abdfc1f68b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:21:47.595253 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:21:47.592532    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c34a9870-db4c-462a-827f-95abdfc1f68b-workload-socket" pod "c34a9870-db4c-462a-827f-95abdfc1f68b" (UID: "c34a9870-db4c-462a-827f-95abdfc1f68b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:21:47.595253 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:21:47.593166    5304 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c34a9870-db4c-462a-827f-95abdfc1f68b-credential-socket\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:21:47.595253 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:21:47.593189    5304 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c34a9870-db4c-462a-827f-95abdfc1f68b-workload-certs\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:21:47.595253 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:21:47.593204    5304 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c34a9870-db4c-462a-827f-95abdfc1f68b-istiod-ca-cert\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:21:47.595253 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:21:47.593217    5304 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c34a9870-db4c-462a-827f-95abdfc1f68b-istio-data\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:21:47.595253 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:21:47.593231    5304 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c34a9870-db4c-462a-827f-95abdfc1f68b-workload-socket\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:21:47.596243 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:21:47.596217    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c34a9870-db4c-462a-827f-95abdfc1f68b-istio-envoy" pod "c34a9870-db4c-462a-827f-95abdfc1f68b" (UID: "c34a9870-db4c-462a-827f-95abdfc1f68b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:21:47.596349 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:21:47.596248    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c34a9870-db4c-462a-827f-95abdfc1f68b-istio-podinfo" pod "c34a9870-db4c-462a-827f-95abdfc1f68b" (UID: "c34a9870-db4c-462a-827f-95abdfc1f68b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 15:21:47.596349 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:21:47.596301    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c34a9870-db4c-462a-827f-95abdfc1f68b-istio-token" pod "c34a9870-db4c-462a-827f-95abdfc1f68b" (UID: "c34a9870-db4c-462a-827f-95abdfc1f68b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:21:47.596349 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:21:47.596335    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c34a9870-db4c-462a-827f-95abdfc1f68b-kube-api-access-zhh54" pod "c34a9870-db4c-462a-827f-95abdfc1f68b" (UID: "c34a9870-db4c-462a-827f-95abdfc1f68b"). InnerVolumeSpecName "kube-api-access-zhh54". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:21:47.695104 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:21:47.694944    5304 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c34a9870-db4c-462a-827f-95abdfc1f68b-istio-envoy\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:21:47.695554 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:21:47.695535    5304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zhh54\" (UniqueName: \"kubernetes.io/projected/c34a9870-db4c-462a-827f-95abdfc1f68b-kube-api-access-zhh54\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:21:47.695690 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:21:47.695677    5304 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c34a9870-db4c-462a-827f-95abdfc1f68b-istio-podinfo\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:21:47.695819 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:21:47.695788    5304 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c34a9870-db4c-462a-827f-95abdfc1f68b-istio-token\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:21:48.011069 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:21:48.010964    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-4ec02d-openshift-default-86955cc5c9-2j2bw" event={"ID":"de0eb9f2-e614-454a-a294-77fc9830a610","Type":"ContainerStarted","Data":"9f2169beab40a8765fdfcbb363eb57d18bd15f131072d9c4e777d228c1dfabff"}
Nov 02 15:21:48.011069 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:21:48.011027    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-4ec02d-openshift-default-86955cc5c9-2j2bw" event={"ID":"de0eb9f2-e614-454a-a294-77fc9830a610","Type":"ContainerStarted","Data":"2347819a82b76a47c127de3314c770b3dff2a928f35aaa3eb03b8bfb9d28063d"}
Nov 02 15:21:48.013013 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:21:48.012986    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-5f9166-openshift-default-5d4b56b869-skdww"
Nov 02 15:21:48.013196 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:21:48.013037    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-5f9166-openshift-default-5d4b56b869-skdww" event={"ID":"c34a9870-db4c-462a-827f-95abdfc1f68b","Type":"ContainerDied","Data":"ff2d4854556f6225c48e381df0d0b7bc547d3e8ebba40c57afb3c9466778ff6a"}
Nov 02 15:21:48.013196 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:21:48.013088    5304 scope.go:122] "RemoveContainer" containerID="7553fcc74706c6f700066b6fa868972f26203f2905a05f1a775b6ebb40fa53cf"
Nov 02 15:21:48.081925 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:21:48.081850    5304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-4ec02d-openshift-default-86955cc5c9-2j2bw" podStartSLOduration=3.081830624 podStartE2EDuration="3.081830624s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:21:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:21:48.076326864 +0000 UTC m=+1508.070151294" watchObservedRunningTime="2026-11-02 15:21:48.081830624 +0000 UTC m=+1508.075655033"
Nov 02 15:21:48.126492 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:21:48.126450    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-5f9166-openshift-default-5d4b56b869-skdww"]
Nov 02 15:21:48.126713 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:21:48.126512    5304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-5f9166-openshift-default-5d4b56b869-skdww"]
Nov 02 15:21:48.972111 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:21:48.972075    5304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-4ec02d-openshift-default-86955cc5c9-2j2bw"
Nov 02 15:21:48.978072 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:21:48.978042    5304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-4ec02d-openshift-default-86955cc5c9-2j2bw"
Nov 02 15:21:49.021541 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:21:49.021509    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-4ec02d-openshift-default-86955cc5c9-2j2bw"
Nov 02 15:21:49.022356 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:21:49.022335    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-4ec02d-openshift-default-86955cc5c9-2j2bw"
Nov 02 15:21:49.800899 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:21:49.800866    5304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-a03e84-65c9f56b59-jxxpb"]
Nov 02 15:21:49.801635 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:21:49.801619    5304 cpu_manager.go:414] "containerMap: removing container" podUID="c34a9870-db4c-462a-827f-95abdfc1f68b" containerName="istio-proxy"
Nov 02 15:21:49.801703 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:21:49.801636    5304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c34a9870-db4c-462a-827f-95abdfc1f68b" containerName="istio-proxy"
Nov 02 15:21:49.801703 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:21:49.801651    5304 cpu_manager.go:414] "containerMap: removing container" podUID="c34a9870-db4c-462a-827f-95abdfc1f68b" containerName="istio-proxy"
Nov 02 15:21:49.801703 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:21:49.801656    5304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c34a9870-db4c-462a-827f-95abdfc1f68b" containerName="istio-proxy"
Nov 02 15:21:49.801804 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:21:49.801768    5304 memory_manager.go:365] "RemoveStaleState removing state" podUID="c34a9870-db4c-462a-827f-95abdfc1f68b" containerName="istio-proxy"
Nov 02 15:21:49.801804 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:21:49.801779    5304 memory_manager.go:365] "RemoveStaleState removing state" podUID="c34a9870-db4c-462a-827f-95abdfc1f68b" containerName="istio-proxy"
Nov 02 15:21:49.806698 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:21:49.806680    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a03e84-65c9f56b59-jxxpb"
Nov 02 15:21:49.823382 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:21:49.823354    5304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-a03e84-dockercfg-szfwq\""
Nov 02 15:21:49.823506 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:21:49.823460    5304 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-a03e84\""
Nov 02 15:21:49.825081 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:21:49.825053    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-a03e84-65c9f56b59-jxxpb"]
Nov 02 15:21:49.972851 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:21:49.971140    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f8f4139-61bb-41be-828c-b1d6a7692ee2-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-a03e84-65c9f56b59-jxxpb\" (UID: \"7f8f4139-61bb-41be-828c-b1d6a7692ee2\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a03e84-65c9f56b59-jxxpb"
Nov 02 15:21:49.972851 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:21:49.971280    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f6287\" (UniqueName: \"kubernetes.io/projected/7f8f4139-61bb-41be-828c-b1d6a7692ee2-kube-api-access-f6287\") pod \"payload-pre-processing-e2e-inf-a-a03e84-65c9f56b59-jxxpb\" (UID: \"7f8f4139-61bb-41be-828c-b1d6a7692ee2\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a03e84-65c9f56b59-jxxpb"
Nov 02 15:21:50.011147 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:21:49.985633    5304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-a03e84-85f8449979-gd8nh"]
Nov 02 15:21:50.038549 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:21:50.012870    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-a03e84-85f8449979-gd8nh"
Nov 02 15:21:50.114154 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:21:50.097839    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dvh5p\" (UniqueName: \"kubernetes.io/projected/29b94683-c6c9-495d-a0bf-1036f9b5bc3b-kube-api-access-dvh5p\") pod \"payload-processing-e2e-inf-a-a03e84-85f8449979-gd8nh\" (UID: \"29b94683-c6c9-495d-a0bf-1036f9b5bc3b\") " pod="openshift-ingress/payload-processing-e2e-inf-a-a03e84-85f8449979-gd8nh"
Nov 02 15:21:50.114154 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:21:50.097942    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f8f4139-61bb-41be-828c-b1d6a7692ee2-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-a03e84-65c9f56b59-jxxpb\" (UID: \"7f8f4139-61bb-41be-828c-b1d6a7692ee2\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a03e84-65c9f56b59-jxxpb"
Nov 02 15:21:50.114154 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:21:50.098019    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f6287\" (UniqueName: \"kubernetes.io/projected/7f8f4139-61bb-41be-828c-b1d6a7692ee2-kube-api-access-f6287\") pod \"payload-pre-processing-e2e-inf-a-a03e84-65c9f56b59-jxxpb\" (UID: \"7f8f4139-61bb-41be-828c-b1d6a7692ee2\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a03e84-65c9f56b59-jxxpb"
Nov 02 15:21:50.114154 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:21:50.098163    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29b94683-c6c9-495d-a0bf-1036f9b5bc3b-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-a03e84-85f8449979-gd8nh\" (UID: \"29b94683-c6c9-495d-a0bf-1036f9b5bc3b\") " pod="openshift-ingress/payload-processing-e2e-inf-a-a03e84-85f8449979-gd8nh"
Nov 02 15:21:50.114154 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:21:50.101435    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f8f4139-61bb-41be-828c-b1d6a7692ee2-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-a03e84-65c9f56b59-jxxpb\" (UID: \"7f8f4139-61bb-41be-828c-b1d6a7692ee2\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a03e84-65c9f56b59-jxxpb"
Nov 02 15:21:50.227798 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:21:50.209170    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29b94683-c6c9-495d-a0bf-1036f9b5bc3b-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-a03e84-85f8449979-gd8nh\" (UID: \"29b94683-c6c9-495d-a0bf-1036f9b5bc3b\") " pod="openshift-ingress/payload-processing-e2e-inf-a-a03e84-85f8449979-gd8nh"
Nov 02 15:21:50.228683 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:21:50.228614    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29b94683-c6c9-495d-a0bf-1036f9b5bc3b-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-a03e84-85f8449979-gd8nh\" (UID: \"29b94683-c6c9-495d-a0bf-1036f9b5bc3b\") " pod="openshift-ingress/payload-processing-e2e-inf-a-a03e84-85f8449979-gd8nh"
Nov 02 15:21:50.228833 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:21:50.228669    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dvh5p\" (UniqueName: \"kubernetes.io/projected/29b94683-c6c9-495d-a0bf-1036f9b5bc3b-kube-api-access-dvh5p\") pod \"payload-processing-e2e-inf-a-a03e84-85f8449979-gd8nh\" (UID: \"29b94683-c6c9-495d-a0bf-1036f9b5bc3b\") " pod="openshift-ingress/payload-processing-e2e-inf-a-a03e84-85f8449979-gd8nh"
Nov 02 15:21:50.283985 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:21:50.283937    5304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c34a9870-db4c-462a-827f-95abdfc1f68b" path="/var/lib/kubelet/pods/c34a9870-db4c-462a-827f-95abdfc1f68b/volumes"
Nov 02 15:21:50.284506 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:21:50.284476    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-a03e84-85f8449979-gd8nh"]
Nov 02 15:21:50.288739 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:21:50.288079    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dvh5p\" (UniqueName: \"kubernetes.io/projected/29b94683-c6c9-495d-a0bf-1036f9b5bc3b-kube-api-access-dvh5p\") pod \"payload-processing-e2e-inf-a-a03e84-85f8449979-gd8nh\" (UID: \"29b94683-c6c9-495d-a0bf-1036f9b5bc3b\") " pod="openshift-ingress/payload-processing-e2e-inf-a-a03e84-85f8449979-gd8nh"
Nov 02 15:21:50.291879 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:21:50.291856    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-a03e84-85f8449979-gd8nh"
Nov 02 15:21:50.300010 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:21:50.299936    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f6287\" (UniqueName: \"kubernetes.io/projected/7f8f4139-61bb-41be-828c-b1d6a7692ee2-kube-api-access-f6287\") pod \"payload-pre-processing-e2e-inf-a-a03e84-65c9f56b59-jxxpb\" (UID: \"7f8f4139-61bb-41be-828c-b1d6a7692ee2\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a03e84-65c9f56b59-jxxpb"
Nov 02 15:21:50.420850 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:21:50.420819    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a03e84-65c9f56b59-jxxpb"
Nov 02 15:21:50.665544 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:21:50.665448    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-a03e84-85f8449979-gd8nh"]
Nov 02 15:21:50.671602 ip-10-0-135-138 kubenswrapper[5304]: W1102 15:21:50.671572    5304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29b94683_c6c9_495d_a0bf_1036f9b5bc3b.slice/crio-b5e763437e62308b5b46508a4b3c7b683a2e461c9cc0249024b65484495d3c50 WatchSource:0}: Error finding container b5e763437e62308b5b46508a4b3c7b683a2e461c9cc0249024b65484495d3c50: Status 404 returned error can't find the container with id b5e763437e62308b5b46508a4b3c7b683a2e461c9cc0249024b65484495d3c50
Nov 02 15:21:50.782529 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:21:50.782498    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-a03e84-65c9f56b59-jxxpb"]
Nov 02 15:21:50.785042 ip-10-0-135-138 kubenswrapper[5304]: W1102 15:21:50.785017    5304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f8f4139_61bb_41be_828c_b1d6a7692ee2.slice/crio-3acb31e09ee4b7502d1bf20bae046c1cccdcd6eb2179524028e39168a05fb9bc WatchSource:0}: Error finding container 3acb31e09ee4b7502d1bf20bae046c1cccdcd6eb2179524028e39168a05fb9bc: Status 404 returned error can't find the container with id 3acb31e09ee4b7502d1bf20bae046c1cccdcd6eb2179524028e39168a05fb9bc
Nov 02 15:21:51.299504 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:21:51.297826    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-a03e84-85f8449979-gd8nh" event={"ID":"29b94683-c6c9-495d-a0bf-1036f9b5bc3b","Type":"ContainerStarted","Data":"319b401588ef2445d3a689ea6983dce27c9de3526f4c0ffdeb1bbf72fbc5b19a"}
Nov 02 15:21:51.299504 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:21:51.297896    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-a03e84-85f8449979-gd8nh" event={"ID":"29b94683-c6c9-495d-a0bf-1036f9b5bc3b","Type":"ContainerStarted","Data":"b5e763437e62308b5b46508a4b3c7b683a2e461c9cc0249024b65484495d3c50"}
Nov 02 15:21:51.299504 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:21:51.298308    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-a03e84-85f8449979-gd8nh"
Nov 02 15:21:51.301547 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:21:51.301265    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a03e84-65c9f56b59-jxxpb" event={"ID":"7f8f4139-61bb-41be-828c-b1d6a7692ee2","Type":"ContainerStarted","Data":"f1d9eb9cc04eb0d2e0bf4f3a27c9aa986b120c6af79498d15b93b690a812382c"}
Nov 02 15:21:51.301547 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:21:51.301325    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a03e84-65c9f56b59-jxxpb" event={"ID":"7f8f4139-61bb-41be-828c-b1d6a7692ee2","Type":"ContainerStarted","Data":"3acb31e09ee4b7502d1bf20bae046c1cccdcd6eb2179524028e39168a05fb9bc"}
Nov 02 15:21:51.301547 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:21:51.301371    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a03e84-65c9f56b59-jxxpb"
Nov 02 15:21:51.362150 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:21:51.351205    5304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-a03e84-85f8449979-gd8nh" podStartSLOduration=2.351186159 podStartE2EDuration="2.351186159s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:21:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:21:51.350717564 +0000 UTC m=+1511.344541971" watchObservedRunningTime="2026-11-02 15:21:51.351186159 +0000 UTC m=+1511.345010574"
Nov 02 15:21:51.397059 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:21:51.397005    5304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a03e84-65c9f56b59-jxxpb" podStartSLOduration=2.3969901 podStartE2EDuration="2.3969901s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:21:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:21:51.392455394 +0000 UTC m=+1511.386279784" watchObservedRunningTime="2026-11-02 15:21:51.3969901 +0000 UTC m=+1511.390814509"
Nov 02 15:21:53.131147 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:21:53.112271    5304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-4ec02d-8498bbbfd9-gm4hk"]
Nov 02 15:21:53.243958 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:21:53.243920    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-4ec02d-8498bbbfd9-gm4hk"
Nov 02 15:21:53.273778 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:21:53.258743    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-4ec02d-8498bbbfd9-gm4hk"]
Nov 02 15:21:53.273778 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:21:53.264778    5304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-4ec02d-dockercfg-jpfq8\""
Nov 02 15:21:53.277489 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:21:53.277405    5304 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-4ec02d\""
Nov 02 15:21:53.353432 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:21:53.353398    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/803c30b4-dde9-4d5f-9571-004152ae6d80-plugins-config-volume\") pod \"payload-processing-e2e-api-b-4ec02d-8498bbbfd9-gm4hk\" (UID: \"803c30b4-dde9-4d5f-9571-004152ae6d80\") " pod="openshift-ingress/payload-processing-e2e-api-b-4ec02d-8498bbbfd9-gm4hk"
Nov 02 15:21:53.353787 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:21:53.353761    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q2wv9\" (UniqueName: \"kubernetes.io/projected/803c30b4-dde9-4d5f-9571-004152ae6d80-kube-api-access-q2wv9\") pod \"payload-processing-e2e-api-b-4ec02d-8498bbbfd9-gm4hk\" (UID: \"803c30b4-dde9-4d5f-9571-004152ae6d80\") " pod="openshift-ingress/payload-processing-e2e-api-b-4ec02d-8498bbbfd9-gm4hk"
Nov 02 15:21:53.455363 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:21:53.455269    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/803c30b4-dde9-4d5f-9571-004152ae6d80-plugins-config-volume\") pod \"payload-processing-e2e-api-b-4ec02d-8498bbbfd9-gm4hk\" (UID: \"803c30b4-dde9-4d5f-9571-004152ae6d80\") " pod="openshift-ingress/payload-processing-e2e-api-b-4ec02d-8498bbbfd9-gm4hk"
Nov 02 15:21:53.455545 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:21:53.455504    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q2wv9\" (UniqueName: \"kubernetes.io/projected/803c30b4-dde9-4d5f-9571-004152ae6d80-kube-api-access-q2wv9\") pod \"payload-processing-e2e-api-b-4ec02d-8498bbbfd9-gm4hk\" (UID: \"803c30b4-dde9-4d5f-9571-004152ae6d80\") " pod="openshift-ingress/payload-processing-e2e-api-b-4ec02d-8498bbbfd9-gm4hk"
Nov 02 15:21:53.456230 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:21:53.456192    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/803c30b4-dde9-4d5f-9571-004152ae6d80-plugins-config-volume\") pod \"payload-processing-e2e-api-b-4ec02d-8498bbbfd9-gm4hk\" (UID: \"803c30b4-dde9-4d5f-9571-004152ae6d80\") " pod="openshift-ingress/payload-processing-e2e-api-b-4ec02d-8498bbbfd9-gm4hk"
Nov 02 15:21:53.482400 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:21:53.479596    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q2wv9\" (UniqueName: \"kubernetes.io/projected/803c30b4-dde9-4d5f-9571-004152ae6d80-kube-api-access-q2wv9\") pod \"payload-processing-e2e-api-b-4ec02d-8498bbbfd9-gm4hk\" (UID: \"803c30b4-dde9-4d5f-9571-004152ae6d80\") " pod="openshift-ingress/payload-processing-e2e-api-b-4ec02d-8498bbbfd9-gm4hk"
Nov 02 15:21:53.628107 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:21:53.628062    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-4ec02d-8498bbbfd9-gm4hk"
Nov 02 15:21:53.953473 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:21:53.953441    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-4ec02d-8498bbbfd9-gm4hk"]
Nov 02 15:21:53.957154 ip-10-0-135-138 kubenswrapper[5304]: W1102 15:21:53.957106    5304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod803c30b4_dde9_4d5f_9571_004152ae6d80.slice/crio-eced5af9bdd6e9a6c8fd2006777c2e2b068fe3cf2735d08c40cf1749bcd22f23 WatchSource:0}: Error finding container eced5af9bdd6e9a6c8fd2006777c2e2b068fe3cf2735d08c40cf1749bcd22f23: Status 404 returned error can't find the container with id eced5af9bdd6e9a6c8fd2006777c2e2b068fe3cf2735d08c40cf1749bcd22f23
Nov 02 15:21:54.342497 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:21:54.342438    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-4ec02d-8498bbbfd9-gm4hk" event={"ID":"803c30b4-dde9-4d5f-9571-004152ae6d80","Type":"ContainerStarted","Data":"8e04479b579a736c3de51c8dac7f6f82b3488f37a3abe1b8bb085fbe87f83de1"}
Nov 02 15:21:54.342497 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:21:54.342493    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-4ec02d-8498bbbfd9-gm4hk" event={"ID":"803c30b4-dde9-4d5f-9571-004152ae6d80","Type":"ContainerStarted","Data":"eced5af9bdd6e9a6c8fd2006777c2e2b068fe3cf2735d08c40cf1749bcd22f23"}
Nov 02 15:21:54.342989 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:21:54.342682    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-4ec02d-8498bbbfd9-gm4hk"
Nov 02 15:21:54.383189 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:21:54.383107    5304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-4ec02d-8498bbbfd9-gm4hk" podStartSLOduration=2.383090565 podStartE2EDuration="2.383090565s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:21:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:21:54.382610097 +0000 UTC m=+1514.376434507" watchObservedRunningTime="2026-11-02 15:21:54.383090565 +0000 UTC m=+1514.376914973"
Nov 02 15:21:54.802945 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:21:54.802863    5304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k6fjk" containerName="extractor"
Nov 02 15:21:57.383343 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:21:57.383308    5304 generic.go:356] "Generic (PLEG): container finished" podID="ba7e1e2d-fe7e-41ab-8fb0-814e76dfb08b" containerID="8662bbc92abbc0a7254dedc4fea3f21fca49213a2da12c7d8f39939caad6a109" exitCode=0
Nov 02 15:21:57.383873 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:21:57.383351    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-db75ff-openshift-default-7558cfb4df-qnbxl" event={"ID":"ba7e1e2d-fe7e-41ab-8fb0-814e76dfb08b","Type":"ContainerDied","Data":"8662bbc92abbc0a7254dedc4fea3f21fca49213a2da12c7d8f39939caad6a109"}
Nov 02 15:21:57.383873 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:21:57.383837    5304 scope.go:122] "RemoveContainer" containerID="8662bbc92abbc0a7254dedc4fea3f21fca49213a2da12c7d8f39939caad6a109"
Nov 02 15:21:57.385909 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:21:57.385870    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:21:57.386027 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:21:57.385954    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:21:57.386027 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:21:57.385991    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:21:58.393021 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:21:58.392987    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-db75ff-openshift-default-7558cfb4df-qnbxl" event={"ID":"ba7e1e2d-fe7e-41ab-8fb0-814e76dfb08b","Type":"ContainerStarted","Data":"9d8770c74296cf691bbfd200950a31905a1098d7b37e6f0cc16a066cd790bafd"}
Nov 02 15:21:58.456602 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:21:58.456562    5304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-db75ff-openshift-default-7558cfb4df-qnbxl"
Nov 02 15:21:58.456821 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:21:58.456769    5304 patch_prober.go:29] interesting pod/e2e-api-a-db75ff-openshift-default-7558cfb4df-qnbxl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body=
Nov 02 15:21:58.456902 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:21:58.456840    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-db75ff-openshift-default-7558cfb4df-qnbxl" podUID="ba7e1e2d-fe7e-41ab-8fb0-814e76dfb08b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused"
Nov 02 15:21:59.456677 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:21:59.456637    5304 patch_prober.go:29] interesting pod/e2e-api-a-db75ff-openshift-default-7558cfb4df-qnbxl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body=
Nov 02 15:21:59.457161 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:21:59.456723    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-db75ff-openshift-default-7558cfb4df-qnbxl" podUID="ba7e1e2d-fe7e-41ab-8fb0-814e76dfb08b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused"
Nov 02 15:22:00.456772 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:22:00.456734    5304 patch_prober.go:29] interesting pod/e2e-api-a-db75ff-openshift-default-7558cfb4df-qnbxl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body=
Nov 02 15:22:00.457381 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:22:00.457351    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-db75ff-openshift-default-7558cfb4df-qnbxl" podUID="ba7e1e2d-fe7e-41ab-8fb0-814e76dfb08b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused"
Nov 02 15:22:01.456644 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:22:01.456603    5304 patch_prober.go:29] interesting pod/e2e-api-a-db75ff-openshift-default-7558cfb4df-qnbxl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body=
Nov 02 15:22:01.456854 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:22:01.456662    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-db75ff-openshift-default-7558cfb4df-qnbxl" podUID="ba7e1e2d-fe7e-41ab-8fb0-814e76dfb08b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused"
Nov 02 15:22:02.314503 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:22:02.314471    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-a03e84-85f8449979-gd8nh"
Nov 02 15:22:02.314708 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:22:02.314541    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a03e84-65c9f56b59-jxxpb"
Nov 02 15:22:02.461654 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:22:02.461618    5304 patch_prober.go:29] interesting pod/e2e-api-a-db75ff-openshift-default-7558cfb4df-qnbxl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body=
Nov 02 15:22:02.462285 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:22:02.462245    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-db75ff-openshift-default-7558cfb4df-qnbxl" podUID="ba7e1e2d-fe7e-41ab-8fb0-814e76dfb08b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused"
Nov 02 15:22:02.971748 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:22:02.970857    5304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-2bbcb5/test-model-2bbcb5-kserve-9597db49f-g6cdn"]
Nov 02 15:22:03.189689 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:22:03.189652    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-db75ff-openshift-default-7558cfb4df-qnbxl"
Nov 02 15:22:03.191059 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:22:03.191033    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-2bbcb5/test-model-2bbcb5-kserve-9597db49f-g6cdn"
Nov 02 15:22:03.214161 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:22:03.208979    5304 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-2bbcb5\"/\"kube-root-ca.crt\""
Nov 02 15:22:03.235567 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:22:03.235488    5304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-2bbcb5\"/\"test-model-2bbcb5-kserve-self-signed-certs\""
Nov 02 15:22:03.235956 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:22:03.235937    5304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-2bbcb5\"/\"default-dockercfg-mqbqk\""
Nov 02 15:22:03.365152 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:22:03.352604    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f3c417d3-fdf3-4951-8246-dcd24d7e3820-tls-certs\") pod \"test-model-2bbcb5-kserve-9597db49f-g6cdn\" (UID: \"f3c417d3-fdf3-4951-8246-dcd24d7e3820\") " pod="ai-tenant-e2e-inf-b-2bbcb5/test-model-2bbcb5-kserve-9597db49f-g6cdn"
Nov 02 15:22:03.420774 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:22:03.420734    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f3c417d3-fdf3-4951-8246-dcd24d7e3820-home\") pod \"test-model-2bbcb5-kserve-9597db49f-g6cdn\" (UID: \"f3c417d3-fdf3-4951-8246-dcd24d7e3820\") " pod="ai-tenant-e2e-inf-b-2bbcb5/test-model-2bbcb5-kserve-9597db49f-g6cdn"
Nov 02 15:22:03.421096 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:22:03.421073    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f3c417d3-fdf3-4951-8246-dcd24d7e3820-dshm\") pod \"test-model-2bbcb5-kserve-9597db49f-g6cdn\" (UID: \"f3c417d3-fdf3-4951-8246-dcd24d7e3820\") " pod="ai-tenant-e2e-inf-b-2bbcb5/test-model-2bbcb5-kserve-9597db49f-g6cdn"
Nov 02 15:22:03.421344 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:22:03.421320    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vq67h\" (UniqueName: \"kubernetes.io/projected/f3c417d3-fdf3-4951-8246-dcd24d7e3820-kube-api-access-vq67h\") pod \"test-model-2bbcb5-kserve-9597db49f-g6cdn\" (UID: \"f3c417d3-fdf3-4951-8246-dcd24d7e3820\") " pod="ai-tenant-e2e-inf-b-2bbcb5/test-model-2bbcb5-kserve-9597db49f-g6cdn"
Nov 02 15:22:03.421434 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:22:03.421406    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f3c417d3-fdf3-4951-8246-dcd24d7e3820-tmp-dir\") pod \"test-model-2bbcb5-kserve-9597db49f-g6cdn\" (UID: \"f3c417d3-fdf3-4951-8246-dcd24d7e3820\") " pod="ai-tenant-e2e-inf-b-2bbcb5/test-model-2bbcb5-kserve-9597db49f-g6cdn"
Nov 02 15:22:03.421492 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:22:03.421443    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f3c417d3-fdf3-4951-8246-dcd24d7e3820-model-cache\") pod \"test-model-2bbcb5-kserve-9597db49f-g6cdn\" (UID: \"f3c417d3-fdf3-4951-8246-dcd24d7e3820\") " pod="ai-tenant-e2e-inf-b-2bbcb5/test-model-2bbcb5-kserve-9597db49f-g6cdn"
Nov 02 15:22:03.421599 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:22:03.415046    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-2bbcb5/test-model-2bbcb5-kserve-9597db49f-g6cdn"]
Nov 02 15:22:03.421697 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:22:03.365486    5304 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-2bbcb5\"/\"openshift-service-ca.crt\""
Nov 02 15:22:03.489446 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:22:03.489356    5304 patch_prober.go:29] interesting pod/e2e-api-a-db75ff-openshift-default-7558cfb4df-qnbxl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body=
Nov 02 15:22:03.504476 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:22:03.504432    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-db75ff-openshift-default-7558cfb4df-qnbxl" podUID="ba7e1e2d-fe7e-41ab-8fb0-814e76dfb08b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused"
Nov 02 15:22:03.547210 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:22:03.536701    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f3c417d3-fdf3-4951-8246-dcd24d7e3820-tls-certs\") pod \"test-model-2bbcb5-kserve-9597db49f-g6cdn\" (UID: \"f3c417d3-fdf3-4951-8246-dcd24d7e3820\") " pod="ai-tenant-e2e-inf-b-2bbcb5/test-model-2bbcb5-kserve-9597db49f-g6cdn"
Nov 02 15:22:03.547210 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:22:03.536836    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f3c417d3-fdf3-4951-8246-dcd24d7e3820-home\") pod \"test-model-2bbcb5-kserve-9597db49f-g6cdn\" (UID: \"f3c417d3-fdf3-4951-8246-dcd24d7e3820\") " pod="ai-tenant-e2e-inf-b-2bbcb5/test-model-2bbcb5-kserve-9597db49f-g6cdn"
Nov 02 15:22:03.547210 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:22:03.536918    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f3c417d3-fdf3-4951-8246-dcd24d7e3820-dshm\") pod \"test-model-2bbcb5-kserve-9597db49f-g6cdn\" (UID: \"f3c417d3-fdf3-4951-8246-dcd24d7e3820\") " pod="ai-tenant-e2e-inf-b-2bbcb5/test-model-2bbcb5-kserve-9597db49f-g6cdn"
Nov 02 15:22:03.547210 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:22:03.536983    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vq67h\" (UniqueName: \"kubernetes.io/projected/f3c417d3-fdf3-4951-8246-dcd24d7e3820-kube-api-access-vq67h\") pod \"test-model-2bbcb5-kserve-9597db49f-g6cdn\" (UID: \"f3c417d3-fdf3-4951-8246-dcd24d7e3820\") " pod="ai-tenant-e2e-inf-b-2bbcb5/test-model-2bbcb5-kserve-9597db49f-g6cdn"
Nov 02 15:22:03.547210 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:22:03.537045    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f3c417d3-fdf3-4951-8246-dcd24d7e3820-tmp-dir\") pod \"test-model-2bbcb5-kserve-9597db49f-g6cdn\" (UID: \"f3c417d3-fdf3-4951-8246-dcd24d7e3820\") " pod="ai-tenant-e2e-inf-b-2bbcb5/test-model-2bbcb5-kserve-9597db49f-g6cdn"
Nov 02 15:22:03.547210 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:22:03.537075    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f3c417d3-fdf3-4951-8246-dcd24d7e3820-model-cache\") pod \"test-model-2bbcb5-kserve-9597db49f-g6cdn\" (UID: \"f3c417d3-fdf3-4951-8246-dcd24d7e3820\") " pod="ai-tenant-e2e-inf-b-2bbcb5/test-model-2bbcb5-kserve-9597db49f-g6cdn"
Nov 02 15:22:03.547210 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:22:03.542000    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f3c417d3-fdf3-4951-8246-dcd24d7e3820-model-cache\") pod \"test-model-2bbcb5-kserve-9597db49f-g6cdn\" (UID: \"f3c417d3-fdf3-4951-8246-dcd24d7e3820\") " pod="ai-tenant-e2e-inf-b-2bbcb5/test-model-2bbcb5-kserve-9597db49f-g6cdn"
Nov 02 15:22:03.587152 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:22:03.570741    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f3c417d3-fdf3-4951-8246-dcd24d7e3820-home\") pod \"test-model-2bbcb5-kserve-9597db49f-g6cdn\" (UID: \"f3c417d3-fdf3-4951-8246-dcd24d7e3820\") " pod="ai-tenant-e2e-inf-b-2bbcb5/test-model-2bbcb5-kserve-9597db49f-g6cdn"
Nov 02 15:22:03.596417 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:22:03.580952    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f3c417d3-fdf3-4951-8246-dcd24d7e3820-tmp-dir\") pod \"test-model-2bbcb5-kserve-9597db49f-g6cdn\" (UID: \"f3c417d3-fdf3-4951-8246-dcd24d7e3820\") " pod="ai-tenant-e2e-inf-b-2bbcb5/test-model-2bbcb5-kserve-9597db49f-g6cdn"
Nov 02 15:22:03.697808 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:22:03.662835    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f3c417d3-fdf3-4951-8246-dcd24d7e3820-tls-certs\") pod \"test-model-2bbcb5-kserve-9597db49f-g6cdn\" (UID: \"f3c417d3-fdf3-4951-8246-dcd24d7e3820\") " pod="ai-tenant-e2e-inf-b-2bbcb5/test-model-2bbcb5-kserve-9597db49f-g6cdn"
Nov 02 15:22:03.713045 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:22:03.709038    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vq67h\" (UniqueName: \"kubernetes.io/projected/f3c417d3-fdf3-4951-8246-dcd24d7e3820-kube-api-access-vq67h\") pod \"test-model-2bbcb5-kserve-9597db49f-g6cdn\" (UID: \"f3c417d3-fdf3-4951-8246-dcd24d7e3820\") " pod="ai-tenant-e2e-inf-b-2bbcb5/test-model-2bbcb5-kserve-9597db49f-g6cdn"
Nov 02 15:22:03.717446 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:22:03.717415    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f3c417d3-fdf3-4951-8246-dcd24d7e3820-dshm\") pod \"test-model-2bbcb5-kserve-9597db49f-g6cdn\" (UID: \"f3c417d3-fdf3-4951-8246-dcd24d7e3820\") " pod="ai-tenant-e2e-inf-b-2bbcb5/test-model-2bbcb5-kserve-9597db49f-g6cdn"
Nov 02 15:22:03.923393 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:22:03.923359    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-2bbcb5/test-model-2bbcb5-kserve-9597db49f-g6cdn"
Nov 02 15:22:04.459735 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:22:04.459678    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-2bbcb5/test-model-2bbcb5-kserve-9597db49f-g6cdn"]
Nov 02 15:22:04.462990 ip-10-0-135-138 kubenswrapper[5304]: W1102 15:22:04.462964    5304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf3c417d3_fdf3_4951_8246_dcd24d7e3820.slice/crio-28c4ae9555e439eac40a696ca76cfb3de24695bcebb271d3d1fad9372677b756 WatchSource:0}: Error finding container 28c4ae9555e439eac40a696ca76cfb3de24695bcebb271d3d1fad9372677b756: Status 404 returned error can't find the container with id 28c4ae9555e439eac40a696ca76cfb3de24695bcebb271d3d1fad9372677b756
Nov 02 15:22:04.735483 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:22:04.735374    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-2bbcb5/test-model-2bbcb5-kserve-9597db49f-g6cdn" event={"ID":"f3c417d3-fdf3-4951-8246-dcd24d7e3820","Type":"ContainerStarted","Data":"07b8f2c2ee423e3be8d9bc7e255be104ae0ecbf94fcc09a6d097aed75f6237c3"}
Nov 02 15:22:04.735483 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:22:04.735438    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-2bbcb5/test-model-2bbcb5-kserve-9597db49f-g6cdn" event={"ID":"f3c417d3-fdf3-4951-8246-dcd24d7e3820","Type":"ContainerStarted","Data":"28c4ae9555e439eac40a696ca76cfb3de24695bcebb271d3d1fad9372677b756"}
Nov 02 15:22:04.756768 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:22:04.756732    5304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-db75ff-openshift-default-7558cfb4df-qnbxl"
Nov 02 15:22:04.758264 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:22:04.758225    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-db75ff-openshift-default-7558cfb4df-qnbxl"
Nov 02 15:22:04.779116 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:22:04.779059    5304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-2bbcb5/test-model-2bbcb5-kserve-9597db49f-g6cdn" podStartSLOduration=2.779040396 podStartE2EDuration="2.779040396s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:22:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:22:04.776822003 +0000 UTC m=+1524.770646412" watchObservedRunningTime="2026-11-02 15:22:04.779040396 +0000 UTC m=+1524.772864806"
Nov 02 15:22:04.924408 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:22:04.924378    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-2bbcb5/test-model-2bbcb5-kserve-9597db49f-g6cdn"
Nov 02 15:22:05.370626 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:22:05.370598    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-4ec02d-8498bbbfd9-gm4hk"
Nov 02 15:22:10.542374 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:22:10.542336    5304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-21ece2-85dc7bbc8c-59ktr"]
Nov 02 15:22:10.546376 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:22:10.546353    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-21ece2-85dc7bbc8c-59ktr"
Nov 02 15:22:10.567431 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:22:10.567403    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-21ece2-85dc7bbc8c-59ktr"]
Nov 02 15:22:10.574515 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:22:10.574492    5304 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-21ece2\""
Nov 02 15:22:10.574515 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:22:10.574508    5304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-21ece2-dockercfg-tnpbd\""
Nov 02 15:22:10.630620 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:22:10.630588    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fl4x7\" (UniqueName: \"kubernetes.io/projected/13e8c2b9-05fb-44e4-99b7-fd3104272bef-kube-api-access-fl4x7\") pod \"payload-pre-processing-e2e-api-b-21ece2-85dc7bbc8c-59ktr\" (UID: \"13e8c2b9-05fb-44e4-99b7-fd3104272bef\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-21ece2-85dc7bbc8c-59ktr"
Nov 02 15:22:10.630827 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:22:10.630771    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13e8c2b9-05fb-44e4-99b7-fd3104272bef-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-21ece2-85dc7bbc8c-59ktr\" (UID: \"13e8c2b9-05fb-44e4-99b7-fd3104272bef\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-21ece2-85dc7bbc8c-59ktr"
Nov 02 15:22:10.738822 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:22:10.738777    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fl4x7\" (UniqueName: \"kubernetes.io/projected/13e8c2b9-05fb-44e4-99b7-fd3104272bef-kube-api-access-fl4x7\") pod \"payload-pre-processing-e2e-api-b-21ece2-85dc7bbc8c-59ktr\" (UID: \"13e8c2b9-05fb-44e4-99b7-fd3104272bef\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-21ece2-85dc7bbc8c-59ktr"
Nov 02 15:22:10.739034 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:22:10.738999    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13e8c2b9-05fb-44e4-99b7-fd3104272bef-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-21ece2-85dc7bbc8c-59ktr\" (UID: \"13e8c2b9-05fb-44e4-99b7-fd3104272bef\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-21ece2-85dc7bbc8c-59ktr"
Nov 02 15:22:10.740582 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:22:10.740555    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13e8c2b9-05fb-44e4-99b7-fd3104272bef-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-21ece2-85dc7bbc8c-59ktr\" (UID: \"13e8c2b9-05fb-44e4-99b7-fd3104272bef\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-21ece2-85dc7bbc8c-59ktr"
Nov 02 15:22:10.785988 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:22:10.782420    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fl4x7\" (UniqueName: \"kubernetes.io/projected/13e8c2b9-05fb-44e4-99b7-fd3104272bef-kube-api-access-fl4x7\") pod \"payload-pre-processing-e2e-api-b-21ece2-85dc7bbc8c-59ktr\" (UID: \"13e8c2b9-05fb-44e4-99b7-fd3104272bef\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-21ece2-85dc7bbc8c-59ktr"
Nov 02 15:22:10.942912 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:22:10.941619    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-21ece2-85dc7bbc8c-59ktr"
Nov 02 15:22:11.307670 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:22:11.307645    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-21ece2-85dc7bbc8c-59ktr"]
Nov 02 15:22:11.309741 ip-10-0-135-138 kubenswrapper[5304]: W1102 15:22:11.309710    5304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13e8c2b9_05fb_44e4_99b7_fd3104272bef.slice/crio-41898dac39eaeb5538a0248d87c4fbe17aad453eaf0f7484f63a497882ea5cb7 WatchSource:0}: Error finding container 41898dac39eaeb5538a0248d87c4fbe17aad453eaf0f7484f63a497882ea5cb7: Status 404 returned error can't find the container with id 41898dac39eaeb5538a0248d87c4fbe17aad453eaf0f7484f63a497882ea5cb7
Nov 02 15:22:11.593155 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:22:11.593108    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-ldcsg"]
Nov 02 15:22:11.593560 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:22:11.593341    5304 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-ldcsg" podUID="27105db3-fb1c-4da0-8cf0-5f301ac6967f" containerName="istio-proxy" containerID="cri-o://e216b1873ddeba69a9821db8f47b65bc4c0a00632d1182c921c472fa261d9a4a" gracePeriod=30
Nov 02 15:22:11.967282 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:22:11.967202    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-21ece2-85dc7bbc8c-59ktr" event={"ID":"13e8c2b9-05fb-44e4-99b7-fd3104272bef","Type":"ContainerStarted","Data":"949da75c2c36820110111ee72f86d794110509e2f7985044c0667f32d3c89766"}
Nov 02 15:22:11.967282 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:22:11.967252    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-21ece2-85dc7bbc8c-59ktr" event={"ID":"13e8c2b9-05fb-44e4-99b7-fd3104272bef","Type":"ContainerStarted","Data":"41898dac39eaeb5538a0248d87c4fbe17aad453eaf0f7484f63a497882ea5cb7"}
Nov 02 15:22:11.967467 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:22:11.967333    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-21ece2-85dc7bbc8c-59ktr"
Nov 02 15:22:12.009174 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:22:12.009094    5304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-21ece2-85dc7bbc8c-59ktr" podStartSLOduration=2.009079441 podStartE2EDuration="2.009079441s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:22:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:22:12.008974113 +0000 UTC m=+1532.002798522" watchObservedRunningTime="2026-11-02 15:22:12.009079441 +0000 UTC m=+1532.002903850"
Nov 02 15:22:13.944846 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:22:13.944804    5304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-2bbcb5/test-model-2bbcb5-kserve-9597db49f-g6cdn"
Nov 02 15:22:13.961372 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:22:13.961344    5304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-2bbcb5/test-model-2bbcb5-kserve-9597db49f-g6cdn"
Nov 02 15:22:13.998839 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:22:13.998806    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-2bbcb5/test-model-2bbcb5-kserve-9597db49f-g6cdn"
Nov 02 15:22:17.023211 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:22:17.023175    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-ldcsg"
Nov 02 15:22:17.070825 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:22:17.070787    5304 generic.go:356] "Generic (PLEG): container finished" podID="27105db3-fb1c-4da0-8cf0-5f301ac6967f" containerID="e216b1873ddeba69a9821db8f47b65bc4c0a00632d1182c921c472fa261d9a4a" exitCode=0
Nov 02 15:22:17.071004 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:22:17.070881    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-ldcsg"
Nov 02 15:22:17.071004 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:22:17.070905    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-ldcsg" event={"ID":"27105db3-fb1c-4da0-8cf0-5f301ac6967f","Type":"ContainerDied","Data":"e216b1873ddeba69a9821db8f47b65bc4c0a00632d1182c921c472fa261d9a4a"}
Nov 02 15:22:17.071004 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:22:17.070978    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-ldcsg" event={"ID":"27105db3-fb1c-4da0-8cf0-5f301ac6967f","Type":"ContainerDied","Data":"945eddde5e4cf858141f8ed45ad6794ad808f8ca82fa6f9edd55c94c2fb70e67"}
Nov 02 15:22:17.071004 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:22:17.071000    5304 scope.go:122] "RemoveContainer" containerID="e216b1873ddeba69a9821db8f47b65bc4c0a00632d1182c921c472fa261d9a4a"
Nov 02 15:22:17.082900 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:22:17.082871    5304 scope.go:122] "RemoveContainer" containerID="a942baeee15555237950f168b16307153d6ecf18f87153f75833ffd30e255eb1"
Nov 02 15:22:17.093364 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:22:17.093342    5304 scope.go:122] "RemoveContainer" containerID="e216b1873ddeba69a9821db8f47b65bc4c0a00632d1182c921c472fa261d9a4a"
Nov 02 15:22:17.093577 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:22:17.093560    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27105db3-fb1c-4da0-8cf0-5f301ac6967f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27105db3-fb1c-4da0-8cf0-5f301ac6967f-credential-socket\") pod \"27105db3-fb1c-4da0-8cf0-5f301ac6967f\" (UID: \"27105db3-fb1c-4da0-8cf0-5f301ac6967f\") "
Nov 02 15:22:17.093674 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:22:17.093597    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27105db3-fb1c-4da0-8cf0-5f301ac6967f-kube-api-access-6696p\" (UniqueName: \"kubernetes.io/projected/27105db3-fb1c-4da0-8cf0-5f301ac6967f-kube-api-access-6696p\") pod \"27105db3-fb1c-4da0-8cf0-5f301ac6967f\" (UID: \"27105db3-fb1c-4da0-8cf0-5f301ac6967f\") "
Nov 02 15:22:17.093674 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:22:17.093624    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27105db3-fb1c-4da0-8cf0-5f301ac6967f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27105db3-fb1c-4da0-8cf0-5f301ac6967f-workload-socket\") pod \"27105db3-fb1c-4da0-8cf0-5f301ac6967f\" (UID: \"27105db3-fb1c-4da0-8cf0-5f301ac6967f\") "
Nov 02 15:22:17.093674 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:22:17.093628    5304 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e216b1873ddeba69a9821db8f47b65bc4c0a00632d1182c921c472fa261d9a4a\": container with ID starting with e216b1873ddeba69a9821db8f47b65bc4c0a00632d1182c921c472fa261d9a4a not found: ID does not exist" containerID="e216b1873ddeba69a9821db8f47b65bc4c0a00632d1182c921c472fa261d9a4a"
Nov 02 15:22:17.093834 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:22:17.093664    5304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e216b1873ddeba69a9821db8f47b65bc4c0a00632d1182c921c472fa261d9a4a"} err="failed to get container status \"e216b1873ddeba69a9821db8f47b65bc4c0a00632d1182c921c472fa261d9a4a\": rpc error: code = NotFound desc = could not find container \"e216b1873ddeba69a9821db8f47b65bc4c0a00632d1182c921c472fa261d9a4a\": container with ID starting with e216b1873ddeba69a9821db8f47b65bc4c0a00632d1182c921c472fa261d9a4a not found: ID does not exist"
Nov 02 15:22:17.093834 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:22:17.093691    5304 scope.go:122] "RemoveContainer" containerID="a942baeee15555237950f168b16307153d6ecf18f87153f75833ffd30e255eb1"
Nov 02 15:22:17.093834 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:22:17.093797    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/27105db3-fb1c-4da0-8cf0-5f301ac6967f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27105db3-fb1c-4da0-8cf0-5f301ac6967f-istio-podinfo\") pod \"27105db3-fb1c-4da0-8cf0-5f301ac6967f\" (UID: \"27105db3-fb1c-4da0-8cf0-5f301ac6967f\") "
Nov 02 15:22:17.093993 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:22:17.093849    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27105db3-fb1c-4da0-8cf0-5f301ac6967f-credential-socket" pod "27105db3-fb1c-4da0-8cf0-5f301ac6967f" (UID: "27105db3-fb1c-4da0-8cf0-5f301ac6967f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:22:17.093993 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:22:17.093915    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27105db3-fb1c-4da0-8cf0-5f301ac6967f-workload-socket" pod "27105db3-fb1c-4da0-8cf0-5f301ac6967f" (UID: "27105db3-fb1c-4da0-8cf0-5f301ac6967f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:22:17.093993 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:22:17.093961    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27105db3-fb1c-4da0-8cf0-5f301ac6967f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27105db3-fb1c-4da0-8cf0-5f301ac6967f-workload-certs\") pod \"27105db3-fb1c-4da0-8cf0-5f301ac6967f\" (UID: \"27105db3-fb1c-4da0-8cf0-5f301ac6967f\") "
Nov 02 15:22:17.094222 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:22:17.093961    5304 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a942baeee15555237950f168b16307153d6ecf18f87153f75833ffd30e255eb1\": container with ID starting with a942baeee15555237950f168b16307153d6ecf18f87153f75833ffd30e255eb1 not found: ID does not exist" containerID="a942baeee15555237950f168b16307153d6ecf18f87153f75833ffd30e255eb1"
Nov 02 15:22:17.094222 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:22:17.094096    5304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a942baeee15555237950f168b16307153d6ecf18f87153f75833ffd30e255eb1"} err="failed to get container status \"a942baeee15555237950f168b16307153d6ecf18f87153f75833ffd30e255eb1\": rpc error: code = NotFound desc = could not find container \"a942baeee15555237950f168b16307153d6ecf18f87153f75833ffd30e255eb1\": container with ID starting with a942baeee15555237950f168b16307153d6ecf18f87153f75833ffd30e255eb1 not found: ID does not exist"
Nov 02 15:22:17.094222 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:22:17.094150    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27105db3-fb1c-4da0-8cf0-5f301ac6967f-workload-certs" pod "27105db3-fb1c-4da0-8cf0-5f301ac6967f" (UID: "27105db3-fb1c-4da0-8cf0-5f301ac6967f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:22:17.094356 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:22:17.094273    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27105db3-fb1c-4da0-8cf0-5f301ac6967f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27105db3-fb1c-4da0-8cf0-5f301ac6967f-istio-data\") pod \"27105db3-fb1c-4da0-8cf0-5f301ac6967f\" (UID: \"27105db3-fb1c-4da0-8cf0-5f301ac6967f\") "
Nov 02 15:22:17.094356 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:22:17.094329    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/27105db3-fb1c-4da0-8cf0-5f301ac6967f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27105db3-fb1c-4da0-8cf0-5f301ac6967f-istiod-ca-cert\") pod \"27105db3-fb1c-4da0-8cf0-5f301ac6967f\" (UID: \"27105db3-fb1c-4da0-8cf0-5f301ac6967f\") "
Nov 02 15:22:17.094434 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:22:17.094371    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27105db3-fb1c-4da0-8cf0-5f301ac6967f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27105db3-fb1c-4da0-8cf0-5f301ac6967f-istio-envoy\") pod \"27105db3-fb1c-4da0-8cf0-5f301ac6967f\" (UID: \"27105db3-fb1c-4da0-8cf0-5f301ac6967f\") "
Nov 02 15:22:17.094545 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:22:17.094518    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27105db3-fb1c-4da0-8cf0-5f301ac6967f-istio-data" pod "27105db3-fb1c-4da0-8cf0-5f301ac6967f" (UID: "27105db3-fb1c-4da0-8cf0-5f301ac6967f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:22:17.094704 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:22:17.094686    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27105db3-fb1c-4da0-8cf0-5f301ac6967f-istiod-ca-cert" pod "27105db3-fb1c-4da0-8cf0-5f301ac6967f" (UID: "27105db3-fb1c-4da0-8cf0-5f301ac6967f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:22:17.094999 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:22:17.094981    5304 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27105db3-fb1c-4da0-8cf0-5f301ac6967f-istio-data\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:22:17.095067 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:22:17.095002    5304 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27105db3-fb1c-4da0-8cf0-5f301ac6967f-istiod-ca-cert\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:22:17.095067 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:22:17.095012    5304 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27105db3-fb1c-4da0-8cf0-5f301ac6967f-credential-socket\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:22:17.095067 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:22:17.095020    5304 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27105db3-fb1c-4da0-8cf0-5f301ac6967f-workload-socket\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:22:17.095067 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:22:17.095029    5304 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27105db3-fb1c-4da0-8cf0-5f301ac6967f-workload-certs\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:22:17.095972 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:22:17.095942    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/27105db3-fb1c-4da0-8cf0-5f301ac6967f-istio-podinfo" pod "27105db3-fb1c-4da0-8cf0-5f301ac6967f" (UID: "27105db3-fb1c-4da0-8cf0-5f301ac6967f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 15:22:17.096075 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:22:17.095996    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27105db3-fb1c-4da0-8cf0-5f301ac6967f-kube-api-access-6696p" pod "27105db3-fb1c-4da0-8cf0-5f301ac6967f" (UID: "27105db3-fb1c-4da0-8cf0-5f301ac6967f"). InnerVolumeSpecName "kube-api-access-6696p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:22:17.096691 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:22:17.096670    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27105db3-fb1c-4da0-8cf0-5f301ac6967f-istio-envoy" pod "27105db3-fb1c-4da0-8cf0-5f301ac6967f" (UID: "27105db3-fb1c-4da0-8cf0-5f301ac6967f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:22:17.195897 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:22:17.195862    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27105db3-fb1c-4da0-8cf0-5f301ac6967f-istio-token\" (UniqueName: \"kubernetes.io/projected/27105db3-fb1c-4da0-8cf0-5f301ac6967f-istio-token\") pod \"27105db3-fb1c-4da0-8cf0-5f301ac6967f\" (UID: \"27105db3-fb1c-4da0-8cf0-5f301ac6967f\") "
Nov 02 15:22:17.196754 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:22:17.196728    5304 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27105db3-fb1c-4da0-8cf0-5f301ac6967f-istio-envoy\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:22:17.196907 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:22:17.196895    5304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6696p\" (UniqueName: \"kubernetes.io/projected/27105db3-fb1c-4da0-8cf0-5f301ac6967f-kube-api-access-6696p\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:22:17.196959 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:22:17.196913    5304 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27105db3-fb1c-4da0-8cf0-5f301ac6967f-istio-podinfo\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:22:17.198001 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:22:17.197980    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27105db3-fb1c-4da0-8cf0-5f301ac6967f-istio-token" pod "27105db3-fb1c-4da0-8cf0-5f301ac6967f" (UID: "27105db3-fb1c-4da0-8cf0-5f301ac6967f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:22:17.298918 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:22:17.298836    5304 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/27105db3-fb1c-4da0-8cf0-5f301ac6967f-istio-token\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:22:17.423491 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:22:17.423442    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-ldcsg"]
Nov 02 15:22:17.456249 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:22:17.456148    5304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-ldcsg"]
Nov 02 15:22:18.121653 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:22:18.121619    5304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="27105db3-fb1c-4da0-8cf0-5f301ac6967f" path="/var/lib/kubelet/pods/27105db3-fb1c-4da0-8cf0-5f301ac6967f/volumes"
Nov 02 15:22:22.981146 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:22:22.980534    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-21ece2-85dc7bbc8c-59ktr"
Nov 02 15:22:24.802218 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:22:24.802175    5304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k6fjk" containerName="extractor"
Nov 02 15:22:26.169155 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:22:26.169049    5304 generic.go:356] "Generic (PLEG): container finished" podID="0ecb8e24-cbc7-4808-a2aa-6536d94590b5" containerID="efc40a69c6cce5f130d98deb4f6cd2f5ba2b24a74d3f5b0c559a0bf990d84bd0" exitCode=0
Nov 02 15:22:26.169637 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:22:26.169158    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-815746-openshift-default-74696d484f-9f99s" event={"ID":"0ecb8e24-cbc7-4808-a2aa-6536d94590b5","Type":"ContainerDied","Data":"efc40a69c6cce5f130d98deb4f6cd2f5ba2b24a74d3f5b0c559a0bf990d84bd0"}
Nov 02 15:22:26.169769 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:22:26.169700    5304 scope.go:122] "RemoveContainer" containerID="efc40a69c6cce5f130d98deb4f6cd2f5ba2b24a74d3f5b0c559a0bf990d84bd0"
Nov 02 15:22:26.171982 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:22:26.171948    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:22:26.172080 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:22:26.172027    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:22:26.172080 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:22:26.172060    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:22:26.872650 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:22:26.872613    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-mp9v6"]
Nov 02 15:22:26.872856 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:22:26.872832    5304 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-mp9v6" podUID="1786cd64-e217-4f69-bcb6-6bda8aace53d" containerName="istio-proxy" containerID="cri-o://ffc66fd9acc334a377664485bc76bcce9222da324488bcf59f177a31a186afd9" gracePeriod=30
Nov 02 15:22:27.179117 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:22:27.179033    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-815746-openshift-default-74696d484f-9f99s" event={"ID":"0ecb8e24-cbc7-4808-a2aa-6536d94590b5","Type":"ContainerStarted","Data":"25bce27ef3f8bd79bed947b2260a48f0b045cfa9200c871de571e2f8e31470f9"}
Nov 02 15:22:27.338842 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:22:27.337292    5304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-815746-openshift-default-74696d484f-9f99s"
Nov 02 15:22:27.338842 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:22:27.337552    5304 patch_prober.go:29] interesting pod/e2e-api-b-815746-openshift-default-74696d484f-9f99s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" start-of-body=
Nov 02 15:22:27.338842 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:22:27.337621    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-815746-openshift-default-74696d484f-9f99s" podUID="0ecb8e24-cbc7-4808-a2aa-6536d94590b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused"
Nov 02 15:22:28.355210 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:22:28.352090    5304 patch_prober.go:29] interesting pod/e2e-api-b-815746-openshift-default-74696d484f-9f99s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" start-of-body=
Nov 02 15:22:28.355757 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:22:28.355248    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-815746-openshift-default-74696d484f-9f99s" podUID="0ecb8e24-cbc7-4808-a2aa-6536d94590b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused"
Nov 02 15:22:29.336508 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:22:29.336470    5304 patch_prober.go:29] interesting pod/e2e-api-b-815746-openshift-default-74696d484f-9f99s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" start-of-body=
Nov 02 15:22:29.336783 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:22:29.336755    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-815746-openshift-default-74696d484f-9f99s" podUID="0ecb8e24-cbc7-4808-a2aa-6536d94590b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused"
Nov 02 15:22:30.355712 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:22:30.355670    5304 patch_prober.go:29] interesting pod/e2e-api-b-815746-openshift-default-74696d484f-9f99s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" start-of-body=
Nov 02 15:22:30.356200 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:22:30.355740    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-815746-openshift-default-74696d484f-9f99s" podUID="0ecb8e24-cbc7-4808-a2aa-6536d94590b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused"
Nov 02 15:22:31.355153 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:22:31.350503    5304 patch_prober.go:29] interesting pod/e2e-api-b-815746-openshift-default-74696d484f-9f99s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" start-of-body=
Nov 02 15:22:31.355153 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:22:31.350566    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-815746-openshift-default-74696d484f-9f99s" podUID="0ecb8e24-cbc7-4808-a2aa-6536d94590b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused"
Nov 02 15:22:32.219794 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:22:32.219711    5304 generic.go:356] "Generic (PLEG): container finished" podID="1786cd64-e217-4f69-bcb6-6bda8aace53d" containerID="ffc66fd9acc334a377664485bc76bcce9222da324488bcf59f177a31a186afd9" exitCode=0
Nov 02 15:22:32.219794 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:22:32.219781    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-mp9v6" event={"ID":"1786cd64-e217-4f69-bcb6-6bda8aace53d","Type":"ContainerDied","Data":"ffc66fd9acc334a377664485bc76bcce9222da324488bcf59f177a31a186afd9"}
Nov 02 15:22:32.306839 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:22:32.306469    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-mp9v6"
Nov 02 15:22:32.328562 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:22:32.328531    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-815746-openshift-default-74696d484f-9f99s"
Nov 02 15:22:32.356027 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:22:32.355991    5304 patch_prober.go:29] interesting pod/e2e-api-b-815746-openshift-default-74696d484f-9f99s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" start-of-body=
Nov 02 15:22:32.356223 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:22:32.356067    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-815746-openshift-default-74696d484f-9f99s" podUID="0ecb8e24-cbc7-4808-a2aa-6536d94590b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused"
Nov 02 15:22:32.447177 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:22:32.443734    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1786cd64-e217-4f69-bcb6-6bda8aace53d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1786cd64-e217-4f69-bcb6-6bda8aace53d-istio-data\") pod \"1786cd64-e217-4f69-bcb6-6bda8aace53d\" (UID: \"1786cd64-e217-4f69-bcb6-6bda8aace53d\") "
Nov 02 15:22:32.447177 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:22:32.443818    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1786cd64-e217-4f69-bcb6-6bda8aace53d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1786cd64-e217-4f69-bcb6-6bda8aace53d-workload-socket\") pod \"1786cd64-e217-4f69-bcb6-6bda8aace53d\" (UID: \"1786cd64-e217-4f69-bcb6-6bda8aace53d\") "
Nov 02 15:22:32.447177 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:22:32.444001    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1786cd64-e217-4f69-bcb6-6bda8aace53d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1786cd64-e217-4f69-bcb6-6bda8aace53d-credential-socket\") pod \"1786cd64-e217-4f69-bcb6-6bda8aace53d\" (UID: \"1786cd64-e217-4f69-bcb6-6bda8aace53d\") "
Nov 02 15:22:32.447177 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:22:32.444056    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1786cd64-e217-4f69-bcb6-6bda8aace53d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1786cd64-e217-4f69-bcb6-6bda8aace53d-istio-envoy\") pod \"1786cd64-e217-4f69-bcb6-6bda8aace53d\" (UID: \"1786cd64-e217-4f69-bcb6-6bda8aace53d\") "
Nov 02 15:22:32.447177 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:22:32.444114    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1786cd64-e217-4f69-bcb6-6bda8aace53d-istio-token\" (UniqueName: \"kubernetes.io/projected/1786cd64-e217-4f69-bcb6-6bda8aace53d-istio-token\") pod \"1786cd64-e217-4f69-bcb6-6bda8aace53d\" (UID: \"1786cd64-e217-4f69-bcb6-6bda8aace53d\") "
Nov 02 15:22:32.447177 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:22:32.444189    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1786cd64-e217-4f69-bcb6-6bda8aace53d-kube-api-access-gtsq5\" (UniqueName: \"kubernetes.io/projected/1786cd64-e217-4f69-bcb6-6bda8aace53d-kube-api-access-gtsq5\") pod \"1786cd64-e217-4f69-bcb6-6bda8aace53d\" (UID: \"1786cd64-e217-4f69-bcb6-6bda8aace53d\") "
Nov 02 15:22:32.447177 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:22:32.444336    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1786cd64-e217-4f69-bcb6-6bda8aace53d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1786cd64-e217-4f69-bcb6-6bda8aace53d-istiod-ca-cert\") pod \"1786cd64-e217-4f69-bcb6-6bda8aace53d\" (UID: \"1786cd64-e217-4f69-bcb6-6bda8aace53d\") "
Nov 02 15:22:32.447177 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:22:32.444399    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1786cd64-e217-4f69-bcb6-6bda8aace53d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1786cd64-e217-4f69-bcb6-6bda8aace53d-istio-podinfo\") pod \"1786cd64-e217-4f69-bcb6-6bda8aace53d\" (UID: \"1786cd64-e217-4f69-bcb6-6bda8aace53d\") "
Nov 02 15:22:32.447177 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:22:32.444432    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1786cd64-e217-4f69-bcb6-6bda8aace53d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1786cd64-e217-4f69-bcb6-6bda8aace53d-workload-certs\") pod \"1786cd64-e217-4f69-bcb6-6bda8aace53d\" (UID: \"1786cd64-e217-4f69-bcb6-6bda8aace53d\") "
Nov 02 15:22:32.457398 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:22:32.457351    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1786cd64-e217-4f69-bcb6-6bda8aace53d-istiod-ca-cert" pod "1786cd64-e217-4f69-bcb6-6bda8aace53d" (UID: "1786cd64-e217-4f69-bcb6-6bda8aace53d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:22:32.457398 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:22:32.457364    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1786cd64-e217-4f69-bcb6-6bda8aace53d-workload-certs" pod "1786cd64-e217-4f69-bcb6-6bda8aace53d" (UID: "1786cd64-e217-4f69-bcb6-6bda8aace53d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:22:32.457950 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:22:32.457922    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1786cd64-e217-4f69-bcb6-6bda8aace53d-credential-socket" pod "1786cd64-e217-4f69-bcb6-6bda8aace53d" (UID: "1786cd64-e217-4f69-bcb6-6bda8aace53d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:22:32.458060 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:22:32.458008    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1786cd64-e217-4f69-bcb6-6bda8aace53d-istio-data" pod "1786cd64-e217-4f69-bcb6-6bda8aace53d" (UID: "1786cd64-e217-4f69-bcb6-6bda8aace53d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:22:32.458225 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:22:32.458185    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1786cd64-e217-4f69-bcb6-6bda8aace53d-workload-socket" pod "1786cd64-e217-4f69-bcb6-6bda8aace53d" (UID: "1786cd64-e217-4f69-bcb6-6bda8aace53d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:22:32.459959 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:22:32.459933    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1786cd64-e217-4f69-bcb6-6bda8aace53d-istio-podinfo" pod "1786cd64-e217-4f69-bcb6-6bda8aace53d" (UID: "1786cd64-e217-4f69-bcb6-6bda8aace53d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 15:22:32.460091 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:22:32.459981    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1786cd64-e217-4f69-bcb6-6bda8aace53d-kube-api-access-gtsq5" pod "1786cd64-e217-4f69-bcb6-6bda8aace53d" (UID: "1786cd64-e217-4f69-bcb6-6bda8aace53d"). InnerVolumeSpecName "kube-api-access-gtsq5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:22:32.461460 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:22:32.461438    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1786cd64-e217-4f69-bcb6-6bda8aace53d-istio-envoy" pod "1786cd64-e217-4f69-bcb6-6bda8aace53d" (UID: "1786cd64-e217-4f69-bcb6-6bda8aace53d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:22:32.461839 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:22:32.461813    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1786cd64-e217-4f69-bcb6-6bda8aace53d-istio-token" pod "1786cd64-e217-4f69-bcb6-6bda8aace53d" (UID: "1786cd64-e217-4f69-bcb6-6bda8aace53d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:22:32.549850 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:22:32.549673    5304 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1786cd64-e217-4f69-bcb6-6bda8aace53d-istiod-ca-cert\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:22:32.549850 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:22:32.549712    5304 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1786cd64-e217-4f69-bcb6-6bda8aace53d-istio-podinfo\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:22:32.549850 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:22:32.549728    5304 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1786cd64-e217-4f69-bcb6-6bda8aace53d-workload-certs\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:22:32.549850 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:22:32.549743    5304 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1786cd64-e217-4f69-bcb6-6bda8aace53d-istio-data\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:22:32.549850 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:22:32.549756    5304 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1786cd64-e217-4f69-bcb6-6bda8aace53d-workload-socket\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:22:32.549850 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:22:32.549770    5304 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1786cd64-e217-4f69-bcb6-6bda8aace53d-credential-socket\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:22:32.549850 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:22:32.549783    5304 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1786cd64-e217-4f69-bcb6-6bda8aace53d-istio-envoy\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:22:32.549850 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:22:32.549796    5304 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1786cd64-e217-4f69-bcb6-6bda8aace53d-istio-token\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:22:32.549850 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:22:32.549811    5304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gtsq5\" (UniqueName: \"kubernetes.io/projected/1786cd64-e217-4f69-bcb6-6bda8aace53d-kube-api-access-gtsq5\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:22:33.229289 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:22:33.229256    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-mp9v6"
Nov 02 15:22:33.229698 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:22:33.229260    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-mp9v6" event={"ID":"1786cd64-e217-4f69-bcb6-6bda8aace53d","Type":"ContainerDied","Data":"760e0529e852ab648e707ddfe2337490b33951940c378960d86b986f778b7825"}
Nov 02 15:22:33.229698 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:22:33.229387    5304 scope.go:122] "RemoveContainer" containerID="ffc66fd9acc334a377664485bc76bcce9222da324488bcf59f177a31a186afd9"
Nov 02 15:22:33.299511 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:22:33.299485    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-mp9v6"]
Nov 02 15:22:33.311049 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:22:33.311022    5304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-mp9v6"]
Nov 02 15:22:33.338732 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:22:33.338285    5304 patch_prober.go:29] interesting pod/e2e-api-b-815746-openshift-default-74696d484f-9f99s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" start-of-body=
Nov 02 15:22:33.338732 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:22:33.338351    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-815746-openshift-default-74696d484f-9f99s" podUID="0ecb8e24-cbc7-4808-a2aa-6536d94590b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused"
Nov 02 15:22:34.238150 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:22:34.235098    5304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1786cd64-e217-4f69-bcb6-6bda8aace53d" path="/var/lib/kubelet/pods/1786cd64-e217-4f69-bcb6-6bda8aace53d/volumes"
Nov 02 15:22:34.345448 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:22:34.345413    5304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-815746-openshift-default-74696d484f-9f99s"
Nov 02 15:22:34.350232 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:22:34.350207    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-815746-openshift-default-74696d484f-9f99s"
Nov 02 15:22:45.701456 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:22:45.701421    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-3d7df16e/e2e-lifecycle-model-3d7df16e-kserve-6c967d4fb5-sh4rx"]
Nov 02 15:22:54.800142 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:22:54.800090    5304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k6fjk" containerName="extractor"
Nov 02 15:23:00.742265 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:00.742209    5304 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-mt-3d7df16e/e2e-lifecycle-model-3d7df16e-kserve-6c967d4fb5-sh4rx" podUID="a786ec8c-9093-4152-8e99-4d334bd5928a" containerName="main" containerID="cri-o://78f3c45e5202b15d51ab630f29cf5d7aed4054651814d316a18a63dcb4137410" gracePeriod=45
Nov 02 15:23:00.848365 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:00.848319    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-3d7df16e/e2e-lifecycle-model-3d7df16e-kserve-6c967d4fb5-sh4rx" podUID="a786ec8c-9093-4152-8e99-4d334bd5928a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.91:8000/ready\": dial tcp 10.133.0.91:8000: connect: connection refused"
Nov 02 15:23:00.890877 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:00.890845    5304 generic.go:356] "Generic (PLEG): container finished" podID="a786ec8c-9093-4152-8e99-4d334bd5928a" containerID="78f3c45e5202b15d51ab630f29cf5d7aed4054651814d316a18a63dcb4137410" exitCode=0
Nov 02 15:23:00.891039 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:00.890920    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-3d7df16e/e2e-lifecycle-model-3d7df16e-kserve-6c967d4fb5-sh4rx" event={"ID":"a786ec8c-9093-4152-8e99-4d334bd5928a","Type":"ContainerDied","Data":"78f3c45e5202b15d51ab630f29cf5d7aed4054651814d316a18a63dcb4137410"}
Nov 02 15:23:01.030263 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:01.030236    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-3d7df16e/e2e-lifecycle-model-3d7df16e-kserve-6c967d4fb5-sh4rx"
Nov 02 15:23:01.171010 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:01.170974    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a786ec8c-9093-4152-8e99-4d334bd5928a-tls-certs\" (UniqueName: \"kubernetes.io/secret/a786ec8c-9093-4152-8e99-4d334bd5928a-tls-certs\") pod \"a786ec8c-9093-4152-8e99-4d334bd5928a\" (UID: \"a786ec8c-9093-4152-8e99-4d334bd5928a\") "
Nov 02 15:23:01.171242 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:01.171077    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a786ec8c-9093-4152-8e99-4d334bd5928a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/a786ec8c-9093-4152-8e99-4d334bd5928a-dshm\") pod \"a786ec8c-9093-4152-8e99-4d334bd5928a\" (UID: \"a786ec8c-9093-4152-8e99-4d334bd5928a\") "
Nov 02 15:23:01.171242 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:01.171178    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a786ec8c-9093-4152-8e99-4d334bd5928a-home\" (UniqueName: \"kubernetes.io/empty-dir/a786ec8c-9093-4152-8e99-4d334bd5928a-home\") pod \"a786ec8c-9093-4152-8e99-4d334bd5928a\" (UID: \"a786ec8c-9093-4152-8e99-4d334bd5928a\") "
Nov 02 15:23:01.171370 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:01.171354    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a786ec8c-9093-4152-8e99-4d334bd5928a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a786ec8c-9093-4152-8e99-4d334bd5928a-model-cache\") pod \"a786ec8c-9093-4152-8e99-4d334bd5928a\" (UID: \"a786ec8c-9093-4152-8e99-4d334bd5928a\") "
Nov 02 15:23:01.171432 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:01.171406    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a786ec8c-9093-4152-8e99-4d334bd5928a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a786ec8c-9093-4152-8e99-4d334bd5928a-tmp-dir\") pod \"a786ec8c-9093-4152-8e99-4d334bd5928a\" (UID: \"a786ec8c-9093-4152-8e99-4d334bd5928a\") "
Nov 02 15:23:01.171486 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:01.171429    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a786ec8c-9093-4152-8e99-4d334bd5928a-home" pod "a786ec8c-9093-4152-8e99-4d334bd5928a" (UID: "a786ec8c-9093-4152-8e99-4d334bd5928a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:23:01.171486 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:01.171444    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a786ec8c-9093-4152-8e99-4d334bd5928a-kube-api-access-6ll2c\" (UniqueName: \"kubernetes.io/projected/a786ec8c-9093-4152-8e99-4d334bd5928a-kube-api-access-6ll2c\") pod \"a786ec8c-9093-4152-8e99-4d334bd5928a\" (UID: \"a786ec8c-9093-4152-8e99-4d334bd5928a\") "
Nov 02 15:23:01.171651 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:01.171621    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a786ec8c-9093-4152-8e99-4d334bd5928a-model-cache" pod "a786ec8c-9093-4152-8e99-4d334bd5928a" (UID: "a786ec8c-9093-4152-8e99-4d334bd5928a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:23:01.171651 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:01.171635    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a786ec8c-9093-4152-8e99-4d334bd5928a-tmp-dir" pod "a786ec8c-9093-4152-8e99-4d334bd5928a" (UID: "a786ec8c-9093-4152-8e99-4d334bd5928a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:23:01.172187 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:01.172168    5304 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a786ec8c-9093-4152-8e99-4d334bd5928a-model-cache\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:23:01.172289 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:01.172190    5304 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a786ec8c-9093-4152-8e99-4d334bd5928a-tmp-dir\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:23:01.172289 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:01.172202    5304 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a786ec8c-9093-4152-8e99-4d334bd5928a-home\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:23:01.173721 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:01.173700    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a786ec8c-9093-4152-8e99-4d334bd5928a-kube-api-access-6ll2c" pod "a786ec8c-9093-4152-8e99-4d334bd5928a" (UID: "a786ec8c-9093-4152-8e99-4d334bd5928a"). InnerVolumeSpecName "kube-api-access-6ll2c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:23:01.173786 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:01.173750    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a786ec8c-9093-4152-8e99-4d334bd5928a-tls-certs" pod "a786ec8c-9093-4152-8e99-4d334bd5928a" (UID: "a786ec8c-9093-4152-8e99-4d334bd5928a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:23:01.173786 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:01.173761    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a786ec8c-9093-4152-8e99-4d334bd5928a-dshm" pod "a786ec8c-9093-4152-8e99-4d334bd5928a" (UID: "a786ec8c-9093-4152-8e99-4d334bd5928a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:23:01.273985 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:01.273887    5304 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a786ec8c-9093-4152-8e99-4d334bd5928a-tls-certs\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:23:01.273985 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:01.273920    5304 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a786ec8c-9093-4152-8e99-4d334bd5928a-dshm\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:23:01.273985 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:01.273935    5304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6ll2c\" (UniqueName: \"kubernetes.io/projected/a786ec8c-9093-4152-8e99-4d334bd5928a-kube-api-access-6ll2c\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:23:02.188171 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:02.185666    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-3d7df16e/e2e-lifecycle-model-3d7df16e-kserve-6c967d4fb5-sh4rx"
Nov 02 15:23:02.207432 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:02.207391    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-3d7df16e/e2e-lifecycle-model-3d7df16e-kserve-6c967d4fb5-sh4rx" event={"ID":"a786ec8c-9093-4152-8e99-4d334bd5928a","Type":"ContainerDied","Data":"e487a98b8131c7054fc7afdc912474415723b0f155c121d4b7ba8858a2d24166"}
Nov 02 15:23:02.207605 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:02.207436    5304 scope.go:122] "RemoveContainer" containerID="78f3c45e5202b15d51ab630f29cf5d7aed4054651814d316a18a63dcb4137410"
Nov 02 15:23:02.302827 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:02.302793    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-3d7df16e/e2e-lifecycle-model-3d7df16e-kserve-6c967d4fb5-sh4rx"]
Nov 02 15:23:02.313911 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:02.313879    5304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-3d7df16e/e2e-lifecycle-model-3d7df16e-kserve-6c967d4fb5-sh4rx"]
Nov 02 15:23:04.121140 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:04.121092    5304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a786ec8c-9093-4152-8e99-4d334bd5928a" path="/var/lib/kubelet/pods/a786ec8c-9093-4152-8e99-4d334bd5928a/volumes"
Nov 02 15:23:22.222107 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:22.222078    5304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-2bbcb5-openshift-default-788c5c5b65-bxbrl"]
Nov 02 15:23:22.222929 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:22.222904    5304 cpu_manager.go:414] "containerMap: removing container" podUID="27105db3-fb1c-4da0-8cf0-5f301ac6967f" containerName="istio-proxy"
Nov 02 15:23:22.222929 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:22.222923    5304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="27105db3-fb1c-4da0-8cf0-5f301ac6967f" containerName="istio-proxy"
Nov 02 15:23:22.223074 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:22.222958    5304 cpu_manager.go:414] "containerMap: removing container" podUID="a786ec8c-9093-4152-8e99-4d334bd5928a" containerName="main"
Nov 02 15:23:22.223074 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:22.222964    5304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a786ec8c-9093-4152-8e99-4d334bd5928a" containerName="main"
Nov 02 15:23:22.223074 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:22.222980    5304 cpu_manager.go:414] "containerMap: removing container" podUID="1786cd64-e217-4f69-bcb6-6bda8aace53d" containerName="istio-proxy"
Nov 02 15:23:22.223074 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:22.222985    5304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1786cd64-e217-4f69-bcb6-6bda8aace53d" containerName="istio-proxy"
Nov 02 15:23:22.223331 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:22.223149    5304 memory_manager.go:365] "RemoveStaleState removing state" podUID="1786cd64-e217-4f69-bcb6-6bda8aace53d" containerName="istio-proxy"
Nov 02 15:23:22.223331 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:22.223165    5304 memory_manager.go:365] "RemoveStaleState removing state" podUID="27105db3-fb1c-4da0-8cf0-5f301ac6967f" containerName="istio-proxy"
Nov 02 15:23:22.223331 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:22.223182    5304 memory_manager.go:365] "RemoveStaleState removing state" podUID="27105db3-fb1c-4da0-8cf0-5f301ac6967f" containerName="istio-proxy"
Nov 02 15:23:22.223331 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:22.223198    5304 memory_manager.go:365] "RemoveStaleState removing state" podUID="a786ec8c-9093-4152-8e99-4d334bd5928a" containerName="main"
Nov 02 15:23:22.228793 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:22.228765    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-2bbcb5-openshift-default-788c5c5b65-bxbrl"
Nov 02 15:23:22.249894 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:22.249868    5304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-2bbcb5-openshift-default-dockercfg-m5gt8\""
Nov 02 15:23:22.266982 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:22.266954    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-2bbcb5-openshift-default-788c5c5b65-bxbrl"]
Nov 02 15:23:22.416695 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:22.416657    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/28cababb-e85d-48e8-b8a3-69e0914862aa-workload-socket\") pod \"e2e-inf-b-2bbcb5-openshift-default-788c5c5b65-bxbrl\" (UID: \"28cababb-e85d-48e8-b8a3-69e0914862aa\") " pod="openshift-ingress/e2e-inf-b-2bbcb5-openshift-default-788c5c5b65-bxbrl"
Nov 02 15:23:22.416891 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:22.416745    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/28cababb-e85d-48e8-b8a3-69e0914862aa-istiod-ca-cert\") pod \"e2e-inf-b-2bbcb5-openshift-default-788c5c5b65-bxbrl\" (UID: \"28cababb-e85d-48e8-b8a3-69e0914862aa\") " pod="openshift-ingress/e2e-inf-b-2bbcb5-openshift-default-788c5c5b65-bxbrl"
Nov 02 15:23:22.416891 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:22.416810    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/28cababb-e85d-48e8-b8a3-69e0914862aa-workload-certs\") pod \"e2e-inf-b-2bbcb5-openshift-default-788c5c5b65-bxbrl\" (UID: \"28cababb-e85d-48e8-b8a3-69e0914862aa\") " pod="openshift-ingress/e2e-inf-b-2bbcb5-openshift-default-788c5c5b65-bxbrl"
Nov 02 15:23:22.417021 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:22.416886    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/28cababb-e85d-48e8-b8a3-69e0914862aa-istio-token\") pod \"e2e-inf-b-2bbcb5-openshift-default-788c5c5b65-bxbrl\" (UID: \"28cababb-e85d-48e8-b8a3-69e0914862aa\") " pod="openshift-ingress/e2e-inf-b-2bbcb5-openshift-default-788c5c5b65-bxbrl"
Nov 02 15:23:22.417084 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:22.417038    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/28cababb-e85d-48e8-b8a3-69e0914862aa-credential-socket\") pod \"e2e-inf-b-2bbcb5-openshift-default-788c5c5b65-bxbrl\" (UID: \"28cababb-e85d-48e8-b8a3-69e0914862aa\") " pod="openshift-ingress/e2e-inf-b-2bbcb5-openshift-default-788c5c5b65-bxbrl"
Nov 02 15:23:22.417159 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:22.417104    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/28cababb-e85d-48e8-b8a3-69e0914862aa-istio-podinfo\") pod \"e2e-inf-b-2bbcb5-openshift-default-788c5c5b65-bxbrl\" (UID: \"28cababb-e85d-48e8-b8a3-69e0914862aa\") " pod="openshift-ingress/e2e-inf-b-2bbcb5-openshift-default-788c5c5b65-bxbrl"
Nov 02 15:23:22.417253 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:22.417238    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/28cababb-e85d-48e8-b8a3-69e0914862aa-istio-data\") pod \"e2e-inf-b-2bbcb5-openshift-default-788c5c5b65-bxbrl\" (UID: \"28cababb-e85d-48e8-b8a3-69e0914862aa\") " pod="openshift-ingress/e2e-inf-b-2bbcb5-openshift-default-788c5c5b65-bxbrl"
Nov 02 15:23:22.417315 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:22.417292    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/28cababb-e85d-48e8-b8a3-69e0914862aa-istio-envoy\") pod \"e2e-inf-b-2bbcb5-openshift-default-788c5c5b65-bxbrl\" (UID: \"28cababb-e85d-48e8-b8a3-69e0914862aa\") " pod="openshift-ingress/e2e-inf-b-2bbcb5-openshift-default-788c5c5b65-bxbrl"
Nov 02 15:23:22.417424 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:22.417406    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8zckp\" (UniqueName: \"kubernetes.io/projected/28cababb-e85d-48e8-b8a3-69e0914862aa-kube-api-access-8zckp\") pod \"e2e-inf-b-2bbcb5-openshift-default-788c5c5b65-bxbrl\" (UID: \"28cababb-e85d-48e8-b8a3-69e0914862aa\") " pod="openshift-ingress/e2e-inf-b-2bbcb5-openshift-default-788c5c5b65-bxbrl"
Nov 02 15:23:22.523789 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:22.522223    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/28cababb-e85d-48e8-b8a3-69e0914862aa-istio-data\") pod \"e2e-inf-b-2bbcb5-openshift-default-788c5c5b65-bxbrl\" (UID: \"28cababb-e85d-48e8-b8a3-69e0914862aa\") " pod="openshift-ingress/e2e-inf-b-2bbcb5-openshift-default-788c5c5b65-bxbrl"
Nov 02 15:23:22.523789 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:22.522301    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/28cababb-e85d-48e8-b8a3-69e0914862aa-istio-envoy\") pod \"e2e-inf-b-2bbcb5-openshift-default-788c5c5b65-bxbrl\" (UID: \"28cababb-e85d-48e8-b8a3-69e0914862aa\") " pod="openshift-ingress/e2e-inf-b-2bbcb5-openshift-default-788c5c5b65-bxbrl"
Nov 02 15:23:22.523789 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:22.522369    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8zckp\" (UniqueName: \"kubernetes.io/projected/28cababb-e85d-48e8-b8a3-69e0914862aa-kube-api-access-8zckp\") pod \"e2e-inf-b-2bbcb5-openshift-default-788c5c5b65-bxbrl\" (UID: \"28cababb-e85d-48e8-b8a3-69e0914862aa\") " pod="openshift-ingress/e2e-inf-b-2bbcb5-openshift-default-788c5c5b65-bxbrl"
Nov 02 15:23:22.523789 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:22.522486    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/28cababb-e85d-48e8-b8a3-69e0914862aa-workload-socket\") pod \"e2e-inf-b-2bbcb5-openshift-default-788c5c5b65-bxbrl\" (UID: \"28cababb-e85d-48e8-b8a3-69e0914862aa\") " pod="openshift-ingress/e2e-inf-b-2bbcb5-openshift-default-788c5c5b65-bxbrl"
Nov 02 15:23:22.523789 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:22.522613    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/28cababb-e85d-48e8-b8a3-69e0914862aa-istiod-ca-cert\") pod \"e2e-inf-b-2bbcb5-openshift-default-788c5c5b65-bxbrl\" (UID: \"28cababb-e85d-48e8-b8a3-69e0914862aa\") " pod="openshift-ingress/e2e-inf-b-2bbcb5-openshift-default-788c5c5b65-bxbrl"
Nov 02 15:23:22.523789 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:22.522679    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/28cababb-e85d-48e8-b8a3-69e0914862aa-workload-certs\") pod \"e2e-inf-b-2bbcb5-openshift-default-788c5c5b65-bxbrl\" (UID: \"28cababb-e85d-48e8-b8a3-69e0914862aa\") " pod="openshift-ingress/e2e-inf-b-2bbcb5-openshift-default-788c5c5b65-bxbrl"
Nov 02 15:23:22.523789 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:22.522716    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/28cababb-e85d-48e8-b8a3-69e0914862aa-istio-token\") pod \"e2e-inf-b-2bbcb5-openshift-default-788c5c5b65-bxbrl\" (UID: \"28cababb-e85d-48e8-b8a3-69e0914862aa\") " pod="openshift-ingress/e2e-inf-b-2bbcb5-openshift-default-788c5c5b65-bxbrl"
Nov 02 15:23:22.523789 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:22.522833    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/28cababb-e85d-48e8-b8a3-69e0914862aa-credential-socket\") pod \"e2e-inf-b-2bbcb5-openshift-default-788c5c5b65-bxbrl\" (UID: \"28cababb-e85d-48e8-b8a3-69e0914862aa\") " pod="openshift-ingress/e2e-inf-b-2bbcb5-openshift-default-788c5c5b65-bxbrl"
Nov 02 15:23:22.523789 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:22.522889    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/28cababb-e85d-48e8-b8a3-69e0914862aa-istio-podinfo\") pod \"e2e-inf-b-2bbcb5-openshift-default-788c5c5b65-bxbrl\" (UID: \"28cababb-e85d-48e8-b8a3-69e0914862aa\") " pod="openshift-ingress/e2e-inf-b-2bbcb5-openshift-default-788c5c5b65-bxbrl"
Nov 02 15:23:22.523789 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:22.523384    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/28cababb-e85d-48e8-b8a3-69e0914862aa-workload-certs\") pod \"e2e-inf-b-2bbcb5-openshift-default-788c5c5b65-bxbrl\" (UID: \"28cababb-e85d-48e8-b8a3-69e0914862aa\") " pod="openshift-ingress/e2e-inf-b-2bbcb5-openshift-default-788c5c5b65-bxbrl"
Nov 02 15:23:22.527382 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:22.527355    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/28cababb-e85d-48e8-b8a3-69e0914862aa-istio-podinfo\") pod \"e2e-inf-b-2bbcb5-openshift-default-788c5c5b65-bxbrl\" (UID: \"28cababb-e85d-48e8-b8a3-69e0914862aa\") " pod="openshift-ingress/e2e-inf-b-2bbcb5-openshift-default-788c5c5b65-bxbrl"
Nov 02 15:23:22.529841 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:22.529276    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/28cababb-e85d-48e8-b8a3-69e0914862aa-credential-socket\") pod \"e2e-inf-b-2bbcb5-openshift-default-788c5c5b65-bxbrl\" (UID: \"28cababb-e85d-48e8-b8a3-69e0914862aa\") " pod="openshift-ingress/e2e-inf-b-2bbcb5-openshift-default-788c5c5b65-bxbrl"
Nov 02 15:23:22.529841 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:22.529341    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/28cababb-e85d-48e8-b8a3-69e0914862aa-workload-socket\") pod \"e2e-inf-b-2bbcb5-openshift-default-788c5c5b65-bxbrl\" (UID: \"28cababb-e85d-48e8-b8a3-69e0914862aa\") " pod="openshift-ingress/e2e-inf-b-2bbcb5-openshift-default-788c5c5b65-bxbrl"
Nov 02 15:23:22.529841 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:22.529369    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/28cababb-e85d-48e8-b8a3-69e0914862aa-istio-data\") pod \"e2e-inf-b-2bbcb5-openshift-default-788c5c5b65-bxbrl\" (UID: \"28cababb-e85d-48e8-b8a3-69e0914862aa\") " pod="openshift-ingress/e2e-inf-b-2bbcb5-openshift-default-788c5c5b65-bxbrl"
Nov 02 15:23:22.529841 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:22.529784    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/28cababb-e85d-48e8-b8a3-69e0914862aa-istiod-ca-cert\") pod \"e2e-inf-b-2bbcb5-openshift-default-788c5c5b65-bxbrl\" (UID: \"28cababb-e85d-48e8-b8a3-69e0914862aa\") " pod="openshift-ingress/e2e-inf-b-2bbcb5-openshift-default-788c5c5b65-bxbrl"
Nov 02 15:23:22.530585 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:22.530566    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/28cababb-e85d-48e8-b8a3-69e0914862aa-istio-envoy\") pod \"e2e-inf-b-2bbcb5-openshift-default-788c5c5b65-bxbrl\" (UID: \"28cababb-e85d-48e8-b8a3-69e0914862aa\") " pod="openshift-ingress/e2e-inf-b-2bbcb5-openshift-default-788c5c5b65-bxbrl"
Nov 02 15:23:22.547610 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:22.547577    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/28cababb-e85d-48e8-b8a3-69e0914862aa-istio-token\") pod \"e2e-inf-b-2bbcb5-openshift-default-788c5c5b65-bxbrl\" (UID: \"28cababb-e85d-48e8-b8a3-69e0914862aa\") " pod="openshift-ingress/e2e-inf-b-2bbcb5-openshift-default-788c5c5b65-bxbrl"
Nov 02 15:23:22.556680 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:22.556637    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8zckp\" (UniqueName: \"kubernetes.io/projected/28cababb-e85d-48e8-b8a3-69e0914862aa-kube-api-access-8zckp\") pod \"e2e-inf-b-2bbcb5-openshift-default-788c5c5b65-bxbrl\" (UID: \"28cababb-e85d-48e8-b8a3-69e0914862aa\") " pod="openshift-ingress/e2e-inf-b-2bbcb5-openshift-default-788c5c5b65-bxbrl"
Nov 02 15:23:22.843796 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:22.843760    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-2bbcb5-openshift-default-788c5c5b65-bxbrl"
Nov 02 15:23:23.192849 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:23.192822    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-2bbcb5-openshift-default-788c5c5b65-bxbrl"]
Nov 02 15:23:23.196795 ip-10-0-135-138 kubenswrapper[5304]: W1102 15:23:23.196760    5304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28cababb_e85d_48e8_b8a3_69e0914862aa.slice/crio-e5588a52b346ffffefdaf9314a2762ebebb6d7e491be2f1507b7a34061737c69 WatchSource:0}: Error finding container e5588a52b346ffffefdaf9314a2762ebebb6d7e491be2f1507b7a34061737c69: Status 404 returned error can't find the container with id e5588a52b346ffffefdaf9314a2762ebebb6d7e491be2f1507b7a34061737c69
Nov 02 15:23:23.221149 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:23.200777    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:23:23.221149 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:23.200880    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:23:23.221149 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:23.200927    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:23:23.438943 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:23.438894    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-2bbcb5-openshift-default-788c5c5b65-bxbrl" event={"ID":"28cababb-e85d-48e8-b8a3-69e0914862aa","Type":"ContainerStarted","Data":"fa82afe42a7012cde8fbce10600f1c051b4e2ad5c1fafd8c1f5ba7f39d9503c9"}
Nov 02 15:23:23.439447 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:23.438970    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-2bbcb5-openshift-default-788c5c5b65-bxbrl" event={"ID":"28cababb-e85d-48e8-b8a3-69e0914862aa","Type":"ContainerStarted","Data":"e5588a52b346ffffefdaf9314a2762ebebb6d7e491be2f1507b7a34061737c69"}
Nov 02 15:23:23.492977 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:23.492869    5304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-2bbcb5-openshift-default-788c5c5b65-bxbrl" podStartSLOduration=1.492850733 podStartE2EDuration="1.492850733s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:23:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:23:23.489309747 +0000 UTC m=+1603.483134163" watchObservedRunningTime="2026-11-02 15:23:23.492850733 +0000 UTC m=+1603.486675145"
Nov 02 15:23:23.844510 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:23.844476    5304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-2bbcb5-openshift-default-788c5c5b65-bxbrl"
Nov 02 15:23:24.802340 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:23:24.802294    5304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k6fjk" containerName="extractor"
Nov 02 15:23:24.861732 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:24.861696    5304 patch_prober.go:29] interesting pod/e2e-inf-b-2bbcb5-openshift-default-788c5c5b65-bxbrl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" start-of-body=
Nov 02 15:23:24.861917 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:24.861758    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-2bbcb5-openshift-default-788c5c5b65-bxbrl" podUID="28cababb-e85d-48e8-b8a3-69e0914862aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused"
Nov 02 15:23:25.858152 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:25.853253    5304 patch_prober.go:29] interesting pod/e2e-inf-b-2bbcb5-openshift-default-788c5c5b65-bxbrl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" start-of-body=
Nov 02 15:23:25.858152 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:25.853325    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-2bbcb5-openshift-default-788c5c5b65-bxbrl" podUID="28cababb-e85d-48e8-b8a3-69e0914862aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused"
Nov 02 15:23:26.862745 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:26.861379    5304 patch_prober.go:29] interesting pod/e2e-inf-b-2bbcb5-openshift-default-788c5c5b65-bxbrl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" start-of-body=
Nov 02 15:23:26.862745 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:26.861464    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-2bbcb5-openshift-default-788c5c5b65-bxbrl" podUID="28cababb-e85d-48e8-b8a3-69e0914862aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused"
Nov 02 15:23:27.870161 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:27.849512    5304 patch_prober.go:29] interesting pod/e2e-inf-b-2bbcb5-openshift-default-788c5c5b65-bxbrl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" start-of-body=
Nov 02 15:23:27.870161 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:27.849572    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-2bbcb5-openshift-default-788c5c5b65-bxbrl" podUID="28cababb-e85d-48e8-b8a3-69e0914862aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused"
Nov 02 15:23:28.848156 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:28.847545    5304 patch_prober.go:29] interesting pod/e2e-inf-b-2bbcb5-openshift-default-788c5c5b65-bxbrl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" start-of-body=
Nov 02 15:23:28.848156 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:28.847629    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-2bbcb5-openshift-default-788c5c5b65-bxbrl" podUID="28cababb-e85d-48e8-b8a3-69e0914862aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused"
Nov 02 15:23:29.513717 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:29.513678    5304 generic.go:356] "Generic (PLEG): container finished" podID="c109588b-9544-4a37-bb13-d81572067546" containerID="8d68a037a516dffec694fa5bc9008369c95dbe007c3efb2c181b9dc643c4597a" exitCode=0
Nov 02 15:23:29.514282 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:29.513745    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a03e84-openshift-default-8f4cc949-4flt2" event={"ID":"c109588b-9544-4a37-bb13-d81572067546","Type":"ContainerDied","Data":"8d68a037a516dffec694fa5bc9008369c95dbe007c3efb2c181b9dc643c4597a"}
Nov 02 15:23:29.514282 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:29.514274    5304 scope.go:122] "RemoveContainer" containerID="8d68a037a516dffec694fa5bc9008369c95dbe007c3efb2c181b9dc643c4597a"
Nov 02 15:23:29.515742 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:29.515714    5304 generic.go:356] "Generic (PLEG): container finished" podID="de0eb9f2-e614-454a-a294-77fc9830a610" containerID="9f2169beab40a8765fdfcbb363eb57d18bd15f131072d9c4e777d228c1dfabff" exitCode=0
Nov 02 15:23:29.515850 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:29.515785    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-4ec02d-openshift-default-86955cc5c9-2j2bw" event={"ID":"de0eb9f2-e614-454a-a294-77fc9830a610","Type":"ContainerDied","Data":"9f2169beab40a8765fdfcbb363eb57d18bd15f131072d9c4e777d228c1dfabff"}
Nov 02 15:23:29.516191 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:29.516173    5304 scope.go:122] "RemoveContainer" containerID="9f2169beab40a8765fdfcbb363eb57d18bd15f131072d9c4e777d228c1dfabff"
Nov 02 15:23:29.516498 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:29.516465    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:23:29.516581 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:29.516539    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:23:29.516640 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:29.516580    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:23:29.518510 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:29.518473    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:23:29.518639 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:29.518565    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:23:29.518639 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:29.518616    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:23:29.868376 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:29.868326    5304 patch_prober.go:29] interesting pod/e2e-inf-b-2bbcb5-openshift-default-788c5c5b65-bxbrl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" start-of-body=
Nov 02 15:23:29.868607 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:29.868388    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-2bbcb5-openshift-default-788c5c5b65-bxbrl" podUID="28cababb-e85d-48e8-b8a3-69e0914862aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused"
Nov 02 15:23:30.545148 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:30.544627    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a03e84-openshift-default-8f4cc949-4flt2" event={"ID":"c109588b-9544-4a37-bb13-d81572067546","Type":"ContainerStarted","Data":"428106e52b4fde94e8785db87868da1cd0730d9066963e510fa436919ebcd580"}
Nov 02 15:23:30.557516 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:30.557481    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-4ec02d-openshift-default-86955cc5c9-2j2bw" event={"ID":"de0eb9f2-e614-454a-a294-77fc9830a610","Type":"ContainerStarted","Data":"8f6743d57906831a955d6819bf57b7df840590afd90ae35e0879082811e3ecc3"}
Nov 02 15:23:30.878248 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:30.874797    5304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-2bbcb5-openshift-default-788c5c5b65-bxbrl"
Nov 02 15:23:30.878248 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:30.876611    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-2bbcb5-openshift-default-788c5c5b65-bxbrl"
Nov 02 15:23:30.880474 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:30.880443    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-2bbcb5-openshift-default-788c5c5b65-bxbrl"
Nov 02 15:23:30.978151 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:30.976506    5304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-4ec02d-openshift-default-86955cc5c9-2j2bw"
Nov 02 15:23:30.983150 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:30.979451    5304 patch_prober.go:29] interesting pod/e2e-api-b-4ec02d-openshift-default-86955cc5c9-2j2bw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" start-of-body=
Nov 02 15:23:30.983150 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:30.979564    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-4ec02d-openshift-default-86955cc5c9-2j2bw" podUID="de0eb9f2-e614-454a-a294-77fc9830a610" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused"
Nov 02 15:23:30.983422 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:30.983223    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-a03e84-openshift-default-8f4cc949-4flt2"
Nov 02 15:23:31.405958 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:31.404829    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-lpbsq"]
Nov 02 15:23:31.405958 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:31.405226    5304 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-lpbsq" podUID="6e1530b5-8c7e-4a83-af8d-67bb9d476b50" containerName="istio-proxy" containerID="cri-o://028c1e2af928f940991002829c6a2a71071b358ea9a5bf46f5dafec298d82351" gracePeriod=30
Nov 02 15:23:31.427630 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:31.423660    5304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-a03e84-openshift-default-8f4cc949-4flt2"
Nov 02 15:23:31.427630 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:31.423806    5304 patch_prober.go:29] interesting pod/e2e-inf-a-a03e84-openshift-default-8f4cc949-4flt2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" start-of-body=
Nov 02 15:23:31.427630 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:31.423850    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a03e84-openshift-default-8f4cc949-4flt2" podUID="c109588b-9544-4a37-bb13-d81572067546" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused"
Nov 02 15:23:31.430068 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:31.430040    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-xtxns"]
Nov 02 15:23:31.430327 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:31.430295    5304 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-xtxns" podUID="abefd6e8-6072-45e0-9585-7c16ae3b6b35" containerName="istio-proxy" containerID="cri-o://1ef1c5c5df17bc3b3bfd3c8cfb4c51ddf09afca3930e7d5312c47a35249e9ee3" gracePeriod=30
Nov 02 15:23:31.540336 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:31.533363    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-zzvrc"]
Nov 02 15:23:31.540336 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:31.533450    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-d7mbd"]
Nov 02 15:23:31.540336 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:31.533709    5304 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-d7mbd" podUID="294f0d9e-6f89-477c-b882-61af0d1bcb5d" containerName="istio-proxy" containerID="cri-o://70a548f0546c3d97932ef0ce8342fe10d633fa2f51f4be994d663d8c3eb5ebf1" gracePeriod=30
Nov 02 15:23:31.540336 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:31.534000    5304 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-zzvrc" podUID="6057a4f9-4a74-4484-930f-501e9f6e6297" containerName="istio-proxy" containerID="cri-o://4dd9daf69f867ab40a288f81eccf8fde5b4951b498e13172f52546fb41cc36d3" gracePeriod=30
Nov 02 15:23:31.981149 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:31.979180    5304 patch_prober.go:29] interesting pod/e2e-api-b-4ec02d-openshift-default-86955cc5c9-2j2bw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" start-of-body=
Nov 02 15:23:31.981149 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:31.979266    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-4ec02d-openshift-default-86955cc5c9-2j2bw" podUID="de0eb9f2-e614-454a-a294-77fc9830a610" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused"
Nov 02 15:23:32.423521 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:32.423479    5304 patch_prober.go:29] interesting pod/e2e-inf-a-a03e84-openshift-default-8f4cc949-4flt2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" start-of-body=
Nov 02 15:23:32.423717 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:32.423611    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a03e84-openshift-default-8f4cc949-4flt2" podUID="c109588b-9544-4a37-bb13-d81572067546" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused"
Nov 02 15:23:33.047151 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:33.042342    5304 patch_prober.go:29] interesting pod/e2e-api-b-4ec02d-openshift-default-86955cc5c9-2j2bw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" start-of-body=
Nov 02 15:23:33.047151 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:33.042437    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-4ec02d-openshift-default-86955cc5c9-2j2bw" podUID="de0eb9f2-e614-454a-a294-77fc9830a610" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused"
Nov 02 15:23:33.423532 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:33.423500    5304 patch_prober.go:29] interesting pod/e2e-inf-a-a03e84-openshift-default-8f4cc949-4flt2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" start-of-body=
Nov 02 15:23:33.423742 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:33.423556    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a03e84-openshift-default-8f4cc949-4flt2" podUID="c109588b-9544-4a37-bb13-d81572067546" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused"
Nov 02 15:23:33.972149 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:33.971671    5304 patch_prober.go:29] interesting pod/e2e-api-b-4ec02d-openshift-default-86955cc5c9-2j2bw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" start-of-body=
Nov 02 15:23:33.972149 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:33.971730    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-4ec02d-openshift-default-86955cc5c9-2j2bw" podUID="de0eb9f2-e614-454a-a294-77fc9830a610" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused"
Nov 02 15:23:34.021682 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:34.021648    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-4ec02d-openshift-default-86955cc5c9-2j2bw"
Nov 02 15:23:34.424535 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:34.423867    5304 patch_prober.go:29] interesting pod/e2e-inf-a-a03e84-openshift-default-8f4cc949-4flt2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" start-of-body=
Nov 02 15:23:34.424535 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:34.423925    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a03e84-openshift-default-8f4cc949-4flt2" podUID="c109588b-9544-4a37-bb13-d81572067546" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused"
Nov 02 15:23:34.992146 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:34.978734    5304 patch_prober.go:29] interesting pod/e2e-api-b-4ec02d-openshift-default-86955cc5c9-2j2bw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" start-of-body=
Nov 02 15:23:34.992146 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:34.978804    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-4ec02d-openshift-default-86955cc5c9-2j2bw" podUID="de0eb9f2-e614-454a-a294-77fc9830a610" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused"
Nov 02 15:23:35.422907 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:35.422865    5304 patch_prober.go:29] interesting pod/e2e-inf-a-a03e84-openshift-default-8f4cc949-4flt2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" start-of-body=
Nov 02 15:23:35.423076 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:35.422926    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a03e84-openshift-default-8f4cc949-4flt2" podUID="c109588b-9544-4a37-bb13-d81572067546" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused"
Nov 02 15:23:35.972610 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:35.972572    5304 patch_prober.go:29] interesting pod/e2e-api-b-4ec02d-openshift-default-86955cc5c9-2j2bw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" start-of-body=
Nov 02 15:23:35.985041 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:35.985002    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-4ec02d-openshift-default-86955cc5c9-2j2bw" podUID="de0eb9f2-e614-454a-a294-77fc9830a610" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused"
Nov 02 15:23:36.422818 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:36.422788    5304 patch_prober.go:29] interesting pod/e2e-inf-a-a03e84-openshift-default-8f4cc949-4flt2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" start-of-body=
Nov 02 15:23:36.422964 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:36.422862    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a03e84-openshift-default-8f4cc949-4flt2" podUID="c109588b-9544-4a37-bb13-d81572067546" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused"
Nov 02 15:23:36.744139 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:36.744084    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-lpbsq"
Nov 02 15:23:36.836908 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:36.836874    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e1530b5-8c7e-4a83-af8d-67bb9d476b50-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6e1530b5-8c7e-4a83-af8d-67bb9d476b50-istio-envoy\") pod \"6e1530b5-8c7e-4a83-af8d-67bb9d476b50\" (UID: \"6e1530b5-8c7e-4a83-af8d-67bb9d476b50\") "
Nov 02 15:23:36.837071 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:36.836935    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6e1530b5-8c7e-4a83-af8d-67bb9d476b50-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6e1530b5-8c7e-4a83-af8d-67bb9d476b50-istio-podinfo\") pod \"6e1530b5-8c7e-4a83-af8d-67bb9d476b50\" (UID: \"6e1530b5-8c7e-4a83-af8d-67bb9d476b50\") "
Nov 02 15:23:36.837071 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:36.836965    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e1530b5-8c7e-4a83-af8d-67bb9d476b50-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e1530b5-8c7e-4a83-af8d-67bb9d476b50-workload-socket\") pod \"6e1530b5-8c7e-4a83-af8d-67bb9d476b50\" (UID: \"6e1530b5-8c7e-4a83-af8d-67bb9d476b50\") "
Nov 02 15:23:36.837071 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:36.837008    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e1530b5-8c7e-4a83-af8d-67bb9d476b50-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6e1530b5-8c7e-4a83-af8d-67bb9d476b50-workload-certs\") pod \"6e1530b5-8c7e-4a83-af8d-67bb9d476b50\" (UID: \"6e1530b5-8c7e-4a83-af8d-67bb9d476b50\") "
Nov 02 15:23:36.837272 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:36.837069    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6e1530b5-8c7e-4a83-af8d-67bb9d476b50-kube-api-access-87cgx\" (UniqueName: \"kubernetes.io/projected/6e1530b5-8c7e-4a83-af8d-67bb9d476b50-kube-api-access-87cgx\") pod \"6e1530b5-8c7e-4a83-af8d-67bb9d476b50\" (UID: \"6e1530b5-8c7e-4a83-af8d-67bb9d476b50\") "
Nov 02 15:23:36.837272 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:36.837206    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e1530b5-8c7e-4a83-af8d-67bb9d476b50-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6e1530b5-8c7e-4a83-af8d-67bb9d476b50-istio-data\") pod \"6e1530b5-8c7e-4a83-af8d-67bb9d476b50\" (UID: \"6e1530b5-8c7e-4a83-af8d-67bb9d476b50\") "
Nov 02 15:23:36.837382 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:36.837333    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6e1530b5-8c7e-4a83-af8d-67bb9d476b50-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6e1530b5-8c7e-4a83-af8d-67bb9d476b50-istiod-ca-cert\") pod \"6e1530b5-8c7e-4a83-af8d-67bb9d476b50\" (UID: \"6e1530b5-8c7e-4a83-af8d-67bb9d476b50\") "
Nov 02 15:23:36.837441 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:36.837381    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e1530b5-8c7e-4a83-af8d-67bb9d476b50-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e1530b5-8c7e-4a83-af8d-67bb9d476b50-credential-socket\") pod \"6e1530b5-8c7e-4a83-af8d-67bb9d476b50\" (UID: \"6e1530b5-8c7e-4a83-af8d-67bb9d476b50\") "
Nov 02 15:23:36.837521 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:36.837503    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6e1530b5-8c7e-4a83-af8d-67bb9d476b50-istio-token\" (UniqueName: \"kubernetes.io/projected/6e1530b5-8c7e-4a83-af8d-67bb9d476b50-istio-token\") pod \"6e1530b5-8c7e-4a83-af8d-67bb9d476b50\" (UID: \"6e1530b5-8c7e-4a83-af8d-67bb9d476b50\") "
Nov 02 15:23:36.837897 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:36.837643    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e1530b5-8c7e-4a83-af8d-67bb9d476b50-istio-data" pod "6e1530b5-8c7e-4a83-af8d-67bb9d476b50" (UID: "6e1530b5-8c7e-4a83-af8d-67bb9d476b50"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:23:36.837897 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:36.837853    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e1530b5-8c7e-4a83-af8d-67bb9d476b50-credential-socket" pod "6e1530b5-8c7e-4a83-af8d-67bb9d476b50" (UID: "6e1530b5-8c7e-4a83-af8d-67bb9d476b50"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:23:36.837897 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:36.837863    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e1530b5-8c7e-4a83-af8d-67bb9d476b50-istiod-ca-cert" pod "6e1530b5-8c7e-4a83-af8d-67bb9d476b50" (UID: "6e1530b5-8c7e-4a83-af8d-67bb9d476b50"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:23:36.838803 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:36.838180    5304 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6e1530b5-8c7e-4a83-af8d-67bb9d476b50-istio-data\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:23:36.838803 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:36.838196    5304 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6e1530b5-8c7e-4a83-af8d-67bb9d476b50-istiod-ca-cert\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:23:36.838803 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:36.838205    5304 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e1530b5-8c7e-4a83-af8d-67bb9d476b50-credential-socket\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:23:36.838803 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:36.838425    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e1530b5-8c7e-4a83-af8d-67bb9d476b50-workload-certs" pod "6e1530b5-8c7e-4a83-af8d-67bb9d476b50" (UID: "6e1530b5-8c7e-4a83-af8d-67bb9d476b50"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:23:36.838803 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:36.838533    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e1530b5-8c7e-4a83-af8d-67bb9d476b50-workload-socket" pod "6e1530b5-8c7e-4a83-af8d-67bb9d476b50" (UID: "6e1530b5-8c7e-4a83-af8d-67bb9d476b50"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:23:36.838803 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:36.838641    5304 generic.go:356] "Generic (PLEG): container finished" podID="6057a4f9-4a74-4484-930f-501e9f6e6297" containerID="4dd9daf69f867ab40a288f81eccf8fde5b4951b498e13172f52546fb41cc36d3" exitCode=0
Nov 02 15:23:36.838803 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:36.838718    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-zzvrc" event={"ID":"6057a4f9-4a74-4484-930f-501e9f6e6297","Type":"ContainerDied","Data":"4dd9daf69f867ab40a288f81eccf8fde5b4951b498e13172f52546fb41cc36d3"}
Nov 02 15:23:36.841382 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:36.841359    5304 generic.go:356] "Generic (PLEG): container finished" podID="abefd6e8-6072-45e0-9585-7c16ae3b6b35" containerID="1ef1c5c5df17bc3b3bfd3c8cfb4c51ddf09afca3930e7d5312c47a35249e9ee3" exitCode=0
Nov 02 15:23:36.841614 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:36.841595    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-xtxns" event={"ID":"abefd6e8-6072-45e0-9585-7c16ae3b6b35","Type":"ContainerDied","Data":"1ef1c5c5df17bc3b3bfd3c8cfb4c51ddf09afca3930e7d5312c47a35249e9ee3"}
Nov 02 15:23:36.844313 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:36.843294    5304 generic.go:356] "Generic (PLEG): container finished" podID="6e1530b5-8c7e-4a83-af8d-67bb9d476b50" containerID="028c1e2af928f940991002829c6a2a71071b358ea9a5bf46f5dafec298d82351" exitCode=0
Nov 02 15:23:36.844313 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:36.843357    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-lpbsq" event={"ID":"6e1530b5-8c7e-4a83-af8d-67bb9d476b50","Type":"ContainerDied","Data":"028c1e2af928f940991002829c6a2a71071b358ea9a5bf46f5dafec298d82351"}
Nov 02 15:23:36.844313 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:36.843396    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-lpbsq" event={"ID":"6e1530b5-8c7e-4a83-af8d-67bb9d476b50","Type":"ContainerDied","Data":"f63e61314607e851ca06981af8b5e81f4c03757d95a58557e132f032e489a5d4"}
Nov 02 15:23:36.844313 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:36.843418    5304 scope.go:122] "RemoveContainer" containerID="028c1e2af928f940991002829c6a2a71071b358ea9a5bf46f5dafec298d82351"
Nov 02 15:23:36.844313 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:36.843577    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-lpbsq"
Nov 02 15:23:36.845035 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:36.845013    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e1530b5-8c7e-4a83-af8d-67bb9d476b50-istio-envoy" pod "6e1530b5-8c7e-4a83-af8d-67bb9d476b50" (UID: "6e1530b5-8c7e-4a83-af8d-67bb9d476b50"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:23:36.845416 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:36.845388    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6e1530b5-8c7e-4a83-af8d-67bb9d476b50-istio-podinfo" pod "6e1530b5-8c7e-4a83-af8d-67bb9d476b50" (UID: "6e1530b5-8c7e-4a83-af8d-67bb9d476b50"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 15:23:36.845585 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:36.845562    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e1530b5-8c7e-4a83-af8d-67bb9d476b50-kube-api-access-87cgx" pod "6e1530b5-8c7e-4a83-af8d-67bb9d476b50" (UID: "6e1530b5-8c7e-4a83-af8d-67bb9d476b50"). InnerVolumeSpecName "kube-api-access-87cgx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:23:36.849724 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:36.849679    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e1530b5-8c7e-4a83-af8d-67bb9d476b50-istio-token" pod "6e1530b5-8c7e-4a83-af8d-67bb9d476b50" (UID: "6e1530b5-8c7e-4a83-af8d-67bb9d476b50"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:23:36.882104 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:36.881774    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-xtxns"
Nov 02 15:23:36.889537 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:36.889315    5304 scope.go:122] "RemoveContainer" containerID="028c1e2af928f940991002829c6a2a71071b358ea9a5bf46f5dafec298d82351"
Nov 02 15:23:36.890051 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:23:36.889948    5304 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"028c1e2af928f940991002829c6a2a71071b358ea9a5bf46f5dafec298d82351\": container with ID starting with 028c1e2af928f940991002829c6a2a71071b358ea9a5bf46f5dafec298d82351 not found: ID does not exist" containerID="028c1e2af928f940991002829c6a2a71071b358ea9a5bf46f5dafec298d82351"
Nov 02 15:23:36.890051 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:36.889991    5304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"028c1e2af928f940991002829c6a2a71071b358ea9a5bf46f5dafec298d82351"} err="failed to get container status \"028c1e2af928f940991002829c6a2a71071b358ea9a5bf46f5dafec298d82351\": rpc error: code = NotFound desc = could not find container \"028c1e2af928f940991002829c6a2a71071b358ea9a5bf46f5dafec298d82351\": container with ID starting with 028c1e2af928f940991002829c6a2a71071b358ea9a5bf46f5dafec298d82351 not found: ID does not exist"
Nov 02 15:23:36.940427 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:36.938740    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/abefd6e8-6072-45e0-9585-7c16ae3b6b35-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/abefd6e8-6072-45e0-9585-7c16ae3b6b35-istio-data\") pod \"abefd6e8-6072-45e0-9585-7c16ae3b6b35\" (UID: \"abefd6e8-6072-45e0-9585-7c16ae3b6b35\") "
Nov 02 15:23:36.940427 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:36.938819    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/abefd6e8-6072-45e0-9585-7c16ae3b6b35-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/abefd6e8-6072-45e0-9585-7c16ae3b6b35-workload-certs\") pod \"abefd6e8-6072-45e0-9585-7c16ae3b6b35\" (UID: \"abefd6e8-6072-45e0-9585-7c16ae3b6b35\") "
Nov 02 15:23:36.940427 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:36.938854    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/abefd6e8-6072-45e0-9585-7c16ae3b6b35-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/abefd6e8-6072-45e0-9585-7c16ae3b6b35-istiod-ca-cert\") pod \"abefd6e8-6072-45e0-9585-7c16ae3b6b35\" (UID: \"abefd6e8-6072-45e0-9585-7c16ae3b6b35\") "
Nov 02 15:23:36.940427 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:36.938983    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/abefd6e8-6072-45e0-9585-7c16ae3b6b35-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/abefd6e8-6072-45e0-9585-7c16ae3b6b35-istio-podinfo\") pod \"abefd6e8-6072-45e0-9585-7c16ae3b6b35\" (UID: \"abefd6e8-6072-45e0-9585-7c16ae3b6b35\") "
Nov 02 15:23:36.940427 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:36.939023    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/abefd6e8-6072-45e0-9585-7c16ae3b6b35-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/abefd6e8-6072-45e0-9585-7c16ae3b6b35-workload-socket\") pod \"abefd6e8-6072-45e0-9585-7c16ae3b6b35\" (UID: \"abefd6e8-6072-45e0-9585-7c16ae3b6b35\") "
Nov 02 15:23:36.940427 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:36.939082    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/abefd6e8-6072-45e0-9585-7c16ae3b6b35-istio-token\" (UniqueName: \"kubernetes.io/projected/abefd6e8-6072-45e0-9585-7c16ae3b6b35-istio-token\") pod \"abefd6e8-6072-45e0-9585-7c16ae3b6b35\" (UID: \"abefd6e8-6072-45e0-9585-7c16ae3b6b35\") "
Nov 02 15:23:36.940427 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:36.939159    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/abefd6e8-6072-45e0-9585-7c16ae3b6b35-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/abefd6e8-6072-45e0-9585-7c16ae3b6b35-istio-envoy\") pod \"abefd6e8-6072-45e0-9585-7c16ae3b6b35\" (UID: \"abefd6e8-6072-45e0-9585-7c16ae3b6b35\") "
Nov 02 15:23:36.940427 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:36.939195    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/abefd6e8-6072-45e0-9585-7c16ae3b6b35-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/abefd6e8-6072-45e0-9585-7c16ae3b6b35-credential-socket\") pod \"abefd6e8-6072-45e0-9585-7c16ae3b6b35\" (UID: \"abefd6e8-6072-45e0-9585-7c16ae3b6b35\") "
Nov 02 15:23:36.940427 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:36.939266    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/abefd6e8-6072-45e0-9585-7c16ae3b6b35-kube-api-access-7rb6z\" (UniqueName: \"kubernetes.io/projected/abefd6e8-6072-45e0-9585-7c16ae3b6b35-kube-api-access-7rb6z\") pod \"abefd6e8-6072-45e0-9585-7c16ae3b6b35\" (UID: \"abefd6e8-6072-45e0-9585-7c16ae3b6b35\") "
Nov 02 15:23:36.940427 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:36.939962    5304 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6e1530b5-8c7e-4a83-af8d-67bb9d476b50-istio-token\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:23:36.940427 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:36.939987    5304 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6e1530b5-8c7e-4a83-af8d-67bb9d476b50-istio-envoy\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:23:36.940427 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:36.940002    5304 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6e1530b5-8c7e-4a83-af8d-67bb9d476b50-istio-podinfo\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:23:36.940427 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:36.940014    5304 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e1530b5-8c7e-4a83-af8d-67bb9d476b50-workload-socket\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:23:36.940427 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:36.940028    5304 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6e1530b5-8c7e-4a83-af8d-67bb9d476b50-workload-certs\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:23:36.940427 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:36.940041    5304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-87cgx\" (UniqueName: \"kubernetes.io/projected/6e1530b5-8c7e-4a83-af8d-67bb9d476b50-kube-api-access-87cgx\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:23:36.941404 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:36.940763    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abefd6e8-6072-45e0-9585-7c16ae3b6b35-istio-data" pod "abefd6e8-6072-45e0-9585-7c16ae3b6b35" (UID: "abefd6e8-6072-45e0-9585-7c16ae3b6b35"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:23:36.941404 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:36.940956    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abefd6e8-6072-45e0-9585-7c16ae3b6b35-workload-certs" pod "abefd6e8-6072-45e0-9585-7c16ae3b6b35" (UID: "abefd6e8-6072-45e0-9585-7c16ae3b6b35"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:23:36.941404 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:36.941278    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/abefd6e8-6072-45e0-9585-7c16ae3b6b35-istiod-ca-cert" pod "abefd6e8-6072-45e0-9585-7c16ae3b6b35" (UID: "abefd6e8-6072-45e0-9585-7c16ae3b6b35"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:23:36.942059 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:36.942031    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abefd6e8-6072-45e0-9585-7c16ae3b6b35-credential-socket" pod "abefd6e8-6072-45e0-9585-7c16ae3b6b35" (UID: "abefd6e8-6072-45e0-9585-7c16ae3b6b35"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:23:36.942227 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:36.942204    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abefd6e8-6072-45e0-9585-7c16ae3b6b35-workload-socket" pod "abefd6e8-6072-45e0-9585-7c16ae3b6b35" (UID: "abefd6e8-6072-45e0-9585-7c16ae3b6b35"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:23:36.945709 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:36.945672    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abefd6e8-6072-45e0-9585-7c16ae3b6b35-kube-api-access-7rb6z" pod "abefd6e8-6072-45e0-9585-7c16ae3b6b35" (UID: "abefd6e8-6072-45e0-9585-7c16ae3b6b35"). InnerVolumeSpecName "kube-api-access-7rb6z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:23:36.946370 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:36.946344    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abefd6e8-6072-45e0-9585-7c16ae3b6b35-istio-token" pod "abefd6e8-6072-45e0-9585-7c16ae3b6b35" (UID: "abefd6e8-6072-45e0-9585-7c16ae3b6b35"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:23:36.947281 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:36.946958    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abefd6e8-6072-45e0-9585-7c16ae3b6b35-istio-envoy" pod "abefd6e8-6072-45e0-9585-7c16ae3b6b35" (UID: "abefd6e8-6072-45e0-9585-7c16ae3b6b35"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:23:36.955170 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:36.954232    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/abefd6e8-6072-45e0-9585-7c16ae3b6b35-istio-podinfo" pod "abefd6e8-6072-45e0-9585-7c16ae3b6b35" (UID: "abefd6e8-6072-45e0-9585-7c16ae3b6b35"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 15:23:36.971501 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:36.971475    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-zzvrc"
Nov 02 15:23:36.977919 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:36.977894    5304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-4ec02d-openshift-default-86955cc5c9-2j2bw"
Nov 02 15:23:36.981478 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:36.981455    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-4ec02d-openshift-default-86955cc5c9-2j2bw"
Nov 02 15:23:37.041286 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:37.041248    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6057a4f9-4a74-4484-930f-501e9f6e6297-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6057a4f9-4a74-4484-930f-501e9f6e6297-istio-data\") pod \"6057a4f9-4a74-4484-930f-501e9f6e6297\" (UID: \"6057a4f9-4a74-4484-930f-501e9f6e6297\") "
Nov 02 15:23:37.041461 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:37.041313    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6057a4f9-4a74-4484-930f-501e9f6e6297-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6057a4f9-4a74-4484-930f-501e9f6e6297-istio-podinfo\") pod \"6057a4f9-4a74-4484-930f-501e9f6e6297\" (UID: \"6057a4f9-4a74-4484-930f-501e9f6e6297\") "
Nov 02 15:23:37.041461 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:37.041377    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6057a4f9-4a74-4484-930f-501e9f6e6297-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6057a4f9-4a74-4484-930f-501e9f6e6297-workload-certs\") pod \"6057a4f9-4a74-4484-930f-501e9f6e6297\" (UID: \"6057a4f9-4a74-4484-930f-501e9f6e6297\") "
Nov 02 15:23:37.041461 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:37.041451    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6057a4f9-4a74-4484-930f-501e9f6e6297-istio-token\" (UniqueName: \"kubernetes.io/projected/6057a4f9-4a74-4484-930f-501e9f6e6297-istio-token\") pod \"6057a4f9-4a74-4484-930f-501e9f6e6297\" (UID: \"6057a4f9-4a74-4484-930f-501e9f6e6297\") "
Nov 02 15:23:37.041656 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:37.041492    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6057a4f9-4a74-4484-930f-501e9f6e6297-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6057a4f9-4a74-4484-930f-501e9f6e6297-workload-socket\") pod \"6057a4f9-4a74-4484-930f-501e9f6e6297\" (UID: \"6057a4f9-4a74-4484-930f-501e9f6e6297\") "
Nov 02 15:23:37.041656 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:37.041580    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6057a4f9-4a74-4484-930f-501e9f6e6297-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6057a4f9-4a74-4484-930f-501e9f6e6297-credential-socket\") pod \"6057a4f9-4a74-4484-930f-501e9f6e6297\" (UID: \"6057a4f9-4a74-4484-930f-501e9f6e6297\") "
Nov 02 15:23:37.041774 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:37.041735    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6057a4f9-4a74-4484-930f-501e9f6e6297-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6057a4f9-4a74-4484-930f-501e9f6e6297-istio-envoy\") pod \"6057a4f9-4a74-4484-930f-501e9f6e6297\" (UID: \"6057a4f9-4a74-4484-930f-501e9f6e6297\") "
Nov 02 15:23:37.041840 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:37.041772    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6057a4f9-4a74-4484-930f-501e9f6e6297-kube-api-access-kwgl6\" (UniqueName: \"kubernetes.io/projected/6057a4f9-4a74-4484-930f-501e9f6e6297-kube-api-access-kwgl6\") pod \"6057a4f9-4a74-4484-930f-501e9f6e6297\" (UID: \"6057a4f9-4a74-4484-930f-501e9f6e6297\") "
Nov 02 15:23:37.041840 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:37.041822    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6057a4f9-4a74-4484-930f-501e9f6e6297-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6057a4f9-4a74-4484-930f-501e9f6e6297-istiod-ca-cert\") pod \"6057a4f9-4a74-4484-930f-501e9f6e6297\" (UID: \"6057a4f9-4a74-4484-930f-501e9f6e6297\") "
Nov 02 15:23:37.042782 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:37.042758    5304 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/abefd6e8-6072-45e0-9585-7c16ae3b6b35-istio-data\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:23:37.042878 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:37.042789    5304 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/abefd6e8-6072-45e0-9585-7c16ae3b6b35-workload-certs\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:23:37.042878 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:37.042804    5304 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/abefd6e8-6072-45e0-9585-7c16ae3b6b35-istiod-ca-cert\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:23:37.042878 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:37.042817    5304 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/abefd6e8-6072-45e0-9585-7c16ae3b6b35-istio-podinfo\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:23:37.042878 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:37.042829    5304 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/abefd6e8-6072-45e0-9585-7c16ae3b6b35-workload-socket\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:23:37.042878 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:37.042843    5304 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/abefd6e8-6072-45e0-9585-7c16ae3b6b35-istio-token\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:23:37.042878 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:37.042855    5304 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/abefd6e8-6072-45e0-9585-7c16ae3b6b35-istio-envoy\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:23:37.042878 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:37.042867    5304 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/abefd6e8-6072-45e0-9585-7c16ae3b6b35-credential-socket\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:23:37.043108 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:37.042880    5304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7rb6z\" (UniqueName: \"kubernetes.io/projected/abefd6e8-6072-45e0-9585-7c16ae3b6b35-kube-api-access-7rb6z\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:23:37.043403 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:37.043378    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6057a4f9-4a74-4484-930f-501e9f6e6297-istio-data" pod "6057a4f9-4a74-4484-930f-501e9f6e6297" (UID: "6057a4f9-4a74-4484-930f-501e9f6e6297"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:23:37.043999 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:37.043972    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6057a4f9-4a74-4484-930f-501e9f6e6297-workload-certs" pod "6057a4f9-4a74-4484-930f-501e9f6e6297" (UID: "6057a4f9-4a74-4484-930f-501e9f6e6297"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:23:37.044202 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:37.044180    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6057a4f9-4a74-4484-930f-501e9f6e6297-workload-socket" pod "6057a4f9-4a74-4484-930f-501e9f6e6297" (UID: "6057a4f9-4a74-4484-930f-501e9f6e6297"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:23:37.044385 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:37.044366    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6057a4f9-4a74-4484-930f-501e9f6e6297-credential-socket" pod "6057a4f9-4a74-4484-930f-501e9f6e6297" (UID: "6057a4f9-4a74-4484-930f-501e9f6e6297"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:23:37.045919 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:37.045877    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6057a4f9-4a74-4484-930f-501e9f6e6297-istiod-ca-cert" pod "6057a4f9-4a74-4484-930f-501e9f6e6297" (UID: "6057a4f9-4a74-4484-930f-501e9f6e6297"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:23:37.046083 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:37.045995    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6057a4f9-4a74-4484-930f-501e9f6e6297-istio-envoy" pod "6057a4f9-4a74-4484-930f-501e9f6e6297" (UID: "6057a4f9-4a74-4484-930f-501e9f6e6297"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:23:37.046976 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:37.046952    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6057a4f9-4a74-4484-930f-501e9f6e6297-istio-podinfo" pod "6057a4f9-4a74-4484-930f-501e9f6e6297" (UID: "6057a4f9-4a74-4484-930f-501e9f6e6297"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 15:23:37.048643 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:37.048615    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6057a4f9-4a74-4484-930f-501e9f6e6297-istio-token" pod "6057a4f9-4a74-4484-930f-501e9f6e6297" (UID: "6057a4f9-4a74-4484-930f-501e9f6e6297"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:23:37.052920 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:37.052889    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6057a4f9-4a74-4484-930f-501e9f6e6297-kube-api-access-kwgl6" pod "6057a4f9-4a74-4484-930f-501e9f6e6297" (UID: "6057a4f9-4a74-4484-930f-501e9f6e6297"). InnerVolumeSpecName "kube-api-access-kwgl6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:23:37.094602 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:37.094570    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-d7mbd"
Nov 02 15:23:37.145347 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:37.145102    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/294f0d9e-6f89-477c-b882-61af0d1bcb5d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/294f0d9e-6f89-477c-b882-61af0d1bcb5d-workload-certs\") pod \"294f0d9e-6f89-477c-b882-61af0d1bcb5d\" (UID: \"294f0d9e-6f89-477c-b882-61af0d1bcb5d\") "
Nov 02 15:23:37.145770 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:37.145645    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/294f0d9e-6f89-477c-b882-61af0d1bcb5d-workload-certs" pod "294f0d9e-6f89-477c-b882-61af0d1bcb5d" (UID: "294f0d9e-6f89-477c-b882-61af0d1bcb5d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:23:37.146025 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:37.146007    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/294f0d9e-6f89-477c-b882-61af0d1bcb5d-kube-api-access-6jt89\" (UniqueName: \"kubernetes.io/projected/294f0d9e-6f89-477c-b882-61af0d1bcb5d-kube-api-access-6jt89\") pod \"294f0d9e-6f89-477c-b882-61af0d1bcb5d\" (UID: \"294f0d9e-6f89-477c-b882-61af0d1bcb5d\") "
Nov 02 15:23:37.146307 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:37.146289    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/294f0d9e-6f89-477c-b882-61af0d1bcb5d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/294f0d9e-6f89-477c-b882-61af0d1bcb5d-istio-data\") pod \"294f0d9e-6f89-477c-b882-61af0d1bcb5d\" (UID: \"294f0d9e-6f89-477c-b882-61af0d1bcb5d\") "
Nov 02 15:23:37.146453 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:37.146439    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/294f0d9e-6f89-477c-b882-61af0d1bcb5d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/294f0d9e-6f89-477c-b882-61af0d1bcb5d-workload-socket\") pod \"294f0d9e-6f89-477c-b882-61af0d1bcb5d\" (UID: \"294f0d9e-6f89-477c-b882-61af0d1bcb5d\") "
Nov 02 15:23:37.146604 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:37.146555    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/294f0d9e-6f89-477c-b882-61af0d1bcb5d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/294f0d9e-6f89-477c-b882-61af0d1bcb5d-istiod-ca-cert\") pod \"294f0d9e-6f89-477c-b882-61af0d1bcb5d\" (UID: \"294f0d9e-6f89-477c-b882-61af0d1bcb5d\") "
Nov 02 15:23:37.146771 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:37.146757    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/294f0d9e-6f89-477c-b882-61af0d1bcb5d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/294f0d9e-6f89-477c-b882-61af0d1bcb5d-istio-envoy\") pod \"294f0d9e-6f89-477c-b882-61af0d1bcb5d\" (UID: \"294f0d9e-6f89-477c-b882-61af0d1bcb5d\") "
Nov 02 15:23:37.146908 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:37.146893    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/294f0d9e-6f89-477c-b882-61af0d1bcb5d-istio-token\" (UniqueName: \"kubernetes.io/projected/294f0d9e-6f89-477c-b882-61af0d1bcb5d-istio-token\") pod \"294f0d9e-6f89-477c-b882-61af0d1bcb5d\" (UID: \"294f0d9e-6f89-477c-b882-61af0d1bcb5d\") "
Nov 02 15:23:37.147072 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:37.147057    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/294f0d9e-6f89-477c-b882-61af0d1bcb5d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/294f0d9e-6f89-477c-b882-61af0d1bcb5d-credential-socket\") pod \"294f0d9e-6f89-477c-b882-61af0d1bcb5d\" (UID: \"294f0d9e-6f89-477c-b882-61af0d1bcb5d\") "
Nov 02 15:23:37.147210 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:37.147195    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/294f0d9e-6f89-477c-b882-61af0d1bcb5d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/294f0d9e-6f89-477c-b882-61af0d1bcb5d-istio-podinfo\") pod \"294f0d9e-6f89-477c-b882-61af0d1bcb5d\" (UID: \"294f0d9e-6f89-477c-b882-61af0d1bcb5d\") "
Nov 02 15:23:37.147893 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:37.147110    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/294f0d9e-6f89-477c-b882-61af0d1bcb5d-istiod-ca-cert" pod "294f0d9e-6f89-477c-b882-61af0d1bcb5d" (UID: "294f0d9e-6f89-477c-b882-61af0d1bcb5d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:23:37.147893 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:37.147535    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/294f0d9e-6f89-477c-b882-61af0d1bcb5d-workload-socket" pod "294f0d9e-6f89-477c-b882-61af0d1bcb5d" (UID: "294f0d9e-6f89-477c-b882-61af0d1bcb5d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:23:37.147893 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:37.147559    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/294f0d9e-6f89-477c-b882-61af0d1bcb5d-istio-data" pod "294f0d9e-6f89-477c-b882-61af0d1bcb5d" (UID: "294f0d9e-6f89-477c-b882-61af0d1bcb5d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:23:37.147893 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:37.147722    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/294f0d9e-6f89-477c-b882-61af0d1bcb5d-credential-socket" pod "294f0d9e-6f89-477c-b882-61af0d1bcb5d" (UID: "294f0d9e-6f89-477c-b882-61af0d1bcb5d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:23:37.148457 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:37.148427    5304 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/294f0d9e-6f89-477c-b882-61af0d1bcb5d-credential-socket\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:23:37.148578 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:37.148564    5304 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6057a4f9-4a74-4484-930f-501e9f6e6297-istio-data\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:23:37.148667 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:37.148657    5304 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6057a4f9-4a74-4484-930f-501e9f6e6297-istio-podinfo\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:23:37.148771 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:37.148759    5304 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/294f0d9e-6f89-477c-b882-61af0d1bcb5d-workload-certs\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:23:37.148864 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:37.148853    5304 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6057a4f9-4a74-4484-930f-501e9f6e6297-workload-certs\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:23:37.148961 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:37.148951    5304 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6057a4f9-4a74-4484-930f-501e9f6e6297-istio-token\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:23:37.149037 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:37.149020    5304 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6057a4f9-4a74-4484-930f-501e9f6e6297-workload-socket\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:23:37.149114 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:37.149039    5304 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6057a4f9-4a74-4484-930f-501e9f6e6297-credential-socket\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:23:37.149114 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:37.149053    5304 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/294f0d9e-6f89-477c-b882-61af0d1bcb5d-istio-data\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:23:37.149114 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:37.149068    5304 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/294f0d9e-6f89-477c-b882-61af0d1bcb5d-workload-socket\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:23:37.149114 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:37.149081    5304 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/294f0d9e-6f89-477c-b882-61af0d1bcb5d-istiod-ca-cert\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:23:37.149114 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:37.149096    5304 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6057a4f9-4a74-4484-930f-501e9f6e6297-istio-envoy\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:23:37.149114 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:37.149109    5304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kwgl6\" (UniqueName: \"kubernetes.io/projected/6057a4f9-4a74-4484-930f-501e9f6e6297-kube-api-access-kwgl6\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:23:37.149541 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:37.149139    5304 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6057a4f9-4a74-4484-930f-501e9f6e6297-istiod-ca-cert\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:23:37.153549 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:37.153524    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/294f0d9e-6f89-477c-b882-61af0d1bcb5d-istio-envoy" pod "294f0d9e-6f89-477c-b882-61af0d1bcb5d" (UID: "294f0d9e-6f89-477c-b882-61af0d1bcb5d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:23:37.153969 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:37.153947    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/294f0d9e-6f89-477c-b882-61af0d1bcb5d-kube-api-access-6jt89" pod "294f0d9e-6f89-477c-b882-61af0d1bcb5d" (UID: "294f0d9e-6f89-477c-b882-61af0d1bcb5d"). InnerVolumeSpecName "kube-api-access-6jt89". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:23:37.154070 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:37.153981    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/294f0d9e-6f89-477c-b882-61af0d1bcb5d-istio-token" pod "294f0d9e-6f89-477c-b882-61af0d1bcb5d" (UID: "294f0d9e-6f89-477c-b882-61af0d1bcb5d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:23:37.154070 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:37.153995    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/294f0d9e-6f89-477c-b882-61af0d1bcb5d-istio-podinfo" pod "294f0d9e-6f89-477c-b882-61af0d1bcb5d" (UID: "294f0d9e-6f89-477c-b882-61af0d1bcb5d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 15:23:37.201121 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:37.201034    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-lpbsq"]
Nov 02 15:23:37.204775 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:37.204742    5304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-lpbsq"]
Nov 02 15:23:37.250463 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:37.250428    5304 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/294f0d9e-6f89-477c-b882-61af0d1bcb5d-istio-token\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:23:37.250463 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:37.250457    5304 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/294f0d9e-6f89-477c-b882-61af0d1bcb5d-istio-podinfo\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:23:37.250463 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:37.250466    5304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6jt89\" (UniqueName: \"kubernetes.io/projected/294f0d9e-6f89-477c-b882-61af0d1bcb5d-kube-api-access-6jt89\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:23:37.250696 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:37.250476    5304 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/294f0d9e-6f89-477c-b882-61af0d1bcb5d-istio-envoy\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:23:37.428740 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:37.428708    5304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-a03e84-openshift-default-8f4cc949-4flt2"
Nov 02 15:23:37.430080 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:37.430050    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-a03e84-openshift-default-8f4cc949-4flt2"
Nov 02 15:23:37.655120 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:37.655087    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-55e4c3-578b56f96d-lmkft"]
Nov 02 15:23:37.655359 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:37.655334    5304 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-55e4c3-578b56f96d-lmkft" podUID="494bd9cf-9a4d-4448-8850-f746c50b5c8e" containerName="maas-api" containerID="cri-o://eab6fd8ca782f752159b380bc059b5ea2125f8eb5edf3fde9493d0a11d2cb00a" gracePeriod=30
Nov 02 15:23:37.859146 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:37.859100    5304 generic.go:356] "Generic (PLEG): container finished" podID="294f0d9e-6f89-477c-b882-61af0d1bcb5d" containerID="70a548f0546c3d97932ef0ce8342fe10d633fa2f51f4be994d663d8c3eb5ebf1" exitCode=0
Nov 02 15:23:37.859331 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:37.859173    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-d7mbd"
Nov 02 15:23:37.859331 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:37.859226    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-d7mbd" event={"ID":"294f0d9e-6f89-477c-b882-61af0d1bcb5d","Type":"ContainerDied","Data":"70a548f0546c3d97932ef0ce8342fe10d633fa2f51f4be994d663d8c3eb5ebf1"}
Nov 02 15:23:37.859331 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:37.859271    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-d7mbd" event={"ID":"294f0d9e-6f89-477c-b882-61af0d1bcb5d","Type":"ContainerDied","Data":"67034ee64af4db69ab6d234d96275769bfd004d2d9df1e37e684d8c5e2ca4ead"}
Nov 02 15:23:37.859331 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:37.859301    5304 scope.go:122] "RemoveContainer" containerID="70a548f0546c3d97932ef0ce8342fe10d633fa2f51f4be994d663d8c3eb5ebf1"
Nov 02 15:23:37.862371 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:37.862344    5304 generic.go:356] "Generic (PLEG): container finished" podID="494bd9cf-9a4d-4448-8850-f746c50b5c8e" containerID="eab6fd8ca782f752159b380bc059b5ea2125f8eb5edf3fde9493d0a11d2cb00a" exitCode=0
Nov 02 15:23:37.862534 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:37.862515    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-55e4c3-578b56f96d-lmkft" event={"ID":"494bd9cf-9a4d-4448-8850-f746c50b5c8e","Type":"ContainerDied","Data":"eab6fd8ca782f752159b380bc059b5ea2125f8eb5edf3fde9493d0a11d2cb00a"}
Nov 02 15:23:37.865619 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:37.865594    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-zzvrc" event={"ID":"6057a4f9-4a74-4484-930f-501e9f6e6297","Type":"ContainerDied","Data":"6a8344e2b6688133c2c9456924381bef60c40591b2c6e52ed07523c052093fcd"}
Nov 02 15:23:37.865751 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:37.865601    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-zzvrc"
Nov 02 15:23:37.872019 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:37.869572    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-xtxns"
Nov 02 15:23:37.872019 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:37.871984    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-xtxns" event={"ID":"abefd6e8-6072-45e0-9585-7c16ae3b6b35","Type":"ContainerDied","Data":"5058c79345a2be62594db9c382adee6d374c86c3db0b4a018712fb3b6036df59"}
Nov 02 15:23:37.889596 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:37.876930    5304 scope.go:122] "RemoveContainer" containerID="65ae4b6158338164d9f26b447de78665d60d5d9499452284efb07562830609b7"
Nov 02 15:23:38.089704 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:38.089672    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-d7mbd"]
Nov 02 15:23:38.097599 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:38.097571    5304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-d7mbd"]
Nov 02 15:23:38.140283 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:38.140247    5304 scope.go:122] "RemoveContainer" containerID="70a548f0546c3d97932ef0ce8342fe10d633fa2f51f4be994d663d8c3eb5ebf1"
Nov 02 15:23:38.159431 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:38.151371    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-zzvrc"]
Nov 02 15:23:38.165749 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:23:38.165272    5304 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"70a548f0546c3d97932ef0ce8342fe10d633fa2f51f4be994d663d8c3eb5ebf1\": container with ID starting with 70a548f0546c3d97932ef0ce8342fe10d633fa2f51f4be994d663d8c3eb5ebf1 not found: ID does not exist" containerID="70a548f0546c3d97932ef0ce8342fe10d633fa2f51f4be994d663d8c3eb5ebf1"
Nov 02 15:23:38.165749 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:38.165321    5304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70a548f0546c3d97932ef0ce8342fe10d633fa2f51f4be994d663d8c3eb5ebf1"} err="failed to get container status \"70a548f0546c3d97932ef0ce8342fe10d633fa2f51f4be994d663d8c3eb5ebf1\": rpc error: code = NotFound desc = could not find container \"70a548f0546c3d97932ef0ce8342fe10d633fa2f51f4be994d663d8c3eb5ebf1\": container with ID starting with 70a548f0546c3d97932ef0ce8342fe10d633fa2f51f4be994d663d8c3eb5ebf1 not found: ID does not exist"
Nov 02 15:23:38.165749 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:38.165351    5304 scope.go:122] "RemoveContainer" containerID="65ae4b6158338164d9f26b447de78665d60d5d9499452284efb07562830609b7"
Nov 02 15:23:38.181097 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:23:38.166653    5304 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"65ae4b6158338164d9f26b447de78665d60d5d9499452284efb07562830609b7\": container with ID starting with 65ae4b6158338164d9f26b447de78665d60d5d9499452284efb07562830609b7 not found: ID does not exist" containerID="65ae4b6158338164d9f26b447de78665d60d5d9499452284efb07562830609b7"
Nov 02 15:23:38.181097 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:38.166699    5304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"65ae4b6158338164d9f26b447de78665d60d5d9499452284efb07562830609b7"} err="failed to get container status \"65ae4b6158338164d9f26b447de78665d60d5d9499452284efb07562830609b7\": rpc error: code = NotFound desc = could not find container \"65ae4b6158338164d9f26b447de78665d60d5d9499452284efb07562830609b7\": container with ID starting with 65ae4b6158338164d9f26b447de78665d60d5d9499452284efb07562830609b7 not found: ID does not exist"
Nov 02 15:23:38.181097 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:38.166721    5304 scope.go:122] "RemoveContainer" containerID="4dd9daf69f867ab40a288f81eccf8fde5b4951b498e13172f52546fb41cc36d3"
Nov 02 15:23:38.181097 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:38.177258    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-55e4c3-578b56f96d-lmkft"
Nov 02 15:23:38.188503 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:38.188473    5304 scope.go:122] "RemoveContainer" containerID="1ef1c5c5df17bc3b3bfd3c8cfb4c51ddf09afca3930e7d5312c47a35249e9ee3"
Nov 02 15:23:38.206148 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:38.202101    5304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="294f0d9e-6f89-477c-b882-61af0d1bcb5d" path="/var/lib/kubelet/pods/294f0d9e-6f89-477c-b882-61af0d1bcb5d/volumes"
Nov 02 15:23:38.207167 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:38.207097    5304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6e1530b5-8c7e-4a83-af8d-67bb9d476b50" path="/var/lib/kubelet/pods/6e1530b5-8c7e-4a83-af8d-67bb9d476b50/volumes"
Nov 02 15:23:38.210663 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:38.210636    5304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-zzvrc"]
Nov 02 15:23:38.210792 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:38.210704    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-xtxns"]
Nov 02 15:23:38.210792 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:38.210731    5304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-xtxns"]
Nov 02 15:23:38.295449 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:38.293170    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/494bd9cf-9a4d-4448-8850-f746c50b5c8e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/494bd9cf-9a4d-4448-8850-f746c50b5c8e-maas-api-tls\") pod \"494bd9cf-9a4d-4448-8850-f746c50b5c8e\" (UID: \"494bd9cf-9a4d-4448-8850-f746c50b5c8e\") "
Nov 02 15:23:38.295449 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:38.293244    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/494bd9cf-9a4d-4448-8850-f746c50b5c8e-kube-api-access-hbsnz\" (UniqueName: \"kubernetes.io/projected/494bd9cf-9a4d-4448-8850-f746c50b5c8e-kube-api-access-hbsnz\") pod \"494bd9cf-9a4d-4448-8850-f746c50b5c8e\" (UID: \"494bd9cf-9a4d-4448-8850-f746c50b5c8e\") "
Nov 02 15:23:38.295449 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:38.293344    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/494bd9cf-9a4d-4448-8850-f746c50b5c8e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/494bd9cf-9a4d-4448-8850-f746c50b5c8e-metrics-certs\") pod \"494bd9cf-9a4d-4448-8850-f746c50b5c8e\" (UID: \"494bd9cf-9a4d-4448-8850-f746c50b5c8e\") "
Nov 02 15:23:38.295449 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:38.293432    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/494bd9cf-9a4d-4448-8850-f746c50b5c8e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/494bd9cf-9a4d-4448-8850-f746c50b5c8e-openshift-service-ca\") pod \"494bd9cf-9a4d-4448-8850-f746c50b5c8e\" (UID: \"494bd9cf-9a4d-4448-8850-f746c50b5c8e\") "
Nov 02 15:23:38.309333 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:38.309060    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/494bd9cf-9a4d-4448-8850-f746c50b5c8e-metrics-certs" pod "494bd9cf-9a4d-4448-8850-f746c50b5c8e" (UID: "494bd9cf-9a4d-4448-8850-f746c50b5c8e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:23:38.314173 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:38.314074    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/494bd9cf-9a4d-4448-8850-f746c50b5c8e-openshift-service-ca" pod "494bd9cf-9a4d-4448-8850-f746c50b5c8e" (UID: "494bd9cf-9a4d-4448-8850-f746c50b5c8e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:23:38.331230 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:38.331181    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/494bd9cf-9a4d-4448-8850-f746c50b5c8e-maas-api-tls" pod "494bd9cf-9a4d-4448-8850-f746c50b5c8e" (UID: "494bd9cf-9a4d-4448-8850-f746c50b5c8e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:23:38.332607 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:38.332576    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/494bd9cf-9a4d-4448-8850-f746c50b5c8e-kube-api-access-hbsnz" pod "494bd9cf-9a4d-4448-8850-f746c50b5c8e" (UID: "494bd9cf-9a4d-4448-8850-f746c50b5c8e"). InnerVolumeSpecName "kube-api-access-hbsnz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:23:38.395718 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:38.395683    5304 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/494bd9cf-9a4d-4448-8850-f746c50b5c8e-maas-api-tls\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:23:38.395718 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:38.395718    5304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hbsnz\" (UniqueName: \"kubernetes.io/projected/494bd9cf-9a4d-4448-8850-f746c50b5c8e-kube-api-access-hbsnz\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:23:38.395905 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:38.395732    5304 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/494bd9cf-9a4d-4448-8850-f746c50b5c8e-metrics-certs\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:23:38.395905 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:38.395745    5304 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/494bd9cf-9a4d-4448-8850-f746c50b5c8e-openshift-service-ca\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:23:38.883404 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:38.883370    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-55e4c3-578b56f96d-lmkft"
Nov 02 15:23:38.883605 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:38.883365    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-55e4c3-578b56f96d-lmkft" event={"ID":"494bd9cf-9a4d-4448-8850-f746c50b5c8e","Type":"ContainerDied","Data":"3f822bac681f82093194c1e4bb690c7f4ce489ae2ca379572f0d8386fa96b5b3"}
Nov 02 15:23:38.883605 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:38.883508    5304 scope.go:122] "RemoveContainer" containerID="eab6fd8ca782f752159b380bc059b5ea2125f8eb5edf3fde9493d0a11d2cb00a"
Nov 02 15:23:38.936081 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:38.936035    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-55e4c3-578b56f96d-lmkft"]
Nov 02 15:23:38.946289 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:38.946261    5304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-55e4c3-578b56f96d-lmkft"]
Nov 02 15:23:39.899937 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:39.899904    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-1cdbe6-86f44f7857-x9klg"]
Nov 02 15:23:39.938407 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:39.938375    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-1cdbe6-6ccf98cccf-76jxh"]
Nov 02 15:23:40.122039 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:40.122004    5304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="494bd9cf-9a4d-4448-8850-f746c50b5c8e" path="/var/lib/kubelet/pods/494bd9cf-9a4d-4448-8850-f746c50b5c8e/volumes"
Nov 02 15:23:40.122642 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:40.122623    5304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6057a4f9-4a74-4484-930f-501e9f6e6297" path="/var/lib/kubelet/pods/6057a4f9-4a74-4484-930f-501e9f6e6297/volumes"
Nov 02 15:23:40.123238 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:40.123219    5304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="abefd6e8-6072-45e0-9585-7c16ae3b6b35" path="/var/lib/kubelet/pods/abefd6e8-6072-45e0-9585-7c16ae3b6b35/volumes"
Nov 02 15:23:44.900535 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:44.900457    5304 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-1cdbe6-86f44f7857-x9klg" podUID="1c24b5d7-d4a5-46bb-94f4-22046cc2e27b" containerName="payload-processing" containerID="cri-o://61b174fa9522bfa0df00475401a31621d6807550614d97a3f90c0fe2cebbfe88" gracePeriod=25
Nov 02 15:23:44.939405 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:44.939354    5304 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-1cdbe6-6ccf98cccf-76jxh" podUID="fc17910d-68da-47a3-95c6-441451c71378" containerName="payload-pre-processing" containerID="cri-o://8c4168eecadd00915928f034e3fed0cf2ee9d1a89eb8f40a7df73f4675cb3d0a" gracePeriod=25
Nov 02 15:23:45.237180 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:45.237153    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-1cdbe6-86f44f7857-x9klg"
Nov 02 15:23:45.265801 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:45.265775    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-1cdbe6-6ccf98cccf-76jxh"
Nov 02 15:23:45.346496 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:45.346464    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc17910d-68da-47a3-95c6-441451c71378-kube-api-access-q96nx\" (UniqueName: \"kubernetes.io/projected/fc17910d-68da-47a3-95c6-441451c71378-kube-api-access-q96nx\") pod \"fc17910d-68da-47a3-95c6-441451c71378\" (UID: \"fc17910d-68da-47a3-95c6-441451c71378\") "
Nov 02 15:23:45.346698 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:45.346662    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c24b5d7-d4a5-46bb-94f4-22046cc2e27b-kube-api-access-lv7hr\" (UniqueName: \"kubernetes.io/projected/1c24b5d7-d4a5-46bb-94f4-22046cc2e27b-kube-api-access-lv7hr\") pod \"1c24b5d7-d4a5-46bb-94f4-22046cc2e27b\" (UID: \"1c24b5d7-d4a5-46bb-94f4-22046cc2e27b\") "
Nov 02 15:23:45.346845 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:45.346828    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1c24b5d7-d4a5-46bb-94f4-22046cc2e27b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c24b5d7-d4a5-46bb-94f4-22046cc2e27b-plugins-config-volume\") pod \"1c24b5d7-d4a5-46bb-94f4-22046cc2e27b\" (UID: \"1c24b5d7-d4a5-46bb-94f4-22046cc2e27b\") "
Nov 02 15:23:45.346937 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:45.346922    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fc17910d-68da-47a3-95c6-441451c71378-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc17910d-68da-47a3-95c6-441451c71378-plugins-config-volume\") pod \"fc17910d-68da-47a3-95c6-441451c71378\" (UID: \"fc17910d-68da-47a3-95c6-441451c71378\") "
Nov 02 15:23:45.347488 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:45.347418    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c24b5d7-d4a5-46bb-94f4-22046cc2e27b-plugins-config-volume" pod "1c24b5d7-d4a5-46bb-94f4-22046cc2e27b" (UID: "1c24b5d7-d4a5-46bb-94f4-22046cc2e27b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:23:45.347630 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:45.347560    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc17910d-68da-47a3-95c6-441451c71378-plugins-config-volume" pod "fc17910d-68da-47a3-95c6-441451c71378" (UID: "fc17910d-68da-47a3-95c6-441451c71378"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:23:45.347870 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:45.347845    5304 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c24b5d7-d4a5-46bb-94f4-22046cc2e27b-plugins-config-volume\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:23:45.347956 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:45.347875    5304 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc17910d-68da-47a3-95c6-441451c71378-plugins-config-volume\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:23:45.349047 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:45.349020    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c24b5d7-d4a5-46bb-94f4-22046cc2e27b-kube-api-access-lv7hr" pod "1c24b5d7-d4a5-46bb-94f4-22046cc2e27b" (UID: "1c24b5d7-d4a5-46bb-94f4-22046cc2e27b"). InnerVolumeSpecName "kube-api-access-lv7hr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:23:45.349296 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:45.349274    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc17910d-68da-47a3-95c6-441451c71378-kube-api-access-q96nx" pod "fc17910d-68da-47a3-95c6-441451c71378" (UID: "fc17910d-68da-47a3-95c6-441451c71378"). InnerVolumeSpecName "kube-api-access-q96nx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:23:45.450056 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:45.449973    5304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q96nx\" (UniqueName: \"kubernetes.io/projected/fc17910d-68da-47a3-95c6-441451c71378-kube-api-access-q96nx\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:23:45.450056 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:45.450006    5304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lv7hr\" (UniqueName: \"kubernetes.io/projected/1c24b5d7-d4a5-46bb-94f4-22046cc2e27b-kube-api-access-lv7hr\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:23:45.940614 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:45.940574    5304 generic.go:356] "Generic (PLEG): container finished" podID="fc17910d-68da-47a3-95c6-441451c71378" containerID="8c4168eecadd00915928f034e3fed0cf2ee9d1a89eb8f40a7df73f4675cb3d0a" exitCode=0
Nov 02 15:23:45.941038 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:45.940620    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-1cdbe6-6ccf98cccf-76jxh"
Nov 02 15:23:45.941038 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:45.940664    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-1cdbe6-6ccf98cccf-76jxh" event={"ID":"fc17910d-68da-47a3-95c6-441451c71378","Type":"ContainerDied","Data":"8c4168eecadd00915928f034e3fed0cf2ee9d1a89eb8f40a7df73f4675cb3d0a"}
Nov 02 15:23:45.941038 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:45.940726    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-1cdbe6-6ccf98cccf-76jxh" event={"ID":"fc17910d-68da-47a3-95c6-441451c71378","Type":"ContainerDied","Data":"c8e59f425821351e6d624e7eb82490fff168762995028f178e144335873c4ec7"}
Nov 02 15:23:45.941038 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:45.940754    5304 scope.go:122] "RemoveContainer" containerID="8c4168eecadd00915928f034e3fed0cf2ee9d1a89eb8f40a7df73f4675cb3d0a"
Nov 02 15:23:45.942451 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:45.942430    5304 generic.go:356] "Generic (PLEG): container finished" podID="1c24b5d7-d4a5-46bb-94f4-22046cc2e27b" containerID="61b174fa9522bfa0df00475401a31621d6807550614d97a3f90c0fe2cebbfe88" exitCode=0
Nov 02 15:23:45.942561 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:45.942482    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-1cdbe6-86f44f7857-x9klg"
Nov 02 15:23:45.942561 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:45.942516    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-1cdbe6-86f44f7857-x9klg" event={"ID":"1c24b5d7-d4a5-46bb-94f4-22046cc2e27b","Type":"ContainerDied","Data":"61b174fa9522bfa0df00475401a31621d6807550614d97a3f90c0fe2cebbfe88"}
Nov 02 15:23:45.942630 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:45.942583    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-1cdbe6-86f44f7857-x9klg" event={"ID":"1c24b5d7-d4a5-46bb-94f4-22046cc2e27b","Type":"ContainerDied","Data":"146a8e7dd4ab461fa1986160129f5558bec6e6085c623585a9b5586836ca7ccc"}
Nov 02 15:23:45.953649 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:45.953630    5304 scope.go:122] "RemoveContainer" containerID="8c4168eecadd00915928f034e3fed0cf2ee9d1a89eb8f40a7df73f4675cb3d0a"
Nov 02 15:23:45.953936 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:23:45.953917    5304 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c4168eecadd00915928f034e3fed0cf2ee9d1a89eb8f40a7df73f4675cb3d0a\": container with ID starting with 8c4168eecadd00915928f034e3fed0cf2ee9d1a89eb8f40a7df73f4675cb3d0a not found: ID does not exist" containerID="8c4168eecadd00915928f034e3fed0cf2ee9d1a89eb8f40a7df73f4675cb3d0a"
Nov 02 15:23:45.954007 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:45.953948    5304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c4168eecadd00915928f034e3fed0cf2ee9d1a89eb8f40a7df73f4675cb3d0a"} err="failed to get container status \"8c4168eecadd00915928f034e3fed0cf2ee9d1a89eb8f40a7df73f4675cb3d0a\": rpc error: code = NotFound desc = could not find container \"8c4168eecadd00915928f034e3fed0cf2ee9d1a89eb8f40a7df73f4675cb3d0a\": container with ID starting with 8c4168eecadd00915928f034e3fed0cf2ee9d1a89eb8f40a7df73f4675cb3d0a not found: ID does not exist"
Nov 02 15:23:45.954007 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:45.953974    5304 scope.go:122] "RemoveContainer" containerID="61b174fa9522bfa0df00475401a31621d6807550614d97a3f90c0fe2cebbfe88"
Nov 02 15:23:45.966734 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:45.966712    5304 scope.go:122] "RemoveContainer" containerID="61b174fa9522bfa0df00475401a31621d6807550614d97a3f90c0fe2cebbfe88"
Nov 02 15:23:45.966996 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:23:45.966975    5304 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"61b174fa9522bfa0df00475401a31621d6807550614d97a3f90c0fe2cebbfe88\": container with ID starting with 61b174fa9522bfa0df00475401a31621d6807550614d97a3f90c0fe2cebbfe88 not found: ID does not exist" containerID="61b174fa9522bfa0df00475401a31621d6807550614d97a3f90c0fe2cebbfe88"
Nov 02 15:23:45.967056 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:45.967006    5304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"61b174fa9522bfa0df00475401a31621d6807550614d97a3f90c0fe2cebbfe88"} err="failed to get container status \"61b174fa9522bfa0df00475401a31621d6807550614d97a3f90c0fe2cebbfe88\": rpc error: code = NotFound desc = could not find container \"61b174fa9522bfa0df00475401a31621d6807550614d97a3f90c0fe2cebbfe88\": container with ID starting with 61b174fa9522bfa0df00475401a31621d6807550614d97a3f90c0fe2cebbfe88 not found: ID does not exist"
Nov 02 15:23:46.002791 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:46.002764    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-1cdbe6-86f44f7857-x9klg"]
Nov 02 15:23:46.024096 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:46.024051    5304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-1cdbe6-86f44f7857-x9klg"]
Nov 02 15:23:46.069764 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:46.069728    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-1cdbe6-6ccf98cccf-76jxh"]
Nov 02 15:23:46.079414 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:46.079383    5304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-1cdbe6-6ccf98cccf-76jxh"]
Nov 02 15:23:46.124254 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:46.124221    5304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c24b5d7-d4a5-46bb-94f4-22046cc2e27b" path="/var/lib/kubelet/pods/1c24b5d7-d4a5-46bb-94f4-22046cc2e27b/volumes"
Nov 02 15:23:46.124757 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:46.124742    5304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fc17910d-68da-47a3-95c6-441451c71378" path="/var/lib/kubelet/pods/fc17910d-68da-47a3-95c6-441451c71378/volumes"
Nov 02 15:23:47.149141 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:47.149096    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-bjrnm"]
Nov 02 15:23:47.945425 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:47.945389    5304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-2abeaa-56f4d84db5-5nll8"]
Nov 02 15:23:47.946158 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:47.946120    5304 cpu_manager.go:414] "containerMap: removing container" podUID="1c24b5d7-d4a5-46bb-94f4-22046cc2e27b" containerName="payload-processing"
Nov 02 15:23:47.946158 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:47.946150    5304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c24b5d7-d4a5-46bb-94f4-22046cc2e27b" containerName="payload-processing"
Nov 02 15:23:47.946276 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:47.946174    5304 cpu_manager.go:414] "containerMap: removing container" podUID="6057a4f9-4a74-4484-930f-501e9f6e6297" containerName="istio-proxy"
Nov 02 15:23:47.946276 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:47.946180    5304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6057a4f9-4a74-4484-930f-501e9f6e6297" containerName="istio-proxy"
Nov 02 15:23:47.946276 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:47.946188    5304 cpu_manager.go:414] "containerMap: removing container" podUID="494bd9cf-9a4d-4448-8850-f746c50b5c8e" containerName="maas-api"
Nov 02 15:23:47.946276 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:47.946193    5304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="494bd9cf-9a4d-4448-8850-f746c50b5c8e" containerName="maas-api"
Nov 02 15:23:47.946276 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:47.946219    5304 cpu_manager.go:414] "containerMap: removing container" podUID="294f0d9e-6f89-477c-b882-61af0d1bcb5d" containerName="istio-proxy"
Nov 02 15:23:47.946276 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:47.946224    5304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="294f0d9e-6f89-477c-b882-61af0d1bcb5d" containerName="istio-proxy"
Nov 02 15:23:47.946276 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:47.946233    5304 cpu_manager.go:414] "containerMap: removing container" podUID="294f0d9e-6f89-477c-b882-61af0d1bcb5d" containerName="istio-proxy"
Nov 02 15:23:47.946276 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:47.946238    5304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="294f0d9e-6f89-477c-b882-61af0d1bcb5d" containerName="istio-proxy"
Nov 02 15:23:47.946276 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:47.946249    5304 cpu_manager.go:414] "containerMap: removing container" podUID="fc17910d-68da-47a3-95c6-441451c71378" containerName="payload-pre-processing"
Nov 02 15:23:47.946276 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:47.946254    5304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc17910d-68da-47a3-95c6-441451c71378" containerName="payload-pre-processing"
Nov 02 15:23:47.946276 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:47.946267    5304 cpu_manager.go:414] "containerMap: removing container" podUID="27105db3-fb1c-4da0-8cf0-5f301ac6967f" containerName="istio-proxy"
Nov 02 15:23:47.946276 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:47.946272    5304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="27105db3-fb1c-4da0-8cf0-5f301ac6967f" containerName="istio-proxy"
Nov 02 15:23:47.946276 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:47.946278    5304 cpu_manager.go:414] "containerMap: removing container" podUID="abefd6e8-6072-45e0-9585-7c16ae3b6b35" containerName="istio-proxy"
Nov 02 15:23:47.946276 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:47.946283    5304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="abefd6e8-6072-45e0-9585-7c16ae3b6b35" containerName="istio-proxy"
Nov 02 15:23:47.946716 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:47.946292    5304 cpu_manager.go:414] "containerMap: removing container" podUID="6e1530b5-8c7e-4a83-af8d-67bb9d476b50" containerName="istio-proxy"
Nov 02 15:23:47.946716 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:47.946297    5304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e1530b5-8c7e-4a83-af8d-67bb9d476b50" containerName="istio-proxy"
Nov 02 15:23:47.946716 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:47.946415    5304 memory_manager.go:365] "RemoveStaleState removing state" podUID="6057a4f9-4a74-4484-930f-501e9f6e6297" containerName="istio-proxy"
Nov 02 15:23:47.946716 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:47.946425    5304 memory_manager.go:365] "RemoveStaleState removing state" podUID="494bd9cf-9a4d-4448-8850-f746c50b5c8e" containerName="maas-api"
Nov 02 15:23:47.946716 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:47.946434    5304 memory_manager.go:365] "RemoveStaleState removing state" podUID="294f0d9e-6f89-477c-b882-61af0d1bcb5d" containerName="istio-proxy"
Nov 02 15:23:47.946716 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:47.946440    5304 memory_manager.go:365] "RemoveStaleState removing state" podUID="294f0d9e-6f89-477c-b882-61af0d1bcb5d" containerName="istio-proxy"
Nov 02 15:23:47.946716 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:47.946446    5304 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc17910d-68da-47a3-95c6-441451c71378" containerName="payload-pre-processing"
Nov 02 15:23:47.946716 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:47.946453    5304 memory_manager.go:365] "RemoveStaleState removing state" podUID="abefd6e8-6072-45e0-9585-7c16ae3b6b35" containerName="istio-proxy"
Nov 02 15:23:47.946716 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:47.946458    5304 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e1530b5-8c7e-4a83-af8d-67bb9d476b50" containerName="istio-proxy"
Nov 02 15:23:47.946716 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:47.946464    5304 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c24b5d7-d4a5-46bb-94f4-22046cc2e27b" containerName="payload-processing"
Nov 02 15:23:47.949948 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:47.949929    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2abeaa-56f4d84db5-5nll8"
Nov 02 15:23:47.959669 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:47.959639    5304 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-bjrnm" podUID="97220a55-1205-4d67-90ce-49d6cdef5c43" containerName="istio-proxy" containerID="cri-o://b5e96d2568c382ccc710ae185bbb4990490c1c53625be6b6cec6a6e7ef49cf2e" gracePeriod=30
Nov 02 15:23:47.967921 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:47.967891    5304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-2abeaa-serving-cert\""
Nov 02 15:23:47.984287 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:47.984244    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-2abeaa-56f4d84db5-5nll8"]
Nov 02 15:23:48.118531 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:48.118491    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0cf2360-2112-4333-8fd9-e597685ea1cf-openshift-service-ca\") pod \"maas-api-e2e-inf-b-2abeaa-56f4d84db5-5nll8\" (UID: \"e0cf2360-2112-4333-8fd9-e597685ea1cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2abeaa-56f4d84db5-5nll8"
Nov 02 15:23:48.118681 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:48.118647    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ccd5x\" (UniqueName: \"kubernetes.io/projected/e0cf2360-2112-4333-8fd9-e597685ea1cf-kube-api-access-ccd5x\") pod \"maas-api-e2e-inf-b-2abeaa-56f4d84db5-5nll8\" (UID: \"e0cf2360-2112-4333-8fd9-e597685ea1cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2abeaa-56f4d84db5-5nll8"
Nov 02 15:23:48.118780 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:48.118763    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e0cf2360-2112-4333-8fd9-e597685ea1cf-metrics-certs\") pod \"maas-api-e2e-inf-b-2abeaa-56f4d84db5-5nll8\" (UID: \"e0cf2360-2112-4333-8fd9-e597685ea1cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2abeaa-56f4d84db5-5nll8"
Nov 02 15:23:48.118944 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:48.118928    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e0cf2360-2112-4333-8fd9-e597685ea1cf-maas-api-tls\") pod \"maas-api-e2e-inf-b-2abeaa-56f4d84db5-5nll8\" (UID: \"e0cf2360-2112-4333-8fd9-e597685ea1cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2abeaa-56f4d84db5-5nll8"
Nov 02 15:23:48.220138 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:48.220029    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ccd5x\" (UniqueName: \"kubernetes.io/projected/e0cf2360-2112-4333-8fd9-e597685ea1cf-kube-api-access-ccd5x\") pod \"maas-api-e2e-inf-b-2abeaa-56f4d84db5-5nll8\" (UID: \"e0cf2360-2112-4333-8fd9-e597685ea1cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2abeaa-56f4d84db5-5nll8"
Nov 02 15:23:48.220684 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:48.220247    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e0cf2360-2112-4333-8fd9-e597685ea1cf-metrics-certs\") pod \"maas-api-e2e-inf-b-2abeaa-56f4d84db5-5nll8\" (UID: \"e0cf2360-2112-4333-8fd9-e597685ea1cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2abeaa-56f4d84db5-5nll8"
Nov 02 15:23:48.220684 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:48.220507    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e0cf2360-2112-4333-8fd9-e597685ea1cf-maas-api-tls\") pod \"maas-api-e2e-inf-b-2abeaa-56f4d84db5-5nll8\" (UID: \"e0cf2360-2112-4333-8fd9-e597685ea1cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2abeaa-56f4d84db5-5nll8"
Nov 02 15:23:48.220684 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:48.220582    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0cf2360-2112-4333-8fd9-e597685ea1cf-openshift-service-ca\") pod \"maas-api-e2e-inf-b-2abeaa-56f4d84db5-5nll8\" (UID: \"e0cf2360-2112-4333-8fd9-e597685ea1cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2abeaa-56f4d84db5-5nll8"
Nov 02 15:23:48.221212 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:48.221191    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0cf2360-2112-4333-8fd9-e597685ea1cf-openshift-service-ca\") pod \"maas-api-e2e-inf-b-2abeaa-56f4d84db5-5nll8\" (UID: \"e0cf2360-2112-4333-8fd9-e597685ea1cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2abeaa-56f4d84db5-5nll8"
Nov 02 15:23:48.223803 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:48.223775    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e0cf2360-2112-4333-8fd9-e597685ea1cf-metrics-certs\") pod \"maas-api-e2e-inf-b-2abeaa-56f4d84db5-5nll8\" (UID: \"e0cf2360-2112-4333-8fd9-e597685ea1cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2abeaa-56f4d84db5-5nll8"
Nov 02 15:23:48.224283 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:48.224249    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e0cf2360-2112-4333-8fd9-e597685ea1cf-maas-api-tls\") pod \"maas-api-e2e-inf-b-2abeaa-56f4d84db5-5nll8\" (UID: \"e0cf2360-2112-4333-8fd9-e597685ea1cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2abeaa-56f4d84db5-5nll8"
Nov 02 15:23:48.244956 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:48.244925    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ccd5x\" (UniqueName: \"kubernetes.io/projected/e0cf2360-2112-4333-8fd9-e597685ea1cf-kube-api-access-ccd5x\") pod \"maas-api-e2e-inf-b-2abeaa-56f4d84db5-5nll8\" (UID: \"e0cf2360-2112-4333-8fd9-e597685ea1cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2abeaa-56f4d84db5-5nll8"
Nov 02 15:23:48.264289 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:48.264259    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2abeaa-56f4d84db5-5nll8"
Nov 02 15:23:48.584198 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:48.577345    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-2abeaa-56f4d84db5-5nll8"]
Nov 02 15:23:48.587276 ip-10-0-135-138 kubenswrapper[5304]: W1102 15:23:48.587238    5304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0cf2360_2112_4333_8fd9_e597685ea1cf.slice/crio-405dce5cee62346e6705e77aa5c691dd4839fe3936f0330480762303a102a1b8 WatchSource:0}: Error finding container 405dce5cee62346e6705e77aa5c691dd4839fe3936f0330480762303a102a1b8: Status 404 returned error can't find the container with id 405dce5cee62346e6705e77aa5c691dd4839fe3936f0330480762303a102a1b8
Nov 02 15:23:48.589247 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:48.589229    5304 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 02 15:23:48.968443 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:48.968406    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2abeaa-56f4d84db5-5nll8" event={"ID":"e0cf2360-2112-4333-8fd9-e597685ea1cf","Type":"ContainerStarted","Data":"405dce5cee62346e6705e77aa5c691dd4839fe3936f0330480762303a102a1b8"}
Nov 02 15:23:49.977521 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:49.977484    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2abeaa-56f4d84db5-5nll8" event={"ID":"e0cf2360-2112-4333-8fd9-e597685ea1cf","Type":"ContainerStarted","Data":"88bdbced8ee085ce743909d97f75472a6f0b048b70da3346a80db388a97a91ae"}
Nov 02 15:23:49.978054 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:49.977611    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2abeaa-56f4d84db5-5nll8"
Nov 02 15:23:50.023855 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:50.023801    5304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2abeaa-56f4d84db5-5nll8" podStartSLOduration=2.687120329 podStartE2EDuration="3.023785096s" totalImagesPullingTime="336.664767ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:23:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:23:50.020101552 +0000 UTC m=+1630.013925963" watchObservedRunningTime="2026-11-02 15:23:50.023785096 +0000 UTC m=+1630.017609546"
Nov 02 15:23:53.239538 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:53.239468    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-bjrnm"
Nov 02 15:23:53.311358 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:53.311320    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97220a55-1205-4d67-90ce-49d6cdef5c43-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97220a55-1205-4d67-90ce-49d6cdef5c43-workload-certs\") pod \"97220a55-1205-4d67-90ce-49d6cdef5c43\" (UID: \"97220a55-1205-4d67-90ce-49d6cdef5c43\") "
Nov 02 15:23:53.311574 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:53.311412    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97220a55-1205-4d67-90ce-49d6cdef5c43-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97220a55-1205-4d67-90ce-49d6cdef5c43-workload-socket\") pod \"97220a55-1205-4d67-90ce-49d6cdef5c43\" (UID: \"97220a55-1205-4d67-90ce-49d6cdef5c43\") "
Nov 02 15:23:53.311574 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:53.311543    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/97220a55-1205-4d67-90ce-49d6cdef5c43-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97220a55-1205-4d67-90ce-49d6cdef5c43-istiod-ca-cert\") pod \"97220a55-1205-4d67-90ce-49d6cdef5c43\" (UID: \"97220a55-1205-4d67-90ce-49d6cdef5c43\") "
Nov 02 15:23:53.311709 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:53.311605    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97220a55-1205-4d67-90ce-49d6cdef5c43-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97220a55-1205-4d67-90ce-49d6cdef5c43-istio-envoy\") pod \"97220a55-1205-4d67-90ce-49d6cdef5c43\" (UID: \"97220a55-1205-4d67-90ce-49d6cdef5c43\") "
Nov 02 15:23:53.311709 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:53.311628    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97220a55-1205-4d67-90ce-49d6cdef5c43-workload-certs" pod "97220a55-1205-4d67-90ce-49d6cdef5c43" (UID: "97220a55-1205-4d67-90ce-49d6cdef5c43"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:23:53.311709 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:53.311667    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97220a55-1205-4d67-90ce-49d6cdef5c43-kube-api-access-889zh\" (UniqueName: \"kubernetes.io/projected/97220a55-1205-4d67-90ce-49d6cdef5c43-kube-api-access-889zh\") pod \"97220a55-1205-4d67-90ce-49d6cdef5c43\" (UID: \"97220a55-1205-4d67-90ce-49d6cdef5c43\") "
Nov 02 15:23:53.311709 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:53.311708    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97220a55-1205-4d67-90ce-49d6cdef5c43-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97220a55-1205-4d67-90ce-49d6cdef5c43-istio-data\") pod \"97220a55-1205-4d67-90ce-49d6cdef5c43\" (UID: \"97220a55-1205-4d67-90ce-49d6cdef5c43\") "
Nov 02 15:23:53.311923 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:53.311806    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97220a55-1205-4d67-90ce-49d6cdef5c43-istio-token\" (UniqueName: \"kubernetes.io/projected/97220a55-1205-4d67-90ce-49d6cdef5c43-istio-token\") pod \"97220a55-1205-4d67-90ce-49d6cdef5c43\" (UID: \"97220a55-1205-4d67-90ce-49d6cdef5c43\") "
Nov 02 15:23:53.311923 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:53.311879    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97220a55-1205-4d67-90ce-49d6cdef5c43-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97220a55-1205-4d67-90ce-49d6cdef5c43-credential-socket\") pod \"97220a55-1205-4d67-90ce-49d6cdef5c43\" (UID: \"97220a55-1205-4d67-90ce-49d6cdef5c43\") "
Nov 02 15:23:53.312027 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:53.311928    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/97220a55-1205-4d67-90ce-49d6cdef5c43-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97220a55-1205-4d67-90ce-49d6cdef5c43-istio-podinfo\") pod \"97220a55-1205-4d67-90ce-49d6cdef5c43\" (UID: \"97220a55-1205-4d67-90ce-49d6cdef5c43\") "
Nov 02 15:23:53.312027 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:53.311982    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97220a55-1205-4d67-90ce-49d6cdef5c43-istiod-ca-cert" pod "97220a55-1205-4d67-90ce-49d6cdef5c43" (UID: "97220a55-1205-4d67-90ce-49d6cdef5c43"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:23:53.312294 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:53.312267    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97220a55-1205-4d67-90ce-49d6cdef5c43-workload-socket" pod "97220a55-1205-4d67-90ce-49d6cdef5c43" (UID: "97220a55-1205-4d67-90ce-49d6cdef5c43"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:23:53.312485 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:53.312464    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97220a55-1205-4d67-90ce-49d6cdef5c43-istio-data" pod "97220a55-1205-4d67-90ce-49d6cdef5c43" (UID: "97220a55-1205-4d67-90ce-49d6cdef5c43"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:23:53.312869 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:53.312844    5304 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97220a55-1205-4d67-90ce-49d6cdef5c43-workload-certs\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:23:53.312967 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:53.312874    5304 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97220a55-1205-4d67-90ce-49d6cdef5c43-workload-socket\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:23:53.312967 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:53.312889    5304 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97220a55-1205-4d67-90ce-49d6cdef5c43-istiod-ca-cert\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:23:53.312967 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:53.312902    5304 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97220a55-1205-4d67-90ce-49d6cdef5c43-istio-data\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:23:53.313454 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:53.313429    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97220a55-1205-4d67-90ce-49d6cdef5c43-credential-socket" pod "97220a55-1205-4d67-90ce-49d6cdef5c43" (UID: "97220a55-1205-4d67-90ce-49d6cdef5c43"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:23:53.315230 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:53.315194    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/97220a55-1205-4d67-90ce-49d6cdef5c43-istio-podinfo" pod "97220a55-1205-4d67-90ce-49d6cdef5c43" (UID: "97220a55-1205-4d67-90ce-49d6cdef5c43"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 15:23:53.316546 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:53.316513    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97220a55-1205-4d67-90ce-49d6cdef5c43-istio-envoy" pod "97220a55-1205-4d67-90ce-49d6cdef5c43" (UID: "97220a55-1205-4d67-90ce-49d6cdef5c43"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:23:53.316786 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:53.316572    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97220a55-1205-4d67-90ce-49d6cdef5c43-istio-token" pod "97220a55-1205-4d67-90ce-49d6cdef5c43" (UID: "97220a55-1205-4d67-90ce-49d6cdef5c43"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:23:53.319356 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:53.319324    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97220a55-1205-4d67-90ce-49d6cdef5c43-kube-api-access-889zh" pod "97220a55-1205-4d67-90ce-49d6cdef5c43" (UID: "97220a55-1205-4d67-90ce-49d6cdef5c43"). InnerVolumeSpecName "kube-api-access-889zh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:23:53.415912 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:53.415870    5304 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97220a55-1205-4d67-90ce-49d6cdef5c43-istio-token\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:23:53.415912 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:53.415899    5304 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97220a55-1205-4d67-90ce-49d6cdef5c43-credential-socket\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:23:53.415912 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:53.415913    5304 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97220a55-1205-4d67-90ce-49d6cdef5c43-istio-podinfo\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:23:53.416499 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:53.415924    5304 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97220a55-1205-4d67-90ce-49d6cdef5c43-istio-envoy\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:23:53.416499 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:53.415938    5304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-889zh\" (UniqueName: \"kubernetes.io/projected/97220a55-1205-4d67-90ce-49d6cdef5c43-kube-api-access-889zh\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:23:54.009164 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:54.009095    5304 generic.go:356] "Generic (PLEG): container finished" podID="97220a55-1205-4d67-90ce-49d6cdef5c43" containerID="b5e96d2568c382ccc710ae185bbb4990490c1c53625be6b6cec6a6e7ef49cf2e" exitCode=0
Nov 02 15:23:54.009372 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:54.009175    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-bjrnm"
Nov 02 15:23:54.009372 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:54.009190    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-bjrnm" event={"ID":"97220a55-1205-4d67-90ce-49d6cdef5c43","Type":"ContainerDied","Data":"b5e96d2568c382ccc710ae185bbb4990490c1c53625be6b6cec6a6e7ef49cf2e"}
Nov 02 15:23:54.009372 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:54.009247    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-bjrnm" event={"ID":"97220a55-1205-4d67-90ce-49d6cdef5c43","Type":"ContainerDied","Data":"8c337bfeaa73c82f070e30190adf0055a68a725ab26b07cab83321e43b6fe674"}
Nov 02 15:23:54.009372 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:54.009271    5304 scope.go:122] "RemoveContainer" containerID="b5e96d2568c382ccc710ae185bbb4990490c1c53625be6b6cec6a6e7ef49cf2e"
Nov 02 15:23:54.021089 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:54.021066    5304 scope.go:122] "RemoveContainer" containerID="b5e96d2568c382ccc710ae185bbb4990490c1c53625be6b6cec6a6e7ef49cf2e"
Nov 02 15:23:54.021387 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:23:54.021369    5304 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b5e96d2568c382ccc710ae185bbb4990490c1c53625be6b6cec6a6e7ef49cf2e\": container with ID starting with b5e96d2568c382ccc710ae185bbb4990490c1c53625be6b6cec6a6e7ef49cf2e not found: ID does not exist" containerID="b5e96d2568c382ccc710ae185bbb4990490c1c53625be6b6cec6a6e7ef49cf2e"
Nov 02 15:23:54.021430 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:54.021396    5304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5e96d2568c382ccc710ae185bbb4990490c1c53625be6b6cec6a6e7ef49cf2e"} err="failed to get container status \"b5e96d2568c382ccc710ae185bbb4990490c1c53625be6b6cec6a6e7ef49cf2e\": rpc error: code = NotFound desc = could not find container \"b5e96d2568c382ccc710ae185bbb4990490c1c53625be6b6cec6a6e7ef49cf2e\": container with ID starting with b5e96d2568c382ccc710ae185bbb4990490c1c53625be6b6cec6a6e7ef49cf2e not found: ID does not exist"
Nov 02 15:23:54.074683 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:54.074647    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-bjrnm"]
Nov 02 15:23:54.086683 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:54.086649    5304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-bjrnm"]
Nov 02 15:23:54.120440 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:54.120400    5304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="97220a55-1205-4d67-90ce-49d6cdef5c43" path="/var/lib/kubelet/pods/97220a55-1205-4d67-90ce-49d6cdef5c43/volumes"
Nov 02 15:23:54.800640 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:23:54.800600    5304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k6fjk" containerName="extractor"
Nov 02 15:23:55.990410 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:55.990383    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2abeaa-56f4d84db5-5nll8"
Nov 02 15:23:57.650090 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:23:57.650056    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-815746-5fd777bdf5-m6gk5"]
Nov 02 15:24:02.650499 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:24:02.650448    5304 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-815746-5fd777bdf5-m6gk5" podUID="c7305f17-8a19-42e5-b537-987df8ca3eb0" containerName="payload-pre-processing" containerID="cri-o://e40d2ee9b48d8d55f5cd2a40706a74bcd08d4f6f8e046f643b933ca8b1dc088c" gracePeriod=25
Nov 02 15:24:02.939819 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:24:02.939792    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-815746-5fd777bdf5-m6gk5"
Nov 02 15:24:03.081504 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:24:03.081469    5304 generic.go:356] "Generic (PLEG): container finished" podID="c7305f17-8a19-42e5-b537-987df8ca3eb0" containerID="e40d2ee9b48d8d55f5cd2a40706a74bcd08d4f6f8e046f643b933ca8b1dc088c" exitCode=0
Nov 02 15:24:03.081699 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:24:03.081522    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-815746-5fd777bdf5-m6gk5"
Nov 02 15:24:03.081699 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:24:03.081543    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-815746-5fd777bdf5-m6gk5" event={"ID":"c7305f17-8a19-42e5-b537-987df8ca3eb0","Type":"ContainerDied","Data":"e40d2ee9b48d8d55f5cd2a40706a74bcd08d4f6f8e046f643b933ca8b1dc088c"}
Nov 02 15:24:03.081699 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:24:03.081601    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-815746-5fd777bdf5-m6gk5" event={"ID":"c7305f17-8a19-42e5-b537-987df8ca3eb0","Type":"ContainerDied","Data":"97486d06e8f39bee4fb430a737cf7b225ac6e57c58f344e790b052ca9da88088"}
Nov 02 15:24:03.081699 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:24:03.081629    5304 scope.go:122] "RemoveContainer" containerID="e40d2ee9b48d8d55f5cd2a40706a74bcd08d4f6f8e046f643b933ca8b1dc088c"
Nov 02 15:24:03.091819 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:24:03.091798    5304 scope.go:122] "RemoveContainer" containerID="e40d2ee9b48d8d55f5cd2a40706a74bcd08d4f6f8e046f643b933ca8b1dc088c"
Nov 02 15:24:03.092164 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:24:03.092144    5304 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e40d2ee9b48d8d55f5cd2a40706a74bcd08d4f6f8e046f643b933ca8b1dc088c\": container with ID starting with e40d2ee9b48d8d55f5cd2a40706a74bcd08d4f6f8e046f643b933ca8b1dc088c not found: ID does not exist" containerID="e40d2ee9b48d8d55f5cd2a40706a74bcd08d4f6f8e046f643b933ca8b1dc088c"
Nov 02 15:24:03.092214 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:24:03.092173    5304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e40d2ee9b48d8d55f5cd2a40706a74bcd08d4f6f8e046f643b933ca8b1dc088c"} err="failed to get container status \"e40d2ee9b48d8d55f5cd2a40706a74bcd08d4f6f8e046f643b933ca8b1dc088c\": rpc error: code = NotFound desc = could not find container \"e40d2ee9b48d8d55f5cd2a40706a74bcd08d4f6f8e046f643b933ca8b1dc088c\": container with ID starting with e40d2ee9b48d8d55f5cd2a40706a74bcd08d4f6f8e046f643b933ca8b1dc088c not found: ID does not exist"
Nov 02 15:24:03.096665 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:24:03.096643    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7305f17-8a19-42e5-b537-987df8ca3eb0-kube-api-access-79tbj\" (UniqueName: \"kubernetes.io/projected/c7305f17-8a19-42e5-b537-987df8ca3eb0-kube-api-access-79tbj\") pod \"c7305f17-8a19-42e5-b537-987df8ca3eb0\" (UID: \"c7305f17-8a19-42e5-b537-987df8ca3eb0\") "
Nov 02 15:24:03.096886 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:24:03.096870    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c7305f17-8a19-42e5-b537-987df8ca3eb0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7305f17-8a19-42e5-b537-987df8ca3eb0-plugins-config-volume\") pod \"c7305f17-8a19-42e5-b537-987df8ca3eb0\" (UID: \"c7305f17-8a19-42e5-b537-987df8ca3eb0\") "
Nov 02 15:24:03.097205 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:24:03.097181    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7305f17-8a19-42e5-b537-987df8ca3eb0-plugins-config-volume" pod "c7305f17-8a19-42e5-b537-987df8ca3eb0" (UID: "c7305f17-8a19-42e5-b537-987df8ca3eb0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:24:03.097551 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:24:03.097534    5304 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7305f17-8a19-42e5-b537-987df8ca3eb0-plugins-config-volume\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:24:03.098764 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:24:03.098741    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7305f17-8a19-42e5-b537-987df8ca3eb0-kube-api-access-79tbj" pod "c7305f17-8a19-42e5-b537-987df8ca3eb0" (UID: "c7305f17-8a19-42e5-b537-987df8ca3eb0"). InnerVolumeSpecName "kube-api-access-79tbj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:24:03.198879 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:24:03.198790    5304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-79tbj\" (UniqueName: \"kubernetes.io/projected/c7305f17-8a19-42e5-b537-987df8ca3eb0-kube-api-access-79tbj\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:24:03.434743 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:24:03.434707    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-815746-5fd777bdf5-m6gk5"]
Nov 02 15:24:03.449016 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:24:03.448948    5304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-815746-5fd777bdf5-m6gk5"]
Nov 02 15:24:04.120309 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:24:04.120271    5304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c7305f17-8a19-42e5-b537-987df8ca3eb0" path="/var/lib/kubelet/pods/c7305f17-8a19-42e5-b537-987df8ca3eb0/volumes"
Nov 02 15:24:13.630152 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:24:13.627281    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-624254/simulator-w3-624254-kserve-56866b7699-z47sd" podUID="345c710f-b414-4e62-86c7-d51ff4e5be15" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:24:13.630152 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:24:13.627711    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7ecfe1/distinct-w4-7ecfe1-kserve-6c847c647d-d2jvv" podUID="f1f1ca5e-cca7-4c54-939f-06f095defcdd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded"
Nov 02 15:24:13.695149 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:24:13.694256    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-739bf1/simulator-w2-739bf1-kserve-7c74f85465-ltk4t" podUID="cf6a1ded-28a3-49eb-878b-b4ad77a0bb5c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:24:13.980378 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:24:13.980263    5304 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-5dd8e9/premium-w1-5dd8e9-kserve-67588456fb-j6lph" podUID="4ccc607a-14e4-43ca-8999-78100d4b799f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/health\": context deadline exceeded"
Nov 02 15:24:14.095149 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:24:14.094363    5304 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-1cc1b2/distinct-w5-1cc1b2-kserve-5b7459bc6d-zl8xr" podUID="211ad15d-9234-4f57-b1b9-181dd2d74208" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/health\": context deadline exceeded"
Nov 02 15:24:14.190158 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:24:14.185926    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5113c2/premium-w6-5113c2-kserve-b84945cb5-pc6f8" podUID="9fb3e992-7bb0-497a-a82c-6e396ba241c1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:24:14.530295 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:24:14.530246    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5dd8e9/premium-w1-5dd8e9-kserve-67588456fb-j6lph" podUID="4ccc607a-14e4-43ca-8999-78100d4b799f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded"
Nov 02 15:24:14.530931 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:24:14.530553    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1cc1b2/distinct-w5-1cc1b2-kserve-5b7459bc6d-zl8xr" podUID="211ad15d-9234-4f57-b1b9-181dd2d74208" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:24:24.805971 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:24:24.805929    5304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k6fjk" containerName="extractor"
Nov 02 15:24:40.145022 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:24:40.144988    5304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-f975ed/test-model-f975ed-kserve-54cff7ccbc-9w5br"]
Nov 02 15:24:40.145691 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:24:40.145674    5304 cpu_manager.go:414] "containerMap: removing container" podUID="c7305f17-8a19-42e5-b537-987df8ca3eb0" containerName="payload-pre-processing"
Nov 02 15:24:40.145737 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:24:40.145692    5304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7305f17-8a19-42e5-b537-987df8ca3eb0" containerName="payload-pre-processing"
Nov 02 15:24:40.145737 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:24:40.145726    5304 cpu_manager.go:414] "containerMap: removing container" podUID="97220a55-1205-4d67-90ce-49d6cdef5c43" containerName="istio-proxy"
Nov 02 15:24:40.145737 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:24:40.145731    5304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97220a55-1205-4d67-90ce-49d6cdef5c43" containerName="istio-proxy"
Nov 02 15:24:40.145864 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:24:40.145854    5304 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7305f17-8a19-42e5-b537-987df8ca3eb0" containerName="payload-pre-processing"
Nov 02 15:24:40.145902 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:24:40.145869    5304 memory_manager.go:365] "RemoveStaleState removing state" podUID="97220a55-1205-4d67-90ce-49d6cdef5c43" containerName="istio-proxy"
Nov 02 15:24:40.149500 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:24:40.149479    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-f975ed/test-model-f975ed-kserve-54cff7ccbc-9w5br"
Nov 02 15:24:40.176980 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:24:40.176950    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-f975ed/test-model-f975ed-kserve-54cff7ccbc-9w5br"]
Nov 02 15:24:40.187717 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:24:40.187690    5304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-f975ed\"/\"test-model-f975ed-kserve-self-signed-certs\""
Nov 02 15:24:40.187875 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:24:40.187761    5304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-f975ed\"/\"default-dockercfg-7n942\""
Nov 02 15:24:40.190698 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:24:40.190677    5304 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-f975ed\"/\"openshift-service-ca.crt\""
Nov 02 15:24:40.190861 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:24:40.190776    5304 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-f975ed\"/\"kube-root-ca.crt\""
Nov 02 15:24:40.229407 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:24:40.229373    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rmgjl\" (UniqueName: \"kubernetes.io/projected/02bfbeb4-6e08-4839-b1e3-822a84cbb9c9-kube-api-access-rmgjl\") pod \"test-model-f975ed-kserve-54cff7ccbc-9w5br\" (UID: \"02bfbeb4-6e08-4839-b1e3-822a84cbb9c9\") " pod="ai-tenant-e2e-inf-a-f975ed/test-model-f975ed-kserve-54cff7ccbc-9w5br"
Nov 02 15:24:40.229621 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:24:40.229601    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/02bfbeb4-6e08-4839-b1e3-822a84cbb9c9-model-cache\") pod \"test-model-f975ed-kserve-54cff7ccbc-9w5br\" (UID: \"02bfbeb4-6e08-4839-b1e3-822a84cbb9c9\") " pod="ai-tenant-e2e-inf-a-f975ed/test-model-f975ed-kserve-54cff7ccbc-9w5br"
Nov 02 15:24:40.229726 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:24:40.229708    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/02bfbeb4-6e08-4839-b1e3-822a84cbb9c9-tmp-dir\") pod \"test-model-f975ed-kserve-54cff7ccbc-9w5br\" (UID: \"02bfbeb4-6e08-4839-b1e3-822a84cbb9c9\") " pod="ai-tenant-e2e-inf-a-f975ed/test-model-f975ed-kserve-54cff7ccbc-9w5br"
Nov 02 15:24:40.229841 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:24:40.229826    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/02bfbeb4-6e08-4839-b1e3-822a84cbb9c9-tls-certs\") pod \"test-model-f975ed-kserve-54cff7ccbc-9w5br\" (UID: \"02bfbeb4-6e08-4839-b1e3-822a84cbb9c9\") " pod="ai-tenant-e2e-inf-a-f975ed/test-model-f975ed-kserve-54cff7ccbc-9w5br"
Nov 02 15:24:40.229895 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:24:40.229882    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/02bfbeb4-6e08-4839-b1e3-822a84cbb9c9-dshm\") pod \"test-model-f975ed-kserve-54cff7ccbc-9w5br\" (UID: \"02bfbeb4-6e08-4839-b1e3-822a84cbb9c9\") " pod="ai-tenant-e2e-inf-a-f975ed/test-model-f975ed-kserve-54cff7ccbc-9w5br"
Nov 02 15:24:40.229968 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:24:40.229953    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/02bfbeb4-6e08-4839-b1e3-822a84cbb9c9-home\") pod \"test-model-f975ed-kserve-54cff7ccbc-9w5br\" (UID: \"02bfbeb4-6e08-4839-b1e3-822a84cbb9c9\") " pod="ai-tenant-e2e-inf-a-f975ed/test-model-f975ed-kserve-54cff7ccbc-9w5br"
Nov 02 15:24:40.331639 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:24:40.331596    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/02bfbeb4-6e08-4839-b1e3-822a84cbb9c9-model-cache\") pod \"test-model-f975ed-kserve-54cff7ccbc-9w5br\" (UID: \"02bfbeb4-6e08-4839-b1e3-822a84cbb9c9\") " pod="ai-tenant-e2e-inf-a-f975ed/test-model-f975ed-kserve-54cff7ccbc-9w5br"
Nov 02 15:24:40.331846 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:24:40.331681    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/02bfbeb4-6e08-4839-b1e3-822a84cbb9c9-tmp-dir\") pod \"test-model-f975ed-kserve-54cff7ccbc-9w5br\" (UID: \"02bfbeb4-6e08-4839-b1e3-822a84cbb9c9\") " pod="ai-tenant-e2e-inf-a-f975ed/test-model-f975ed-kserve-54cff7ccbc-9w5br"
Nov 02 15:24:40.331846 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:24:40.331786    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/02bfbeb4-6e08-4839-b1e3-822a84cbb9c9-tls-certs\") pod \"test-model-f975ed-kserve-54cff7ccbc-9w5br\" (UID: \"02bfbeb4-6e08-4839-b1e3-822a84cbb9c9\") " pod="ai-tenant-e2e-inf-a-f975ed/test-model-f975ed-kserve-54cff7ccbc-9w5br"
Nov 02 15:24:40.331963 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:24:40.331860    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/02bfbeb4-6e08-4839-b1e3-822a84cbb9c9-dshm\") pod \"test-model-f975ed-kserve-54cff7ccbc-9w5br\" (UID: \"02bfbeb4-6e08-4839-b1e3-822a84cbb9c9\") " pod="ai-tenant-e2e-inf-a-f975ed/test-model-f975ed-kserve-54cff7ccbc-9w5br"
Nov 02 15:24:40.331963 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:24:40.331933    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/02bfbeb4-6e08-4839-b1e3-822a84cbb9c9-home\") pod \"test-model-f975ed-kserve-54cff7ccbc-9w5br\" (UID: \"02bfbeb4-6e08-4839-b1e3-822a84cbb9c9\") " pod="ai-tenant-e2e-inf-a-f975ed/test-model-f975ed-kserve-54cff7ccbc-9w5br"
Nov 02 15:24:40.332064 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:24:40.332031    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/02bfbeb4-6e08-4839-b1e3-822a84cbb9c9-model-cache\") pod \"test-model-f975ed-kserve-54cff7ccbc-9w5br\" (UID: \"02bfbeb4-6e08-4839-b1e3-822a84cbb9c9\") " pod="ai-tenant-e2e-inf-a-f975ed/test-model-f975ed-kserve-54cff7ccbc-9w5br"
Nov 02 15:24:40.332064 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:24:40.332048    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rmgjl\" (UniqueName: \"kubernetes.io/projected/02bfbeb4-6e08-4839-b1e3-822a84cbb9c9-kube-api-access-rmgjl\") pod \"test-model-f975ed-kserve-54cff7ccbc-9w5br\" (UID: \"02bfbeb4-6e08-4839-b1e3-822a84cbb9c9\") " pod="ai-tenant-e2e-inf-a-f975ed/test-model-f975ed-kserve-54cff7ccbc-9w5br"
Nov 02 15:24:40.332064 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:24:40.332034    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/02bfbeb4-6e08-4839-b1e3-822a84cbb9c9-tmp-dir\") pod \"test-model-f975ed-kserve-54cff7ccbc-9w5br\" (UID: \"02bfbeb4-6e08-4839-b1e3-822a84cbb9c9\") " pod="ai-tenant-e2e-inf-a-f975ed/test-model-f975ed-kserve-54cff7ccbc-9w5br"
Nov 02 15:24:40.332335 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:24:40.332268    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/02bfbeb4-6e08-4839-b1e3-822a84cbb9c9-home\") pod \"test-model-f975ed-kserve-54cff7ccbc-9w5br\" (UID: \"02bfbeb4-6e08-4839-b1e3-822a84cbb9c9\") " pod="ai-tenant-e2e-inf-a-f975ed/test-model-f975ed-kserve-54cff7ccbc-9w5br"
Nov 02 15:24:40.334110 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:24:40.334087    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/02bfbeb4-6e08-4839-b1e3-822a84cbb9c9-dshm\") pod \"test-model-f975ed-kserve-54cff7ccbc-9w5br\" (UID: \"02bfbeb4-6e08-4839-b1e3-822a84cbb9c9\") " pod="ai-tenant-e2e-inf-a-f975ed/test-model-f975ed-kserve-54cff7ccbc-9w5br"
Nov 02 15:24:40.347940 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:24:40.347918    5304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-f975ed\"/\"test-model-f975ed-kserve-self-signed-certs\""
Nov 02 15:24:40.355454 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:24:40.355431    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/02bfbeb4-6e08-4839-b1e3-822a84cbb9c9-tls-certs\") pod \"test-model-f975ed-kserve-54cff7ccbc-9w5br\" (UID: \"02bfbeb4-6e08-4839-b1e3-822a84cbb9c9\") " pod="ai-tenant-e2e-inf-a-f975ed/test-model-f975ed-kserve-54cff7ccbc-9w5br"
Nov 02 15:24:40.360710 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:24:40.360690    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rmgjl\" (UniqueName: \"kubernetes.io/projected/02bfbeb4-6e08-4839-b1e3-822a84cbb9c9-kube-api-access-rmgjl\") pod \"test-model-f975ed-kserve-54cff7ccbc-9w5br\" (UID: \"02bfbeb4-6e08-4839-b1e3-822a84cbb9c9\") " pod="ai-tenant-e2e-inf-a-f975ed/test-model-f975ed-kserve-54cff7ccbc-9w5br"
Nov 02 15:24:40.483562 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:24:40.483471    5304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-f975ed\"/\"default-dockercfg-7n942\""
Nov 02 15:24:40.483562 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:24:40.483511    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-f975ed/test-model-f975ed-kserve-54cff7ccbc-9w5br"
Nov 02 15:24:40.842961 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:24:40.842934    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-f975ed/test-model-f975ed-kserve-54cff7ccbc-9w5br"]
Nov 02 15:24:40.845050 ip-10-0-135-138 kubenswrapper[5304]: W1102 15:24:40.845015    5304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod02bfbeb4_6e08_4839_b1e3_822a84cbb9c9.slice/crio-2651182a9cbb5721bb4421244c6a155fd26be71bf9dde0fc7d6bb7ba393f4bfa WatchSource:0}: Error finding container 2651182a9cbb5721bb4421244c6a155fd26be71bf9dde0fc7d6bb7ba393f4bfa: Status 404 returned error can't find the container with id 2651182a9cbb5721bb4421244c6a155fd26be71bf9dde0fc7d6bb7ba393f4bfa
Nov 02 15:24:41.632667 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:24:41.632624    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-f975ed/test-model-f975ed-kserve-54cff7ccbc-9w5br" event={"ID":"02bfbeb4-6e08-4839-b1e3-822a84cbb9c9","Type":"ContainerStarted","Data":"c23ac83b97b4b70e65e6fcae93bb5b61cbf6d3ea7111cde9b99c4534cb0a5888"}
Nov 02 15:24:41.633203 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:24:41.632674    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-f975ed/test-model-f975ed-kserve-54cff7ccbc-9w5br" event={"ID":"02bfbeb4-6e08-4839-b1e3-822a84cbb9c9","Type":"ContainerStarted","Data":"2651182a9cbb5721bb4421244c6a155fd26be71bf9dde0fc7d6bb7ba393f4bfa"}
Nov 02 15:24:41.678235 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:24:41.678101    5304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-f975ed/test-model-f975ed-kserve-54cff7ccbc-9w5br" podStartSLOduration=1.678087343 podStartE2EDuration="1.678087343s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:24:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:24:41.677048381 +0000 UTC m=+1681.670872801" watchObservedRunningTime="2026-11-02 15:24:41.678087343 +0000 UTC m=+1681.671911752"
Nov 02 15:24:42.483621 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:24:42.483584    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-f975ed/test-model-f975ed-kserve-54cff7ccbc-9w5br"
Nov 02 15:24:50.484105 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:24:50.484065    5304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-f975ed/test-model-f975ed-kserve-54cff7ccbc-9w5br"
Nov 02 15:24:50.501881 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:24:50.501847    5304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-f975ed/test-model-f975ed-kserve-54cff7ccbc-9w5br"
Nov 02 15:24:50.713537 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:24:50.713502    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-f975ed/test-model-f975ed-kserve-54cff7ccbc-9w5br"
Nov 02 15:24:54.800575 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:24:54.800527    5304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k6fjk" containerName="extractor"
Nov 02 15:25:07.261761 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:07.261722    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-db75ff-5d4fbcd75-ddk6r"]
Nov 02 15:25:07.262988 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:07.262945    5304 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-db75ff-5d4fbcd75-ddk6r" podUID="97e9ede2-d4ac-4a06-a126-81bec21c1280" containerName="maas-api" containerID="cri-o://ac8961f10a2319da64f8746d626d78b260ca59204911333db8472854dd3d17ae" gracePeriod=30
Nov 02 15:25:07.910975 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:07.909561    5304 generic.go:356] "Generic (PLEG): container finished" podID="97e9ede2-d4ac-4a06-a126-81bec21c1280" containerID="ac8961f10a2319da64f8746d626d78b260ca59204911333db8472854dd3d17ae" exitCode=0
Nov 02 15:25:07.910975 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:07.909627    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-db75ff-5d4fbcd75-ddk6r" event={"ID":"97e9ede2-d4ac-4a06-a126-81bec21c1280","Type":"ContainerDied","Data":"ac8961f10a2319da64f8746d626d78b260ca59204911333db8472854dd3d17ae"}
Nov 02 15:25:07.920381 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:07.920353    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-db75ff-5d4fbcd75-ddk6r"
Nov 02 15:25:08.006434 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:08.005253    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/97e9ede2-d4ac-4a06-a126-81bec21c1280-metrics-certs\" (UniqueName: \"kubernetes.io/secret/97e9ede2-d4ac-4a06-a126-81bec21c1280-metrics-certs\") pod \"97e9ede2-d4ac-4a06-a126-81bec21c1280\" (UID: \"97e9ede2-d4ac-4a06-a126-81bec21c1280\") "
Nov 02 15:25:08.006434 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:08.005359    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/97e9ede2-d4ac-4a06-a126-81bec21c1280-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/97e9ede2-d4ac-4a06-a126-81bec21c1280-openshift-service-ca\") pod \"97e9ede2-d4ac-4a06-a126-81bec21c1280\" (UID: \"97e9ede2-d4ac-4a06-a126-81bec21c1280\") "
Nov 02 15:25:08.006434 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:08.005485    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97e9ede2-d4ac-4a06-a126-81bec21c1280-kube-api-access-v959h\" (UniqueName: \"kubernetes.io/projected/97e9ede2-d4ac-4a06-a126-81bec21c1280-kube-api-access-v959h\") pod \"97e9ede2-d4ac-4a06-a126-81bec21c1280\" (UID: \"97e9ede2-d4ac-4a06-a126-81bec21c1280\") "
Nov 02 15:25:08.006434 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:08.005717    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/97e9ede2-d4ac-4a06-a126-81bec21c1280-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/97e9ede2-d4ac-4a06-a126-81bec21c1280-maas-api-tls\") pod \"97e9ede2-d4ac-4a06-a126-81bec21c1280\" (UID: \"97e9ede2-d4ac-4a06-a126-81bec21c1280\") "
Nov 02 15:25:08.008846 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:08.008805    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/97e9ede2-d4ac-4a06-a126-81bec21c1280-maas-api-tls" pod "97e9ede2-d4ac-4a06-a126-81bec21c1280" (UID: "97e9ede2-d4ac-4a06-a126-81bec21c1280"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:25:08.011212 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:08.011178    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/97e9ede2-d4ac-4a06-a126-81bec21c1280-metrics-certs" pod "97e9ede2-d4ac-4a06-a126-81bec21c1280" (UID: "97e9ede2-d4ac-4a06-a126-81bec21c1280"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:25:08.011527 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:08.011504    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97e9ede2-d4ac-4a06-a126-81bec21c1280-openshift-service-ca" pod "97e9ede2-d4ac-4a06-a126-81bec21c1280" (UID: "97e9ede2-d4ac-4a06-a126-81bec21c1280"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:25:08.013748 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:08.013719    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97e9ede2-d4ac-4a06-a126-81bec21c1280-kube-api-access-v959h" pod "97e9ede2-d4ac-4a06-a126-81bec21c1280" (UID: "97e9ede2-d4ac-4a06-a126-81bec21c1280"). InnerVolumeSpecName "kube-api-access-v959h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:25:08.108049 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:08.108016    5304 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/97e9ede2-d4ac-4a06-a126-81bec21c1280-maas-api-tls\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:25:08.108049 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:08.108047    5304 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/97e9ede2-d4ac-4a06-a126-81bec21c1280-metrics-certs\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:25:08.108269 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:08.108079    5304 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/97e9ede2-d4ac-4a06-a126-81bec21c1280-openshift-service-ca\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:25:08.108269 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:08.108090    5304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v959h\" (UniqueName: \"kubernetes.io/projected/97e9ede2-d4ac-4a06-a126-81bec21c1280-kube-api-access-v959h\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:25:08.918601 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:08.918568    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-db75ff-5d4fbcd75-ddk6r"
Nov 02 15:25:08.918601 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:08.918580    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-db75ff-5d4fbcd75-ddk6r" event={"ID":"97e9ede2-d4ac-4a06-a126-81bec21c1280","Type":"ContainerDied","Data":"2e7f64ebe52ed6fca8fd03bfd89bd14c8b9080b7db2e065a80bcb642ff5fd8d8"}
Nov 02 15:25:08.919165 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:08.918640    5304 scope.go:122] "RemoveContainer" containerID="ac8961f10a2319da64f8746d626d78b260ca59204911333db8472854dd3d17ae"
Nov 02 15:25:08.981920 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:08.981884    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-db75ff-5d4fbcd75-ddk6r"]
Nov 02 15:25:08.994053 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:08.994020    5304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-db75ff-5d4fbcd75-ddk6r"]
Nov 02 15:25:10.121752 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:10.121717    5304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="97e9ede2-d4ac-4a06-a126-81bec21c1280" path="/var/lib/kubelet/pods/97e9ede2-d4ac-4a06-a126-81bec21c1280/volumes"
Nov 02 15:25:10.834993 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:10.834955    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-4ec02d-8498bbbfd9-gm4hk"]
Nov 02 15:25:14.070332 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:14.070293    5304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-a03e84/test-model-a03e84-kserve-554589b56f-tq4bb"]
Nov 02 15:25:14.071402 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:14.071375    5304 cpu_manager.go:414] "containerMap: removing container" podUID="97e9ede2-d4ac-4a06-a126-81bec21c1280" containerName="maas-api"
Nov 02 15:25:14.071402 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:14.071402    5304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97e9ede2-d4ac-4a06-a126-81bec21c1280" containerName="maas-api"
Nov 02 15:25:14.071631 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:14.071565    5304 memory_manager.go:365] "RemoveStaleState removing state" podUID="97e9ede2-d4ac-4a06-a126-81bec21c1280" containerName="maas-api"
Nov 02 15:25:14.077115 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:14.077089    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-a03e84/test-model-a03e84-kserve-554589b56f-tq4bb"
Nov 02 15:25:14.100796 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:14.100767    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-a03e84/test-model-a03e84-kserve-554589b56f-tq4bb"]
Nov 02 15:25:14.110470 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:14.110439    5304 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-a03e84\"/\"kube-root-ca.crt\""
Nov 02 15:25:14.110470 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:14.110463    5304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-a03e84\"/\"default-dockercfg-92c8b\""
Nov 02 15:25:14.110699 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:14.110551    5304 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-a03e84\"/\"openshift-service-ca.crt\""
Nov 02 15:25:14.110699 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:14.110613    5304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-a03e84\"/\"test-model-a03e84-kserve-self-signed-certs\""
Nov 02 15:25:14.213019 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:14.212980    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7hlhd\" (UniqueName: \"kubernetes.io/projected/917f905c-457e-401b-af26-2af1201b5d28-kube-api-access-7hlhd\") pod \"test-model-a03e84-kserve-554589b56f-tq4bb\" (UID: \"917f905c-457e-401b-af26-2af1201b5d28\") " pod="ai-tenant-e2e-inf-a-a03e84/test-model-a03e84-kserve-554589b56f-tq4bb"
Nov 02 15:25:14.213231 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:14.213095    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/917f905c-457e-401b-af26-2af1201b5d28-dshm\") pod \"test-model-a03e84-kserve-554589b56f-tq4bb\" (UID: \"917f905c-457e-401b-af26-2af1201b5d28\") " pod="ai-tenant-e2e-inf-a-a03e84/test-model-a03e84-kserve-554589b56f-tq4bb"
Nov 02 15:25:14.213231 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:14.213184    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/917f905c-457e-401b-af26-2af1201b5d28-tls-certs\") pod \"test-model-a03e84-kserve-554589b56f-tq4bb\" (UID: \"917f905c-457e-401b-af26-2af1201b5d28\") " pod="ai-tenant-e2e-inf-a-a03e84/test-model-a03e84-kserve-554589b56f-tq4bb"
Nov 02 15:25:14.213231 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:14.213215    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/917f905c-457e-401b-af26-2af1201b5d28-tmp-dir\") pod \"test-model-a03e84-kserve-554589b56f-tq4bb\" (UID: \"917f905c-457e-401b-af26-2af1201b5d28\") " pod="ai-tenant-e2e-inf-a-a03e84/test-model-a03e84-kserve-554589b56f-tq4bb"
Nov 02 15:25:14.213392 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:14.213275    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/917f905c-457e-401b-af26-2af1201b5d28-model-cache\") pod \"test-model-a03e84-kserve-554589b56f-tq4bb\" (UID: \"917f905c-457e-401b-af26-2af1201b5d28\") " pod="ai-tenant-e2e-inf-a-a03e84/test-model-a03e84-kserve-554589b56f-tq4bb"
Nov 02 15:25:14.213456 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:14.213408    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/917f905c-457e-401b-af26-2af1201b5d28-home\") pod \"test-model-a03e84-kserve-554589b56f-tq4bb\" (UID: \"917f905c-457e-401b-af26-2af1201b5d28\") " pod="ai-tenant-e2e-inf-a-a03e84/test-model-a03e84-kserve-554589b56f-tq4bb"
Nov 02 15:25:14.336285 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:14.331774    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/917f905c-457e-401b-af26-2af1201b5d28-home\") pod \"test-model-a03e84-kserve-554589b56f-tq4bb\" (UID: \"917f905c-457e-401b-af26-2af1201b5d28\") " pod="ai-tenant-e2e-inf-a-a03e84/test-model-a03e84-kserve-554589b56f-tq4bb"
Nov 02 15:25:14.348147 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:14.345146    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7hlhd\" (UniqueName: \"kubernetes.io/projected/917f905c-457e-401b-af26-2af1201b5d28-kube-api-access-7hlhd\") pod \"test-model-a03e84-kserve-554589b56f-tq4bb\" (UID: \"917f905c-457e-401b-af26-2af1201b5d28\") " pod="ai-tenant-e2e-inf-a-a03e84/test-model-a03e84-kserve-554589b56f-tq4bb"
Nov 02 15:25:14.348147 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:14.345293    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/917f905c-457e-401b-af26-2af1201b5d28-dshm\") pod \"test-model-a03e84-kserve-554589b56f-tq4bb\" (UID: \"917f905c-457e-401b-af26-2af1201b5d28\") " pod="ai-tenant-e2e-inf-a-a03e84/test-model-a03e84-kserve-554589b56f-tq4bb"
Nov 02 15:25:14.348147 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:14.345344    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/917f905c-457e-401b-af26-2af1201b5d28-tls-certs\") pod \"test-model-a03e84-kserve-554589b56f-tq4bb\" (UID: \"917f905c-457e-401b-af26-2af1201b5d28\") " pod="ai-tenant-e2e-inf-a-a03e84/test-model-a03e84-kserve-554589b56f-tq4bb"
Nov 02 15:25:14.348147 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:14.345371    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/917f905c-457e-401b-af26-2af1201b5d28-tmp-dir\") pod \"test-model-a03e84-kserve-554589b56f-tq4bb\" (UID: \"917f905c-457e-401b-af26-2af1201b5d28\") " pod="ai-tenant-e2e-inf-a-a03e84/test-model-a03e84-kserve-554589b56f-tq4bb"
Nov 02 15:25:14.348147 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:14.345462    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/917f905c-457e-401b-af26-2af1201b5d28-model-cache\") pod \"test-model-a03e84-kserve-554589b56f-tq4bb\" (UID: \"917f905c-457e-401b-af26-2af1201b5d28\") " pod="ai-tenant-e2e-inf-a-a03e84/test-model-a03e84-kserve-554589b56f-tq4bb"
Nov 02 15:25:14.348147 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:14.346422    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/917f905c-457e-401b-af26-2af1201b5d28-home\") pod \"test-model-a03e84-kserve-554589b56f-tq4bb\" (UID: \"917f905c-457e-401b-af26-2af1201b5d28\") " pod="ai-tenant-e2e-inf-a-a03e84/test-model-a03e84-kserve-554589b56f-tq4bb"
Nov 02 15:25:14.348147 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:14.347594    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/917f905c-457e-401b-af26-2af1201b5d28-model-cache\") pod \"test-model-a03e84-kserve-554589b56f-tq4bb\" (UID: \"917f905c-457e-401b-af26-2af1201b5d28\") " pod="ai-tenant-e2e-inf-a-a03e84/test-model-a03e84-kserve-554589b56f-tq4bb"
Nov 02 15:25:14.348147 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:14.347804    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/917f905c-457e-401b-af26-2af1201b5d28-tmp-dir\") pod \"test-model-a03e84-kserve-554589b56f-tq4bb\" (UID: \"917f905c-457e-401b-af26-2af1201b5d28\") " pod="ai-tenant-e2e-inf-a-a03e84/test-model-a03e84-kserve-554589b56f-tq4bb"
Nov 02 15:25:14.367686 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:14.352280    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/917f905c-457e-401b-af26-2af1201b5d28-tls-certs\") pod \"test-model-a03e84-kserve-554589b56f-tq4bb\" (UID: \"917f905c-457e-401b-af26-2af1201b5d28\") " pod="ai-tenant-e2e-inf-a-a03e84/test-model-a03e84-kserve-554589b56f-tq4bb"
Nov 02 15:25:14.367686 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:14.367600    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/917f905c-457e-401b-af26-2af1201b5d28-dshm\") pod \"test-model-a03e84-kserve-554589b56f-tq4bb\" (UID: \"917f905c-457e-401b-af26-2af1201b5d28\") " pod="ai-tenant-e2e-inf-a-a03e84/test-model-a03e84-kserve-554589b56f-tq4bb"
Nov 02 15:25:14.375820 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:14.375785    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7hlhd\" (UniqueName: \"kubernetes.io/projected/917f905c-457e-401b-af26-2af1201b5d28-kube-api-access-7hlhd\") pod \"test-model-a03e84-kserve-554589b56f-tq4bb\" (UID: \"917f905c-457e-401b-af26-2af1201b5d28\") " pod="ai-tenant-e2e-inf-a-a03e84/test-model-a03e84-kserve-554589b56f-tq4bb"
Nov 02 15:25:14.408493 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:14.408197    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-a03e84/test-model-a03e84-kserve-554589b56f-tq4bb"
Nov 02 15:25:14.740969 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:14.740935    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-a03e84/test-model-a03e84-kserve-554589b56f-tq4bb"]
Nov 02 15:25:14.962030 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:14.961992    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-a03e84/test-model-a03e84-kserve-554589b56f-tq4bb" event={"ID":"917f905c-457e-401b-af26-2af1201b5d28","Type":"ContainerStarted","Data":"70310ece1db88ecf64ebb7bde2c00a3726f33ade6fff8e2464b1a2e0b8056332"}
Nov 02 15:25:14.962240 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:14.962042    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-a03e84/test-model-a03e84-kserve-554589b56f-tq4bb" event={"ID":"917f905c-457e-401b-af26-2af1201b5d28","Type":"ContainerStarted","Data":"c712556767213cf9104ac9ade1c46d7caf28bcea3d75f8c5f7859080ba931b0a"}
Nov 02 15:25:15.015092 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:15.014976    5304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-a03e84/test-model-a03e84-kserve-554589b56f-tq4bb" podStartSLOduration=1.014955943 podStartE2EDuration="1.014955943s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:25:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:25:15.011465831 +0000 UTC m=+1715.005290239" watchObservedRunningTime="2026-11-02 15:25:15.014955943 +0000 UTC m=+1715.008780354"
Nov 02 15:25:15.408571 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:15.408538    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-a03e84/test-model-a03e84-kserve-554589b56f-tq4bb"
Nov 02 15:25:15.835512 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:15.835466    5304 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-4ec02d-8498bbbfd9-gm4hk" podUID="803c30b4-dde9-4d5f-9571-004152ae6d80" containerName="payload-processing" containerID="cri-o://8e04479b579a736c3de51c8dac7f6f82b3488f37a3abe1b8bb085fbe87f83de1" gracePeriod=25
Nov 02 15:25:15.972426 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:15.972385    5304 generic.go:356] "Generic (PLEG): container finished" podID="803c30b4-dde9-4d5f-9571-004152ae6d80" containerID="8e04479b579a736c3de51c8dac7f6f82b3488f37a3abe1b8bb085fbe87f83de1" exitCode=0
Nov 02 15:25:15.972614 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:15.972461    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-4ec02d-8498bbbfd9-gm4hk" event={"ID":"803c30b4-dde9-4d5f-9571-004152ae6d80","Type":"ContainerDied","Data":"8e04479b579a736c3de51c8dac7f6f82b3488f37a3abe1b8bb085fbe87f83de1"}
Nov 02 15:25:16.128850 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:16.128823    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-4ec02d-8498bbbfd9-gm4hk"
Nov 02 15:25:16.174617 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:16.174264    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/803c30b4-dde9-4d5f-9571-004152ae6d80-kube-api-access-q2wv9\" (UniqueName: \"kubernetes.io/projected/803c30b4-dde9-4d5f-9571-004152ae6d80-kube-api-access-q2wv9\") pod \"803c30b4-dde9-4d5f-9571-004152ae6d80\" (UID: \"803c30b4-dde9-4d5f-9571-004152ae6d80\") "
Nov 02 15:25:16.174617 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:16.174481    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/803c30b4-dde9-4d5f-9571-004152ae6d80-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/803c30b4-dde9-4d5f-9571-004152ae6d80-plugins-config-volume\") pod \"803c30b4-dde9-4d5f-9571-004152ae6d80\" (UID: \"803c30b4-dde9-4d5f-9571-004152ae6d80\") "
Nov 02 15:25:16.175620 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:16.175593    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/803c30b4-dde9-4d5f-9571-004152ae6d80-plugins-config-volume" pod "803c30b4-dde9-4d5f-9571-004152ae6d80" (UID: "803c30b4-dde9-4d5f-9571-004152ae6d80"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:25:16.178770 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:16.178423    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/803c30b4-dde9-4d5f-9571-004152ae6d80-kube-api-access-q2wv9" pod "803c30b4-dde9-4d5f-9571-004152ae6d80" (UID: "803c30b4-dde9-4d5f-9571-004152ae6d80"). InnerVolumeSpecName "kube-api-access-q2wv9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:25:16.277063 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:16.277023    5304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q2wv9\" (UniqueName: \"kubernetes.io/projected/803c30b4-dde9-4d5f-9571-004152ae6d80-kube-api-access-q2wv9\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:25:16.277063 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:16.277057    5304 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/803c30b4-dde9-4d5f-9571-004152ae6d80-plugins-config-volume\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:25:16.981593 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:16.981564    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-4ec02d-8498bbbfd9-gm4hk"
Nov 02 15:25:16.982084 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:16.981601    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-4ec02d-8498bbbfd9-gm4hk" event={"ID":"803c30b4-dde9-4d5f-9571-004152ae6d80","Type":"ContainerDied","Data":"eced5af9bdd6e9a6c8fd2006777c2e2b068fe3cf2735d08c40cf1749bcd22f23"}
Nov 02 15:25:16.982084 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:16.981670    5304 scope.go:122] "RemoveContainer" containerID="8e04479b579a736c3de51c8dac7f6f82b3488f37a3abe1b8bb085fbe87f83de1"
Nov 02 15:25:17.040454 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:17.040415    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-4ec02d-8498bbbfd9-gm4hk"]
Nov 02 15:25:17.054509 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:17.054483    5304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-4ec02d-8498bbbfd9-gm4hk"]
Nov 02 15:25:18.119850 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:18.119816    5304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="803c30b4-dde9-4d5f-9571-004152ae6d80" path="/var/lib/kubelet/pods/803c30b4-dde9-4d5f-9571-004152ae6d80/volumes"
Nov 02 15:25:24.409297 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:24.409209    5304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-a03e84/test-model-a03e84-kserve-554589b56f-tq4bb"
Nov 02 15:25:24.422754 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:24.422728    5304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-a03e84/test-model-a03e84-kserve-554589b56f-tq4bb"
Nov 02 15:25:24.799729 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:25:24.799644    5304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k6fjk" containerName="extractor"
Nov 02 15:25:25.053364 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:25.053278    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-a03e84/test-model-a03e84-kserve-554589b56f-tq4bb"
Nov 02 15:25:25.434607 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:25.434571    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-db75ff-openshift-default-7558cfb4df-qnbxl"]
Nov 02 15:25:25.435105 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:25.434843    5304 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-db75ff-openshift-default-7558cfb4df-qnbxl" podUID="ba7e1e2d-fe7e-41ab-8fb0-814e76dfb08b" containerName="istio-proxy" containerID="cri-o://9d8770c74296cf691bbfd200950a31905a1098d7b37e6f0cc16a066cd790bafd" gracePeriod=30
Nov 02 15:25:27.690666 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:27.690629    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-815746-openshift-default-74696d484f-9f99s"]
Nov 02 15:25:27.691610 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:27.691576    5304 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-815746-openshift-default-74696d484f-9f99s" podUID="0ecb8e24-cbc7-4808-a2aa-6536d94590b5" containerName="istio-proxy" containerID="cri-o://25bce27ef3f8bd79bed947b2260a48f0b045cfa9200c871de571e2f8e31470f9" gracePeriod=30
Nov 02 15:25:27.907884 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:27.907833    5304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-28f621-openshift-default-7dd8c5c789-jvmsw"]
Nov 02 15:25:27.909115 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:27.909078    5304 cpu_manager.go:414] "containerMap: removing container" podUID="803c30b4-dde9-4d5f-9571-004152ae6d80" containerName="payload-processing"
Nov 02 15:25:27.909329 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:27.909311    5304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="803c30b4-dde9-4d5f-9571-004152ae6d80" containerName="payload-processing"
Nov 02 15:25:27.910009 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:27.909992    5304 memory_manager.go:365] "RemoveStaleState removing state" podUID="803c30b4-dde9-4d5f-9571-004152ae6d80" containerName="payload-processing"
Nov 02 15:25:27.927638 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:27.927574    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-28f621-openshift-default-7dd8c5c789-jvmsw"
Nov 02 15:25:27.957225 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:27.957177    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-28f621-openshift-default-7dd8c5c789-jvmsw"]
Nov 02 15:25:28.019287 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:28.014621    5304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-28f621-openshift-default-dockercfg-qv46v\""
Nov 02 15:25:28.099208 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:28.099171    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4bb2a0e-a2da-44e1-ac19-ee296cfff71c-istio-data\") pod \"e2e-inf-a-28f621-openshift-default-7dd8c5c789-jvmsw\" (UID: \"e4bb2a0e-a2da-44e1-ac19-ee296cfff71c\") " pod="openshift-ingress/e2e-inf-a-28f621-openshift-default-7dd8c5c789-jvmsw"
Nov 02 15:25:28.099392 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:28.099354    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4bb2a0e-a2da-44e1-ac19-ee296cfff71c-istio-podinfo\") pod \"e2e-inf-a-28f621-openshift-default-7dd8c5c789-jvmsw\" (UID: \"e4bb2a0e-a2da-44e1-ac19-ee296cfff71c\") " pod="openshift-ingress/e2e-inf-a-28f621-openshift-default-7dd8c5c789-jvmsw"
Nov 02 15:25:28.099392 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:28.099383    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e4bb2a0e-a2da-44e1-ac19-ee296cfff71c-istio-token\") pod \"e2e-inf-a-28f621-openshift-default-7dd8c5c789-jvmsw\" (UID: \"e4bb2a0e-a2da-44e1-ac19-ee296cfff71c\") " pod="openshift-ingress/e2e-inf-a-28f621-openshift-default-7dd8c5c789-jvmsw"
Nov 02 15:25:28.099502 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:28.099407    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4bb2a0e-a2da-44e1-ac19-ee296cfff71c-istio-envoy\") pod \"e2e-inf-a-28f621-openshift-default-7dd8c5c789-jvmsw\" (UID: \"e4bb2a0e-a2da-44e1-ac19-ee296cfff71c\") " pod="openshift-ingress/e2e-inf-a-28f621-openshift-default-7dd8c5c789-jvmsw"
Nov 02 15:25:28.099589 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:28.099566    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4bb2a0e-a2da-44e1-ac19-ee296cfff71c-workload-socket\") pod \"e2e-inf-a-28f621-openshift-default-7dd8c5c789-jvmsw\" (UID: \"e4bb2a0e-a2da-44e1-ac19-ee296cfff71c\") " pod="openshift-ingress/e2e-inf-a-28f621-openshift-default-7dd8c5c789-jvmsw"
Nov 02 15:25:28.099638 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:28.099610    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pw7kj\" (UniqueName: \"kubernetes.io/projected/e4bb2a0e-a2da-44e1-ac19-ee296cfff71c-kube-api-access-pw7kj\") pod \"e2e-inf-a-28f621-openshift-default-7dd8c5c789-jvmsw\" (UID: \"e4bb2a0e-a2da-44e1-ac19-ee296cfff71c\") " pod="openshift-ingress/e2e-inf-a-28f621-openshift-default-7dd8c5c789-jvmsw"
Nov 02 15:25:28.099638 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:28.099633    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4bb2a0e-a2da-44e1-ac19-ee296cfff71c-istiod-ca-cert\") pod \"e2e-inf-a-28f621-openshift-default-7dd8c5c789-jvmsw\" (UID: \"e4bb2a0e-a2da-44e1-ac19-ee296cfff71c\") " pod="openshift-ingress/e2e-inf-a-28f621-openshift-default-7dd8c5c789-jvmsw"
Nov 02 15:25:28.099735 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:28.099724    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4bb2a0e-a2da-44e1-ac19-ee296cfff71c-workload-certs\") pod \"e2e-inf-a-28f621-openshift-default-7dd8c5c789-jvmsw\" (UID: \"e4bb2a0e-a2da-44e1-ac19-ee296cfff71c\") " pod="openshift-ingress/e2e-inf-a-28f621-openshift-default-7dd8c5c789-jvmsw"
Nov 02 15:25:28.099773 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:28.099749    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4bb2a0e-a2da-44e1-ac19-ee296cfff71c-credential-socket\") pod \"e2e-inf-a-28f621-openshift-default-7dd8c5c789-jvmsw\" (UID: \"e4bb2a0e-a2da-44e1-ac19-ee296cfff71c\") " pod="openshift-ingress/e2e-inf-a-28f621-openshift-default-7dd8c5c789-jvmsw"
Nov 02 15:25:28.200912 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:28.200818    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4bb2a0e-a2da-44e1-ac19-ee296cfff71c-workload-socket\") pod \"e2e-inf-a-28f621-openshift-default-7dd8c5c789-jvmsw\" (UID: \"e4bb2a0e-a2da-44e1-ac19-ee296cfff71c\") " pod="openshift-ingress/e2e-inf-a-28f621-openshift-default-7dd8c5c789-jvmsw"
Nov 02 15:25:28.200912 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:28.200865    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pw7kj\" (UniqueName: \"kubernetes.io/projected/e4bb2a0e-a2da-44e1-ac19-ee296cfff71c-kube-api-access-pw7kj\") pod \"e2e-inf-a-28f621-openshift-default-7dd8c5c789-jvmsw\" (UID: \"e4bb2a0e-a2da-44e1-ac19-ee296cfff71c\") " pod="openshift-ingress/e2e-inf-a-28f621-openshift-default-7dd8c5c789-jvmsw"
Nov 02 15:25:28.200912 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:28.200892    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4bb2a0e-a2da-44e1-ac19-ee296cfff71c-istiod-ca-cert\") pod \"e2e-inf-a-28f621-openshift-default-7dd8c5c789-jvmsw\" (UID: \"e4bb2a0e-a2da-44e1-ac19-ee296cfff71c\") " pod="openshift-ingress/e2e-inf-a-28f621-openshift-default-7dd8c5c789-jvmsw"
Nov 02 15:25:28.201238 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:28.200923    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4bb2a0e-a2da-44e1-ac19-ee296cfff71c-workload-certs\") pod \"e2e-inf-a-28f621-openshift-default-7dd8c5c789-jvmsw\" (UID: \"e4bb2a0e-a2da-44e1-ac19-ee296cfff71c\") " pod="openshift-ingress/e2e-inf-a-28f621-openshift-default-7dd8c5c789-jvmsw"
Nov 02 15:25:28.201238 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:28.200942    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4bb2a0e-a2da-44e1-ac19-ee296cfff71c-credential-socket\") pod \"e2e-inf-a-28f621-openshift-default-7dd8c5c789-jvmsw\" (UID: \"e4bb2a0e-a2da-44e1-ac19-ee296cfff71c\") " pod="openshift-ingress/e2e-inf-a-28f621-openshift-default-7dd8c5c789-jvmsw"
Nov 02 15:25:28.201338 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:28.201261    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4bb2a0e-a2da-44e1-ac19-ee296cfff71c-workload-socket\") pod \"e2e-inf-a-28f621-openshift-default-7dd8c5c789-jvmsw\" (UID: \"e4bb2a0e-a2da-44e1-ac19-ee296cfff71c\") " pod="openshift-ingress/e2e-inf-a-28f621-openshift-default-7dd8c5c789-jvmsw"
Nov 02 15:25:28.201402 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:28.201377    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4bb2a0e-a2da-44e1-ac19-ee296cfff71c-workload-certs\") pod \"e2e-inf-a-28f621-openshift-default-7dd8c5c789-jvmsw\" (UID: \"e4bb2a0e-a2da-44e1-ac19-ee296cfff71c\") " pod="openshift-ingress/e2e-inf-a-28f621-openshift-default-7dd8c5c789-jvmsw"
Nov 02 15:25:28.201442 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:28.201417    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4bb2a0e-a2da-44e1-ac19-ee296cfff71c-credential-socket\") pod \"e2e-inf-a-28f621-openshift-default-7dd8c5c789-jvmsw\" (UID: \"e4bb2a0e-a2da-44e1-ac19-ee296cfff71c\") " pod="openshift-ingress/e2e-inf-a-28f621-openshift-default-7dd8c5c789-jvmsw"
Nov 02 15:25:28.201477 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:28.201468    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4bb2a0e-a2da-44e1-ac19-ee296cfff71c-istio-data\") pod \"e2e-inf-a-28f621-openshift-default-7dd8c5c789-jvmsw\" (UID: \"e4bb2a0e-a2da-44e1-ac19-ee296cfff71c\") " pod="openshift-ingress/e2e-inf-a-28f621-openshift-default-7dd8c5c789-jvmsw"
Nov 02 15:25:28.201616 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:28.201598    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4bb2a0e-a2da-44e1-ac19-ee296cfff71c-istio-podinfo\") pod \"e2e-inf-a-28f621-openshift-default-7dd8c5c789-jvmsw\" (UID: \"e4bb2a0e-a2da-44e1-ac19-ee296cfff71c\") " pod="openshift-ingress/e2e-inf-a-28f621-openshift-default-7dd8c5c789-jvmsw"
Nov 02 15:25:28.201671 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:28.201633    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e4bb2a0e-a2da-44e1-ac19-ee296cfff71c-istio-token\") pod \"e2e-inf-a-28f621-openshift-default-7dd8c5c789-jvmsw\" (UID: \"e4bb2a0e-a2da-44e1-ac19-ee296cfff71c\") " pod="openshift-ingress/e2e-inf-a-28f621-openshift-default-7dd8c5c789-jvmsw"
Nov 02 15:25:28.201671 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:28.201666    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4bb2a0e-a2da-44e1-ac19-ee296cfff71c-istio-envoy\") pod \"e2e-inf-a-28f621-openshift-default-7dd8c5c789-jvmsw\" (UID: \"e4bb2a0e-a2da-44e1-ac19-ee296cfff71c\") " pod="openshift-ingress/e2e-inf-a-28f621-openshift-default-7dd8c5c789-jvmsw"
Nov 02 15:25:28.201777 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:28.201758    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4bb2a0e-a2da-44e1-ac19-ee296cfff71c-istio-data\") pod \"e2e-inf-a-28f621-openshift-default-7dd8c5c789-jvmsw\" (UID: \"e4bb2a0e-a2da-44e1-ac19-ee296cfff71c\") " pod="openshift-ingress/e2e-inf-a-28f621-openshift-default-7dd8c5c789-jvmsw"
Nov 02 15:25:28.201836 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:28.201771    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4bb2a0e-a2da-44e1-ac19-ee296cfff71c-istiod-ca-cert\") pod \"e2e-inf-a-28f621-openshift-default-7dd8c5c789-jvmsw\" (UID: \"e4bb2a0e-a2da-44e1-ac19-ee296cfff71c\") " pod="openshift-ingress/e2e-inf-a-28f621-openshift-default-7dd8c5c789-jvmsw"
Nov 02 15:25:28.203909 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:28.203880    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4bb2a0e-a2da-44e1-ac19-ee296cfff71c-istio-envoy\") pod \"e2e-inf-a-28f621-openshift-default-7dd8c5c789-jvmsw\" (UID: \"e4bb2a0e-a2da-44e1-ac19-ee296cfff71c\") " pod="openshift-ingress/e2e-inf-a-28f621-openshift-default-7dd8c5c789-jvmsw"
Nov 02 15:25:28.204284 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:28.204263    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4bb2a0e-a2da-44e1-ac19-ee296cfff71c-istio-podinfo\") pod \"e2e-inf-a-28f621-openshift-default-7dd8c5c789-jvmsw\" (UID: \"e4bb2a0e-a2da-44e1-ac19-ee296cfff71c\") " pod="openshift-ingress/e2e-inf-a-28f621-openshift-default-7dd8c5c789-jvmsw"
Nov 02 15:25:28.226823 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:28.226781    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e4bb2a0e-a2da-44e1-ac19-ee296cfff71c-istio-token\") pod \"e2e-inf-a-28f621-openshift-default-7dd8c5c789-jvmsw\" (UID: \"e4bb2a0e-a2da-44e1-ac19-ee296cfff71c\") " pod="openshift-ingress/e2e-inf-a-28f621-openshift-default-7dd8c5c789-jvmsw"
Nov 02 15:25:28.226964 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:28.226830    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pw7kj\" (UniqueName: \"kubernetes.io/projected/e4bb2a0e-a2da-44e1-ac19-ee296cfff71c-kube-api-access-pw7kj\") pod \"e2e-inf-a-28f621-openshift-default-7dd8c5c789-jvmsw\" (UID: \"e4bb2a0e-a2da-44e1-ac19-ee296cfff71c\") " pod="openshift-ingress/e2e-inf-a-28f621-openshift-default-7dd8c5c789-jvmsw"
Nov 02 15:25:28.259211 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:28.259156    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-28f621-openshift-default-7dd8c5c789-jvmsw"
Nov 02 15:25:28.591552 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:28.591519    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-28f621-openshift-default-7dd8c5c789-jvmsw"]
Nov 02 15:25:28.592037 ip-10-0-135-138 kubenswrapper[5304]: W1102 15:25:28.592010    5304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4bb2a0e_a2da_44e1_ac19_ee296cfff71c.slice/crio-f0dc7449945750645e479080b03b5dabdbdf3eab1f2ec3f5e06d49702bb5e9b1 WatchSource:0}: Error finding container f0dc7449945750645e479080b03b5dabdbdf3eab1f2ec3f5e06d49702bb5e9b1: Status 404 returned error can't find the container with id f0dc7449945750645e479080b03b5dabdbdf3eab1f2ec3f5e06d49702bb5e9b1
Nov 02 15:25:28.595544 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:28.595497    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:25:28.595672 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:28.595594    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:25:28.595672 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:28.595643    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:25:28.830836 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:28.830797    5304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-2f8b7c-openshift-default-769cb98f9-xm9dm"]
Nov 02 15:25:28.836496 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:28.836407    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-2f8b7c-openshift-default-769cb98f9-xm9dm"
Nov 02 15:25:28.857731 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:28.857702    5304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-2f8b7c-openshift-default-dockercfg-448bd\""
Nov 02 15:25:28.871669 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:28.871628    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-2f8b7c-openshift-default-769cb98f9-xm9dm"]
Nov 02 15:25:28.909849 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:28.909813    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f8cf102-3f23-49fe-804f-694c50509462-workload-certs\") pod \"e2e-inf-a-2f8b7c-openshift-default-769cb98f9-xm9dm\" (UID: \"6f8cf102-3f23-49fe-804f-694c50509462\") " pod="openshift-ingress/e2e-inf-a-2f8b7c-openshift-default-769cb98f9-xm9dm"
Nov 02 15:25:28.909849 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:28.909859    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f8cf102-3f23-49fe-804f-694c50509462-istio-podinfo\") pod \"e2e-inf-a-2f8b7c-openshift-default-769cb98f9-xm9dm\" (UID: \"6f8cf102-3f23-49fe-804f-694c50509462\") " pod="openshift-ingress/e2e-inf-a-2f8b7c-openshift-default-769cb98f9-xm9dm"
Nov 02 15:25:28.910140 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:28.910026    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s4xr8\" (UniqueName: \"kubernetes.io/projected/6f8cf102-3f23-49fe-804f-694c50509462-kube-api-access-s4xr8\") pod \"e2e-inf-a-2f8b7c-openshift-default-769cb98f9-xm9dm\" (UID: \"6f8cf102-3f23-49fe-804f-694c50509462\") " pod="openshift-ingress/e2e-inf-a-2f8b7c-openshift-default-769cb98f9-xm9dm"
Nov 02 15:25:28.910230 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:28.910204    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f8cf102-3f23-49fe-804f-694c50509462-istiod-ca-cert\") pod \"e2e-inf-a-2f8b7c-openshift-default-769cb98f9-xm9dm\" (UID: \"6f8cf102-3f23-49fe-804f-694c50509462\") " pod="openshift-ingress/e2e-inf-a-2f8b7c-openshift-default-769cb98f9-xm9dm"
Nov 02 15:25:28.910326 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:28.910308    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f8cf102-3f23-49fe-804f-694c50509462-istio-data\") pod \"e2e-inf-a-2f8b7c-openshift-default-769cb98f9-xm9dm\" (UID: \"6f8cf102-3f23-49fe-804f-694c50509462\") " pod="openshift-ingress/e2e-inf-a-2f8b7c-openshift-default-769cb98f9-xm9dm"
Nov 02 15:25:28.910412 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:28.910396    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f8cf102-3f23-49fe-804f-694c50509462-credential-socket\") pod \"e2e-inf-a-2f8b7c-openshift-default-769cb98f9-xm9dm\" (UID: \"6f8cf102-3f23-49fe-804f-694c50509462\") " pod="openshift-ingress/e2e-inf-a-2f8b7c-openshift-default-769cb98f9-xm9dm"
Nov 02 15:25:28.910459 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:28.910449    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f8cf102-3f23-49fe-804f-694c50509462-istio-envoy\") pod \"e2e-inf-a-2f8b7c-openshift-default-769cb98f9-xm9dm\" (UID: \"6f8cf102-3f23-49fe-804f-694c50509462\") " pod="openshift-ingress/e2e-inf-a-2f8b7c-openshift-default-769cb98f9-xm9dm"
Nov 02 15:25:28.910580 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:28.910561    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f8cf102-3f23-49fe-804f-694c50509462-workload-socket\") pod \"e2e-inf-a-2f8b7c-openshift-default-769cb98f9-xm9dm\" (UID: \"6f8cf102-3f23-49fe-804f-694c50509462\") " pod="openshift-ingress/e2e-inf-a-2f8b7c-openshift-default-769cb98f9-xm9dm"
Nov 02 15:25:28.910641 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:28.910605    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f8cf102-3f23-49fe-804f-694c50509462-istio-token\") pod \"e2e-inf-a-2f8b7c-openshift-default-769cb98f9-xm9dm\" (UID: \"6f8cf102-3f23-49fe-804f-694c50509462\") " pod="openshift-ingress/e2e-inf-a-2f8b7c-openshift-default-769cb98f9-xm9dm"
Nov 02 15:25:29.018146 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:29.017589    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f8cf102-3f23-49fe-804f-694c50509462-workload-certs\") pod \"e2e-inf-a-2f8b7c-openshift-default-769cb98f9-xm9dm\" (UID: \"6f8cf102-3f23-49fe-804f-694c50509462\") " pod="openshift-ingress/e2e-inf-a-2f8b7c-openshift-default-769cb98f9-xm9dm"
Nov 02 15:25:29.018146 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:29.017667    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f8cf102-3f23-49fe-804f-694c50509462-istio-podinfo\") pod \"e2e-inf-a-2f8b7c-openshift-default-769cb98f9-xm9dm\" (UID: \"6f8cf102-3f23-49fe-804f-694c50509462\") " pod="openshift-ingress/e2e-inf-a-2f8b7c-openshift-default-769cb98f9-xm9dm"
Nov 02 15:25:29.018146 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:29.017801    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s4xr8\" (UniqueName: \"kubernetes.io/projected/6f8cf102-3f23-49fe-804f-694c50509462-kube-api-access-s4xr8\") pod \"e2e-inf-a-2f8b7c-openshift-default-769cb98f9-xm9dm\" (UID: \"6f8cf102-3f23-49fe-804f-694c50509462\") " pod="openshift-ingress/e2e-inf-a-2f8b7c-openshift-default-769cb98f9-xm9dm"
Nov 02 15:25:29.018146 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:29.017875    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f8cf102-3f23-49fe-804f-694c50509462-istiod-ca-cert\") pod \"e2e-inf-a-2f8b7c-openshift-default-769cb98f9-xm9dm\" (UID: \"6f8cf102-3f23-49fe-804f-694c50509462\") " pod="openshift-ingress/e2e-inf-a-2f8b7c-openshift-default-769cb98f9-xm9dm"
Nov 02 15:25:29.018146 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:29.017947    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f8cf102-3f23-49fe-804f-694c50509462-istio-data\") pod \"e2e-inf-a-2f8b7c-openshift-default-769cb98f9-xm9dm\" (UID: \"6f8cf102-3f23-49fe-804f-694c50509462\") " pod="openshift-ingress/e2e-inf-a-2f8b7c-openshift-default-769cb98f9-xm9dm"
Nov 02 15:25:29.018146 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:29.018043    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f8cf102-3f23-49fe-804f-694c50509462-credential-socket\") pod \"e2e-inf-a-2f8b7c-openshift-default-769cb98f9-xm9dm\" (UID: \"6f8cf102-3f23-49fe-804f-694c50509462\") " pod="openshift-ingress/e2e-inf-a-2f8b7c-openshift-default-769cb98f9-xm9dm"
Nov 02 15:25:29.018146 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:29.018102    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f8cf102-3f23-49fe-804f-694c50509462-istio-envoy\") pod \"e2e-inf-a-2f8b7c-openshift-default-769cb98f9-xm9dm\" (UID: \"6f8cf102-3f23-49fe-804f-694c50509462\") " pod="openshift-ingress/e2e-inf-a-2f8b7c-openshift-default-769cb98f9-xm9dm"
Nov 02 15:25:29.018687 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:29.018195    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f8cf102-3f23-49fe-804f-694c50509462-workload-socket\") pod \"e2e-inf-a-2f8b7c-openshift-default-769cb98f9-xm9dm\" (UID: \"6f8cf102-3f23-49fe-804f-694c50509462\") " pod="openshift-ingress/e2e-inf-a-2f8b7c-openshift-default-769cb98f9-xm9dm"
Nov 02 15:25:29.018687 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:29.018235    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f8cf102-3f23-49fe-804f-694c50509462-istio-token\") pod \"e2e-inf-a-2f8b7c-openshift-default-769cb98f9-xm9dm\" (UID: \"6f8cf102-3f23-49fe-804f-694c50509462\") " pod="openshift-ingress/e2e-inf-a-2f8b7c-openshift-default-769cb98f9-xm9dm"
Nov 02 15:25:29.022144 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:29.019027    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f8cf102-3f23-49fe-804f-694c50509462-workload-certs\") pod \"e2e-inf-a-2f8b7c-openshift-default-769cb98f9-xm9dm\" (UID: \"6f8cf102-3f23-49fe-804f-694c50509462\") " pod="openshift-ingress/e2e-inf-a-2f8b7c-openshift-default-769cb98f9-xm9dm"
Nov 02 15:25:29.030154 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:29.023935    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f8cf102-3f23-49fe-804f-694c50509462-istio-data\") pod \"e2e-inf-a-2f8b7c-openshift-default-769cb98f9-xm9dm\" (UID: \"6f8cf102-3f23-49fe-804f-694c50509462\") " pod="openshift-ingress/e2e-inf-a-2f8b7c-openshift-default-769cb98f9-xm9dm"
Nov 02 15:25:29.030154 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:29.024923    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f8cf102-3f23-49fe-804f-694c50509462-istiod-ca-cert\") pod \"e2e-inf-a-2f8b7c-openshift-default-769cb98f9-xm9dm\" (UID: \"6f8cf102-3f23-49fe-804f-694c50509462\") " pod="openshift-ingress/e2e-inf-a-2f8b7c-openshift-default-769cb98f9-xm9dm"
Nov 02 15:25:29.052151 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:29.037546    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f8cf102-3f23-49fe-804f-694c50509462-workload-socket\") pod \"e2e-inf-a-2f8b7c-openshift-default-769cb98f9-xm9dm\" (UID: \"6f8cf102-3f23-49fe-804f-694c50509462\") " pod="openshift-ingress/e2e-inf-a-2f8b7c-openshift-default-769cb98f9-xm9dm"
Nov 02 15:25:29.052151 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:29.037824    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f8cf102-3f23-49fe-804f-694c50509462-credential-socket\") pod \"e2e-inf-a-2f8b7c-openshift-default-769cb98f9-xm9dm\" (UID: \"6f8cf102-3f23-49fe-804f-694c50509462\") " pod="openshift-ingress/e2e-inf-a-2f8b7c-openshift-default-769cb98f9-xm9dm"
Nov 02 15:25:29.052151 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:29.046895    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f8cf102-3f23-49fe-804f-694c50509462-istio-podinfo\") pod \"e2e-inf-a-2f8b7c-openshift-default-769cb98f9-xm9dm\" (UID: \"6f8cf102-3f23-49fe-804f-694c50509462\") " pod="openshift-ingress/e2e-inf-a-2f8b7c-openshift-default-769cb98f9-xm9dm"
Nov 02 15:25:29.052151 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:29.047573    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f8cf102-3f23-49fe-804f-694c50509462-istio-envoy\") pod \"e2e-inf-a-2f8b7c-openshift-default-769cb98f9-xm9dm\" (UID: \"6f8cf102-3f23-49fe-804f-694c50509462\") " pod="openshift-ingress/e2e-inf-a-2f8b7c-openshift-default-769cb98f9-xm9dm"
Nov 02 15:25:29.064155 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:29.059734    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f8cf102-3f23-49fe-804f-694c50509462-istio-token\") pod \"e2e-inf-a-2f8b7c-openshift-default-769cb98f9-xm9dm\" (UID: \"6f8cf102-3f23-49fe-804f-694c50509462\") " pod="openshift-ingress/e2e-inf-a-2f8b7c-openshift-default-769cb98f9-xm9dm"
Nov 02 15:25:29.068146 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:29.065864    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s4xr8\" (UniqueName: \"kubernetes.io/projected/6f8cf102-3f23-49fe-804f-694c50509462-kube-api-access-s4xr8\") pod \"e2e-inf-a-2f8b7c-openshift-default-769cb98f9-xm9dm\" (UID: \"6f8cf102-3f23-49fe-804f-694c50509462\") " pod="openshift-ingress/e2e-inf-a-2f8b7c-openshift-default-769cb98f9-xm9dm"
Nov 02 15:25:29.120205 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:29.116694    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-28f621-openshift-default-7dd8c5c789-jvmsw" event={"ID":"e4bb2a0e-a2da-44e1-ac19-ee296cfff71c","Type":"ContainerStarted","Data":"7e4a3e30cfa510238243eb2e994d8e6d8236791608cbebd8fb72a14a3396aa63"}
Nov 02 15:25:29.120205 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:29.116765    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-28f621-openshift-default-7dd8c5c789-jvmsw" event={"ID":"e4bb2a0e-a2da-44e1-ac19-ee296cfff71c","Type":"ContainerStarted","Data":"f0dc7449945750645e479080b03b5dabdbdf3eab1f2ec3f5e06d49702bb5e9b1"}
Nov 02 15:25:29.181213 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:29.165458    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-2f8b7c-openshift-default-769cb98f9-xm9dm"
Nov 02 15:25:29.181213 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:29.179838    5304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-28f621-openshift-default-7dd8c5c789-jvmsw" podStartSLOduration=2.179817071 podStartE2EDuration="2.179817071s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:25:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:25:29.167225809 +0000 UTC m=+1729.161050219" watchObservedRunningTime="2026-11-02 15:25:29.179817071 +0000 UTC m=+1729.173641482"
Nov 02 15:25:29.259942 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:29.259905    5304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-28f621-openshift-default-7dd8c5c789-jvmsw"
Nov 02 15:25:29.261382 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:29.261356    5304 patch_prober.go:29] interesting pod/e2e-inf-a-28f621-openshift-default-7dd8c5c789-jvmsw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" start-of-body=
Nov 02 15:25:29.261531 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:29.261420    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-28f621-openshift-default-7dd8c5c789-jvmsw" podUID="e4bb2a0e-a2da-44e1-ac19-ee296cfff71c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused"
Nov 02 15:25:29.524847 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:29.524818    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-2f8b7c-openshift-default-769cb98f9-xm9dm"]
Nov 02 15:25:29.526971 ip-10-0-135-138 kubenswrapper[5304]: W1102 15:25:29.526936    5304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f8cf102_3f23_49fe_804f_694c50509462.slice/crio-111e895eb5546fb9fc2557c767ac5a6399f9bb0f38c5bb7254e4e575725d44de WatchSource:0}: Error finding container 111e895eb5546fb9fc2557c767ac5a6399f9bb0f38c5bb7254e4e575725d44de: Status 404 returned error can't find the container with id 111e895eb5546fb9fc2557c767ac5a6399f9bb0f38c5bb7254e4e575725d44de
Nov 02 15:25:29.530839 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:29.530798    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:25:29.530933 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:29.530876    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:25:29.530933 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:29.530908    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:25:30.127326 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:30.127288    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-2f8b7c-openshift-default-769cb98f9-xm9dm" event={"ID":"6f8cf102-3f23-49fe-804f-694c50509462","Type":"ContainerStarted","Data":"21a548dc52c0aa6d983371c2a530a668823ee88279a6b33b6df874dfb1cedcfc"}
Nov 02 15:25:30.127781 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:30.127345    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-2f8b7c-openshift-default-769cb98f9-xm9dm" event={"ID":"6f8cf102-3f23-49fe-804f-694c50509462","Type":"ContainerStarted","Data":"111e895eb5546fb9fc2557c767ac5a6399f9bb0f38c5bb7254e4e575725d44de"}
Nov 02 15:25:30.166287 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:30.166251    5304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-2f8b7c-openshift-default-769cb98f9-xm9dm"
Nov 02 15:25:30.172938 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:30.172911    5304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-2f8b7c-openshift-default-769cb98f9-xm9dm"
Nov 02 15:25:30.179620 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:30.179574    5304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-2f8b7c-openshift-default-769cb98f9-xm9dm" podStartSLOduration=2.179559709 podStartE2EDuration="2.179559709s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:25:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:25:30.177279379 +0000 UTC m=+1730.171103798" watchObservedRunningTime="2026-11-02 15:25:30.179559709 +0000 UTC m=+1730.173384157"
Nov 02 15:25:30.264686 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:30.264653    5304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-28f621-openshift-default-7dd8c5c789-jvmsw"
Nov 02 15:25:30.730656 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:30.730630    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-db75ff-openshift-default-7558cfb4df-qnbxl"
Nov 02 15:25:30.852632 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:30.852586    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba7e1e2d-fe7e-41ab-8fb0-814e76dfb08b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba7e1e2d-fe7e-41ab-8fb0-814e76dfb08b-workload-socket\") pod \"ba7e1e2d-fe7e-41ab-8fb0-814e76dfb08b\" (UID: \"ba7e1e2d-fe7e-41ab-8fb0-814e76dfb08b\") "
Nov 02 15:25:30.852839 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:30.852639    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ba7e1e2d-fe7e-41ab-8fb0-814e76dfb08b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba7e1e2d-fe7e-41ab-8fb0-814e76dfb08b-istio-podinfo\") pod \"ba7e1e2d-fe7e-41ab-8fb0-814e76dfb08b\" (UID: \"ba7e1e2d-fe7e-41ab-8fb0-814e76dfb08b\") "
Nov 02 15:25:30.852839 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:30.852717    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ba7e1e2d-fe7e-41ab-8fb0-814e76dfb08b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba7e1e2d-fe7e-41ab-8fb0-814e76dfb08b-istiod-ca-cert\") pod \"ba7e1e2d-fe7e-41ab-8fb0-814e76dfb08b\" (UID: \"ba7e1e2d-fe7e-41ab-8fb0-814e76dfb08b\") "
Nov 02 15:25:30.852971 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:30.852877    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba7e1e2d-fe7e-41ab-8fb0-814e76dfb08b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba7e1e2d-fe7e-41ab-8fb0-814e76dfb08b-istio-envoy\") pod \"ba7e1e2d-fe7e-41ab-8fb0-814e76dfb08b\" (UID: \"ba7e1e2d-fe7e-41ab-8fb0-814e76dfb08b\") "
Nov 02 15:25:30.853034 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:30.852973    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba7e1e2d-fe7e-41ab-8fb0-814e76dfb08b-workload-socket" pod "ba7e1e2d-fe7e-41ab-8fb0-814e76dfb08b" (UID: "ba7e1e2d-fe7e-41ab-8fb0-814e76dfb08b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:25:30.853034 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:30.852993    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba7e1e2d-fe7e-41ab-8fb0-814e76dfb08b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba7e1e2d-fe7e-41ab-8fb0-814e76dfb08b-workload-certs\") pod \"ba7e1e2d-fe7e-41ab-8fb0-814e76dfb08b\" (UID: \"ba7e1e2d-fe7e-41ab-8fb0-814e76dfb08b\") "
Nov 02 15:25:30.853034 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:30.853027    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba7e1e2d-fe7e-41ab-8fb0-814e76dfb08b-kube-api-access-w2kzh\" (UniqueName: \"kubernetes.io/projected/ba7e1e2d-fe7e-41ab-8fb0-814e76dfb08b-kube-api-access-w2kzh\") pod \"ba7e1e2d-fe7e-41ab-8fb0-814e76dfb08b\" (UID: \"ba7e1e2d-fe7e-41ab-8fb0-814e76dfb08b\") "
Nov 02 15:25:30.853219 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:30.853203    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba7e1e2d-fe7e-41ab-8fb0-814e76dfb08b-workload-certs" pod "ba7e1e2d-fe7e-41ab-8fb0-814e76dfb08b" (UID: "ba7e1e2d-fe7e-41ab-8fb0-814e76dfb08b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:25:30.853404 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:30.853373    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba7e1e2d-fe7e-41ab-8fb0-814e76dfb08b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba7e1e2d-fe7e-41ab-8fb0-814e76dfb08b-istio-data\") pod \"ba7e1e2d-fe7e-41ab-8fb0-814e76dfb08b\" (UID: \"ba7e1e2d-fe7e-41ab-8fb0-814e76dfb08b\") "
Nov 02 15:25:30.853504 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:30.853416    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba7e1e2d-fe7e-41ab-8fb0-814e76dfb08b-istio-token\" (UniqueName: \"kubernetes.io/projected/ba7e1e2d-fe7e-41ab-8fb0-814e76dfb08b-istio-token\") pod \"ba7e1e2d-fe7e-41ab-8fb0-814e76dfb08b\" (UID: \"ba7e1e2d-fe7e-41ab-8fb0-814e76dfb08b\") "
Nov 02 15:25:30.853504 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:30.853428    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba7e1e2d-fe7e-41ab-8fb0-814e76dfb08b-istiod-ca-cert" pod "ba7e1e2d-fe7e-41ab-8fb0-814e76dfb08b" (UID: "ba7e1e2d-fe7e-41ab-8fb0-814e76dfb08b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:25:30.853504 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:30.853491    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba7e1e2d-fe7e-41ab-8fb0-814e76dfb08b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba7e1e2d-fe7e-41ab-8fb0-814e76dfb08b-credential-socket\") pod \"ba7e1e2d-fe7e-41ab-8fb0-814e76dfb08b\" (UID: \"ba7e1e2d-fe7e-41ab-8fb0-814e76dfb08b\") "
Nov 02 15:25:30.854952 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:30.854245    5304 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba7e1e2d-fe7e-41ab-8fb0-814e76dfb08b-workload-socket\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:25:30.854952 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:30.854276    5304 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba7e1e2d-fe7e-41ab-8fb0-814e76dfb08b-istiod-ca-cert\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:25:30.854952 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:30.854290    5304 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba7e1e2d-fe7e-41ab-8fb0-814e76dfb08b-workload-certs\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:25:30.854952 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:30.854097    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba7e1e2d-fe7e-41ab-8fb0-814e76dfb08b-istio-data" pod "ba7e1e2d-fe7e-41ab-8fb0-814e76dfb08b" (UID: "ba7e1e2d-fe7e-41ab-8fb0-814e76dfb08b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:25:30.854952 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:30.854504    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba7e1e2d-fe7e-41ab-8fb0-814e76dfb08b-credential-socket" pod "ba7e1e2d-fe7e-41ab-8fb0-814e76dfb08b" (UID: "ba7e1e2d-fe7e-41ab-8fb0-814e76dfb08b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:25:30.856221 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:30.856188    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba7e1e2d-fe7e-41ab-8fb0-814e76dfb08b-istio-envoy" pod "ba7e1e2d-fe7e-41ab-8fb0-814e76dfb08b" (UID: "ba7e1e2d-fe7e-41ab-8fb0-814e76dfb08b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:25:30.856320 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:30.856297    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba7e1e2d-fe7e-41ab-8fb0-814e76dfb08b-istio-token" pod "ba7e1e2d-fe7e-41ab-8fb0-814e76dfb08b" (UID: "ba7e1e2d-fe7e-41ab-8fb0-814e76dfb08b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:25:30.856548 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:30.856528    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ba7e1e2d-fe7e-41ab-8fb0-814e76dfb08b-istio-podinfo" pod "ba7e1e2d-fe7e-41ab-8fb0-814e76dfb08b" (UID: "ba7e1e2d-fe7e-41ab-8fb0-814e76dfb08b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 15:25:30.857465 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:30.857446    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba7e1e2d-fe7e-41ab-8fb0-814e76dfb08b-kube-api-access-w2kzh" pod "ba7e1e2d-fe7e-41ab-8fb0-814e76dfb08b" (UID: "ba7e1e2d-fe7e-41ab-8fb0-814e76dfb08b"). InnerVolumeSpecName "kube-api-access-w2kzh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:25:30.956040 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:30.955998    5304 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba7e1e2d-fe7e-41ab-8fb0-814e76dfb08b-istio-data\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:25:30.956040 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:30.956031    5304 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba7e1e2d-fe7e-41ab-8fb0-814e76dfb08b-istio-token\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:25:30.956040 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:30.956044    5304 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba7e1e2d-fe7e-41ab-8fb0-814e76dfb08b-credential-socket\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:25:30.956329 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:30.956058    5304 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba7e1e2d-fe7e-41ab-8fb0-814e76dfb08b-istio-podinfo\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:25:30.956329 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:30.956079    5304 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba7e1e2d-fe7e-41ab-8fb0-814e76dfb08b-istio-envoy\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:25:30.956329 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:30.956093    5304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w2kzh\" (UniqueName: \"kubernetes.io/projected/ba7e1e2d-fe7e-41ab-8fb0-814e76dfb08b-kube-api-access-w2kzh\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:25:31.135768 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:31.135737    5304 generic.go:356] "Generic (PLEG): container finished" podID="ba7e1e2d-fe7e-41ab-8fb0-814e76dfb08b" containerID="9d8770c74296cf691bbfd200950a31905a1098d7b37e6f0cc16a066cd790bafd" exitCode=0
Nov 02 15:25:31.136355 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:31.135781    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-db75ff-openshift-default-7558cfb4df-qnbxl"
Nov 02 15:25:31.136355 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:31.135811    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-db75ff-openshift-default-7558cfb4df-qnbxl" event={"ID":"ba7e1e2d-fe7e-41ab-8fb0-814e76dfb08b","Type":"ContainerDied","Data":"9d8770c74296cf691bbfd200950a31905a1098d7b37e6f0cc16a066cd790bafd"}
Nov 02 15:25:31.136355 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:31.135871    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-db75ff-openshift-default-7558cfb4df-qnbxl" event={"ID":"ba7e1e2d-fe7e-41ab-8fb0-814e76dfb08b","Type":"ContainerDied","Data":"895041ea777ca2e9e7ba5e60506fddbd33b270b2b64560849fb930d763223985"}
Nov 02 15:25:31.136355 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:31.135902    5304 scope.go:122] "RemoveContainer" containerID="9d8770c74296cf691bbfd200950a31905a1098d7b37e6f0cc16a066cd790bafd"
Nov 02 15:25:31.136592 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:31.136552    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-28f621-openshift-default-7dd8c5c789-jvmsw"
Nov 02 15:25:31.137099 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:31.137061    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-2f8b7c-openshift-default-769cb98f9-xm9dm"
Nov 02 15:25:31.138339 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:31.138317    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-2f8b7c-openshift-default-769cb98f9-xm9dm"
Nov 02 15:25:31.138461 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:31.138442    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-28f621-openshift-default-7dd8c5c789-jvmsw"
Nov 02 15:25:31.149109 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:31.149080    5304 scope.go:122] "RemoveContainer" containerID="8662bbc92abbc0a7254dedc4fea3f21fca49213a2da12c7d8f39939caad6a109"
Nov 02 15:25:31.161187 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:31.161160    5304 scope.go:122] "RemoveContainer" containerID="9d8770c74296cf691bbfd200950a31905a1098d7b37e6f0cc16a066cd790bafd"
Nov 02 15:25:31.161505 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:25:31.161487    5304 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d8770c74296cf691bbfd200950a31905a1098d7b37e6f0cc16a066cd790bafd\": container with ID starting with 9d8770c74296cf691bbfd200950a31905a1098d7b37e6f0cc16a066cd790bafd not found: ID does not exist" containerID="9d8770c74296cf691bbfd200950a31905a1098d7b37e6f0cc16a066cd790bafd"
Nov 02 15:25:31.161563 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:31.161515    5304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d8770c74296cf691bbfd200950a31905a1098d7b37e6f0cc16a066cd790bafd"} err="failed to get container status \"9d8770c74296cf691bbfd200950a31905a1098d7b37e6f0cc16a066cd790bafd\": rpc error: code = NotFound desc = could not find container \"9d8770c74296cf691bbfd200950a31905a1098d7b37e6f0cc16a066cd790bafd\": container with ID starting with 9d8770c74296cf691bbfd200950a31905a1098d7b37e6f0cc16a066cd790bafd not found: ID does not exist"
Nov 02 15:25:31.161563 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:31.161533    5304 scope.go:122] "RemoveContainer" containerID="8662bbc92abbc0a7254dedc4fea3f21fca49213a2da12c7d8f39939caad6a109"
Nov 02 15:25:31.161818 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:25:31.161802    5304 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8662bbc92abbc0a7254dedc4fea3f21fca49213a2da12c7d8f39939caad6a109\": container with ID starting with 8662bbc92abbc0a7254dedc4fea3f21fca49213a2da12c7d8f39939caad6a109 not found: ID does not exist" containerID="8662bbc92abbc0a7254dedc4fea3f21fca49213a2da12c7d8f39939caad6a109"
Nov 02 15:25:31.161877 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:31.161820    5304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8662bbc92abbc0a7254dedc4fea3f21fca49213a2da12c7d8f39939caad6a109"} err="failed to get container status \"8662bbc92abbc0a7254dedc4fea3f21fca49213a2da12c7d8f39939caad6a109\": rpc error: code = NotFound desc = could not find container \"8662bbc92abbc0a7254dedc4fea3f21fca49213a2da12c7d8f39939caad6a109\": container with ID starting with 8662bbc92abbc0a7254dedc4fea3f21fca49213a2da12c7d8f39939caad6a109 not found: ID does not exist"
Nov 02 15:25:31.251820 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:31.251782    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-db75ff-openshift-default-7558cfb4df-qnbxl"]
Nov 02 15:25:31.260346 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:31.260314    5304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-db75ff-openshift-default-7558cfb4df-qnbxl"]
Nov 02 15:25:32.119931 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:32.119898    5304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ba7e1e2d-fe7e-41ab-8fb0-814e76dfb08b" path="/var/lib/kubelet/pods/ba7e1e2d-fe7e-41ab-8fb0-814e76dfb08b/volumes"
Nov 02 15:25:32.923861 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:32.923831    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-815746-openshift-default-74696d484f-9f99s"
Nov 02 15:25:32.984008 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:32.983926    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ecb8e24-cbc7-4808-a2aa-6536d94590b5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ecb8e24-cbc7-4808-a2aa-6536d94590b5-workload-certs\") pod \"0ecb8e24-cbc7-4808-a2aa-6536d94590b5\" (UID: \"0ecb8e24-cbc7-4808-a2aa-6536d94590b5\") "
Nov 02 15:25:32.984008 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:32.983994    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ecb8e24-cbc7-4808-a2aa-6536d94590b5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ecb8e24-cbc7-4808-a2aa-6536d94590b5-workload-socket\") pod \"0ecb8e24-cbc7-4808-a2aa-6536d94590b5\" (UID: \"0ecb8e24-cbc7-4808-a2aa-6536d94590b5\") "
Nov 02 15:25:32.984280 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:32.984018    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ecb8e24-cbc7-4808-a2aa-6536d94590b5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ecb8e24-cbc7-4808-a2aa-6536d94590b5-credential-socket\") pod \"0ecb8e24-cbc7-4808-a2aa-6536d94590b5\" (UID: \"0ecb8e24-cbc7-4808-a2aa-6536d94590b5\") "
Nov 02 15:25:32.984280 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:32.984057    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0ecb8e24-cbc7-4808-a2aa-6536d94590b5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ecb8e24-cbc7-4808-a2aa-6536d94590b5-istiod-ca-cert\") pod \"0ecb8e24-cbc7-4808-a2aa-6536d94590b5\" (UID: \"0ecb8e24-cbc7-4808-a2aa-6536d94590b5\") "
Nov 02 15:25:32.984280 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:32.984093    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ecb8e24-cbc7-4808-a2aa-6536d94590b5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ecb8e24-cbc7-4808-a2aa-6536d94590b5-istio-envoy\") pod \"0ecb8e24-cbc7-4808-a2aa-6536d94590b5\" (UID: \"0ecb8e24-cbc7-4808-a2aa-6536d94590b5\") "
Nov 02 15:25:32.984280 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:32.984177    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ecb8e24-cbc7-4808-a2aa-6536d94590b5-istio-token\" (UniqueName: \"kubernetes.io/projected/0ecb8e24-cbc7-4808-a2aa-6536d94590b5-istio-token\") pod \"0ecb8e24-cbc7-4808-a2aa-6536d94590b5\" (UID: \"0ecb8e24-cbc7-4808-a2aa-6536d94590b5\") "
Nov 02 15:25:32.984280 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:32.984267    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0ecb8e24-cbc7-4808-a2aa-6536d94590b5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ecb8e24-cbc7-4808-a2aa-6536d94590b5-istio-podinfo\") pod \"0ecb8e24-cbc7-4808-a2aa-6536d94590b5\" (UID: \"0ecb8e24-cbc7-4808-a2aa-6536d94590b5\") "
Nov 02 15:25:32.984280 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:32.984262    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ecb8e24-cbc7-4808-a2aa-6536d94590b5-workload-certs" pod "0ecb8e24-cbc7-4808-a2aa-6536d94590b5" (UID: "0ecb8e24-cbc7-4808-a2aa-6536d94590b5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:25:32.984584 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:32.984399    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ecb8e24-cbc7-4808-a2aa-6536d94590b5-kube-api-access-snklp\" (UniqueName: \"kubernetes.io/projected/0ecb8e24-cbc7-4808-a2aa-6536d94590b5-kube-api-access-snklp\") pod \"0ecb8e24-cbc7-4808-a2aa-6536d94590b5\" (UID: \"0ecb8e24-cbc7-4808-a2aa-6536d94590b5\") "
Nov 02 15:25:32.984584 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:32.984441    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ecb8e24-cbc7-4808-a2aa-6536d94590b5-workload-socket" pod "0ecb8e24-cbc7-4808-a2aa-6536d94590b5" (UID: "0ecb8e24-cbc7-4808-a2aa-6536d94590b5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:25:32.984584 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:32.984456    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ecb8e24-cbc7-4808-a2aa-6536d94590b5-credential-socket" pod "0ecb8e24-cbc7-4808-a2aa-6536d94590b5" (UID: "0ecb8e24-cbc7-4808-a2aa-6536d94590b5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:25:32.984584 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:32.984553    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ecb8e24-cbc7-4808-a2aa-6536d94590b5-istiod-ca-cert" pod "0ecb8e24-cbc7-4808-a2aa-6536d94590b5" (UID: "0ecb8e24-cbc7-4808-a2aa-6536d94590b5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:25:32.984771 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:32.984592    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ecb8e24-cbc7-4808-a2aa-6536d94590b5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ecb8e24-cbc7-4808-a2aa-6536d94590b5-istio-data\") pod \"0ecb8e24-cbc7-4808-a2aa-6536d94590b5\" (UID: \"0ecb8e24-cbc7-4808-a2aa-6536d94590b5\") "
Nov 02 15:25:32.984988 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:32.984966    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ecb8e24-cbc7-4808-a2aa-6536d94590b5-istio-data" pod "0ecb8e24-cbc7-4808-a2aa-6536d94590b5" (UID: "0ecb8e24-cbc7-4808-a2aa-6536d94590b5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:25:32.985440 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:32.985418    5304 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ecb8e24-cbc7-4808-a2aa-6536d94590b5-workload-certs\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:25:32.985519 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:32.985447    5304 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ecb8e24-cbc7-4808-a2aa-6536d94590b5-workload-socket\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:25:32.985519 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:32.985461    5304 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ecb8e24-cbc7-4808-a2aa-6536d94590b5-credential-socket\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:25:32.985519 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:32.985475    5304 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ecb8e24-cbc7-4808-a2aa-6536d94590b5-istiod-ca-cert\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:25:32.985519 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:32.985489    5304 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ecb8e24-cbc7-4808-a2aa-6536d94590b5-istio-data\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:25:32.986726 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:32.986699    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0ecb8e24-cbc7-4808-a2aa-6536d94590b5-istio-podinfo" pod "0ecb8e24-cbc7-4808-a2aa-6536d94590b5" (UID: "0ecb8e24-cbc7-4808-a2aa-6536d94590b5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 15:25:32.987016 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:32.986991    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ecb8e24-cbc7-4808-a2aa-6536d94590b5-istio-envoy" pod "0ecb8e24-cbc7-4808-a2aa-6536d94590b5" (UID: "0ecb8e24-cbc7-4808-a2aa-6536d94590b5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:25:32.987016 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:32.986997    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ecb8e24-cbc7-4808-a2aa-6536d94590b5-istio-token" pod "0ecb8e24-cbc7-4808-a2aa-6536d94590b5" (UID: "0ecb8e24-cbc7-4808-a2aa-6536d94590b5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:25:32.987206 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:32.987025    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ecb8e24-cbc7-4808-a2aa-6536d94590b5-kube-api-access-snklp" pod "0ecb8e24-cbc7-4808-a2aa-6536d94590b5" (UID: "0ecb8e24-cbc7-4808-a2aa-6536d94590b5"). InnerVolumeSpecName "kube-api-access-snklp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:25:33.087355 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:33.087310    5304 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ecb8e24-cbc7-4808-a2aa-6536d94590b5-istio-envoy\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:25:33.087355 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:33.087354    5304 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ecb8e24-cbc7-4808-a2aa-6536d94590b5-istio-token\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:25:33.087355 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:33.087367    5304 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ecb8e24-cbc7-4808-a2aa-6536d94590b5-istio-podinfo\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:25:33.087609 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:33.087380    5304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-snklp\" (UniqueName: \"kubernetes.io/projected/0ecb8e24-cbc7-4808-a2aa-6536d94590b5-kube-api-access-snklp\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:25:33.152597 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:33.152560    5304 generic.go:356] "Generic (PLEG): container finished" podID="0ecb8e24-cbc7-4808-a2aa-6536d94590b5" containerID="25bce27ef3f8bd79bed947b2260a48f0b045cfa9200c871de571e2f8e31470f9" exitCode=0
Nov 02 15:25:33.152786 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:33.152614    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-815746-openshift-default-74696d484f-9f99s"
Nov 02 15:25:33.152786 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:33.152643    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-815746-openshift-default-74696d484f-9f99s" event={"ID":"0ecb8e24-cbc7-4808-a2aa-6536d94590b5","Type":"ContainerDied","Data":"25bce27ef3f8bd79bed947b2260a48f0b045cfa9200c871de571e2f8e31470f9"}
Nov 02 15:25:33.152786 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:33.152701    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-815746-openshift-default-74696d484f-9f99s" event={"ID":"0ecb8e24-cbc7-4808-a2aa-6536d94590b5","Type":"ContainerDied","Data":"8fe73f3c1c5bca67bed0fb7d27386836f4bebac8656230c89b0d286680f1fedc"}
Nov 02 15:25:33.152786 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:33.152724    5304 scope.go:122] "RemoveContainer" containerID="25bce27ef3f8bd79bed947b2260a48f0b045cfa9200c871de571e2f8e31470f9"
Nov 02 15:25:33.164548 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:33.164529    5304 scope.go:122] "RemoveContainer" containerID="efc40a69c6cce5f130d98deb4f6cd2f5ba2b24a74d3f5b0c559a0bf990d84bd0"
Nov 02 15:25:33.176153 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:33.176105    5304 scope.go:122] "RemoveContainer" containerID="25bce27ef3f8bd79bed947b2260a48f0b045cfa9200c871de571e2f8e31470f9"
Nov 02 15:25:33.176452 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:25:33.176422    5304 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"25bce27ef3f8bd79bed947b2260a48f0b045cfa9200c871de571e2f8e31470f9\": container with ID starting with 25bce27ef3f8bd79bed947b2260a48f0b045cfa9200c871de571e2f8e31470f9 not found: ID does not exist" containerID="25bce27ef3f8bd79bed947b2260a48f0b045cfa9200c871de571e2f8e31470f9"
Nov 02 15:25:33.176548 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:33.176462    5304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"25bce27ef3f8bd79bed947b2260a48f0b045cfa9200c871de571e2f8e31470f9"} err="failed to get container status \"25bce27ef3f8bd79bed947b2260a48f0b045cfa9200c871de571e2f8e31470f9\": rpc error: code = NotFound desc = could not find container \"25bce27ef3f8bd79bed947b2260a48f0b045cfa9200c871de571e2f8e31470f9\": container with ID starting with 25bce27ef3f8bd79bed947b2260a48f0b045cfa9200c871de571e2f8e31470f9 not found: ID does not exist"
Nov 02 15:25:33.176548 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:33.176485    5304 scope.go:122] "RemoveContainer" containerID="efc40a69c6cce5f130d98deb4f6cd2f5ba2b24a74d3f5b0c559a0bf990d84bd0"
Nov 02 15:25:33.176729 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:25:33.176710    5304 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"efc40a69c6cce5f130d98deb4f6cd2f5ba2b24a74d3f5b0c559a0bf990d84bd0\": container with ID starting with efc40a69c6cce5f130d98deb4f6cd2f5ba2b24a74d3f5b0c559a0bf990d84bd0 not found: ID does not exist" containerID="efc40a69c6cce5f130d98deb4f6cd2f5ba2b24a74d3f5b0c559a0bf990d84bd0"
Nov 02 15:25:33.176784 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:33.176736    5304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"efc40a69c6cce5f130d98deb4f6cd2f5ba2b24a74d3f5b0c559a0bf990d84bd0"} err="failed to get container status \"efc40a69c6cce5f130d98deb4f6cd2f5ba2b24a74d3f5b0c559a0bf990d84bd0\": rpc error: code = NotFound desc = could not find container \"efc40a69c6cce5f130d98deb4f6cd2f5ba2b24a74d3f5b0c559a0bf990d84bd0\": container with ID starting with efc40a69c6cce5f130d98deb4f6cd2f5ba2b24a74d3f5b0c559a0bf990d84bd0 not found: ID does not exist"
Nov 02 15:25:33.215263 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:33.215230    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-815746-openshift-default-74696d484f-9f99s"]
Nov 02 15:25:33.233532 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:33.233504    5304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-815746-openshift-default-74696d484f-9f99s"]
Nov 02 15:25:34.121004 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:34.120970    5304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0ecb8e24-cbc7-4808-a2aa-6536d94590b5" path="/var/lib/kubelet/pods/0ecb8e24-cbc7-4808-a2aa-6536d94590b5/volumes"
Nov 02 15:25:36.492322 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:36.492287    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-21ece2-85dc7bbc8c-59ktr"]
Nov 02 15:25:37.665257 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:37.665218    5304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-3daba5/test-model-3daba5-kserve-6f59d8687-qp658"]
Nov 02 15:25:37.666238 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:37.666219    5304 cpu_manager.go:414] "containerMap: removing container" podUID="0ecb8e24-cbc7-4808-a2aa-6536d94590b5" containerName="istio-proxy"
Nov 02 15:25:37.666287 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:37.666241    5304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ecb8e24-cbc7-4808-a2aa-6536d94590b5" containerName="istio-proxy"
Nov 02 15:25:37.666287 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:37.666258    5304 cpu_manager.go:414] "containerMap: removing container" podUID="ba7e1e2d-fe7e-41ab-8fb0-814e76dfb08b" containerName="istio-proxy"
Nov 02 15:25:37.666287 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:37.666268    5304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba7e1e2d-fe7e-41ab-8fb0-814e76dfb08b" containerName="istio-proxy"
Nov 02 15:25:37.666431 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:37.666308    5304 cpu_manager.go:414] "containerMap: removing container" podUID="0ecb8e24-cbc7-4808-a2aa-6536d94590b5" containerName="istio-proxy"
Nov 02 15:25:37.666431 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:37.666317    5304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ecb8e24-cbc7-4808-a2aa-6536d94590b5" containerName="istio-proxy"
Nov 02 15:25:37.666431 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:37.666338    5304 cpu_manager.go:414] "containerMap: removing container" podUID="ba7e1e2d-fe7e-41ab-8fb0-814e76dfb08b" containerName="istio-proxy"
Nov 02 15:25:37.666431 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:37.666345    5304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba7e1e2d-fe7e-41ab-8fb0-814e76dfb08b" containerName="istio-proxy"
Nov 02 15:25:37.666574 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:37.666517    5304 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ecb8e24-cbc7-4808-a2aa-6536d94590b5" containerName="istio-proxy"
Nov 02 15:25:37.666574 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:37.666531    5304 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba7e1e2d-fe7e-41ab-8fb0-814e76dfb08b" containerName="istio-proxy"
Nov 02 15:25:37.666574 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:37.666548    5304 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba7e1e2d-fe7e-41ab-8fb0-814e76dfb08b" containerName="istio-proxy"
Nov 02 15:25:37.671872 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:37.671843    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-3daba5/test-model-3daba5-kserve-6f59d8687-qp658"
Nov 02 15:25:37.691018 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:37.690983    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-3daba5/test-model-3daba5-kserve-6f59d8687-qp658"]
Nov 02 15:25:37.700706 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:37.700675    5304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-3daba5\"/\"test-model-3daba5-kserve-self-signed-certs\""
Nov 02 15:25:37.700706 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:37.700701    5304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-3daba5\"/\"default-dockercfg-6rv6t\""
Nov 02 15:25:37.708169 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:37.708144    5304 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-3daba5\"/\"kube-root-ca.crt\""
Nov 02 15:25:37.708169 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:37.708144    5304 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-3daba5\"/\"openshift-service-ca.crt\""
Nov 02 15:25:37.741528 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:37.741496    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/33fc6836-e163-4b43-8a3b-b04ce4e6f4bc-tls-certs\") pod \"test-model-3daba5-kserve-6f59d8687-qp658\" (UID: \"33fc6836-e163-4b43-8a3b-b04ce4e6f4bc\") " pod="ai-tenant-e2e-inf-b-3daba5/test-model-3daba5-kserve-6f59d8687-qp658"
Nov 02 15:25:37.741710 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:37.741591    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/33fc6836-e163-4b43-8a3b-b04ce4e6f4bc-home\") pod \"test-model-3daba5-kserve-6f59d8687-qp658\" (UID: \"33fc6836-e163-4b43-8a3b-b04ce4e6f4bc\") " pod="ai-tenant-e2e-inf-b-3daba5/test-model-3daba5-kserve-6f59d8687-qp658"
Nov 02 15:25:37.741778 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:37.741745    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/33fc6836-e163-4b43-8a3b-b04ce4e6f4bc-dshm\") pod \"test-model-3daba5-kserve-6f59d8687-qp658\" (UID: \"33fc6836-e163-4b43-8a3b-b04ce4e6f4bc\") " pod="ai-tenant-e2e-inf-b-3daba5/test-model-3daba5-kserve-6f59d8687-qp658"
Nov 02 15:25:37.741842 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:37.741826    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/33fc6836-e163-4b43-8a3b-b04ce4e6f4bc-tmp-dir\") pod \"test-model-3daba5-kserve-6f59d8687-qp658\" (UID: \"33fc6836-e163-4b43-8a3b-b04ce4e6f4bc\") " pod="ai-tenant-e2e-inf-b-3daba5/test-model-3daba5-kserve-6f59d8687-qp658"
Nov 02 15:25:37.741898 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:37.741873    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wl2ht\" (UniqueName: \"kubernetes.io/projected/33fc6836-e163-4b43-8a3b-b04ce4e6f4bc-kube-api-access-wl2ht\") pod \"test-model-3daba5-kserve-6f59d8687-qp658\" (UID: \"33fc6836-e163-4b43-8a3b-b04ce4e6f4bc\") " pod="ai-tenant-e2e-inf-b-3daba5/test-model-3daba5-kserve-6f59d8687-qp658"
Nov 02 15:25:37.741898 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:37.741895    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/33fc6836-e163-4b43-8a3b-b04ce4e6f4bc-model-cache\") pod \"test-model-3daba5-kserve-6f59d8687-qp658\" (UID: \"33fc6836-e163-4b43-8a3b-b04ce4e6f4bc\") " pod="ai-tenant-e2e-inf-b-3daba5/test-model-3daba5-kserve-6f59d8687-qp658"
Nov 02 15:25:37.843496 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:37.843453    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/33fc6836-e163-4b43-8a3b-b04ce4e6f4bc-dshm\") pod \"test-model-3daba5-kserve-6f59d8687-qp658\" (UID: \"33fc6836-e163-4b43-8a3b-b04ce4e6f4bc\") " pod="ai-tenant-e2e-inf-b-3daba5/test-model-3daba5-kserve-6f59d8687-qp658"
Nov 02 15:25:37.843688 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:37.843536    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/33fc6836-e163-4b43-8a3b-b04ce4e6f4bc-tmp-dir\") pod \"test-model-3daba5-kserve-6f59d8687-qp658\" (UID: \"33fc6836-e163-4b43-8a3b-b04ce4e6f4bc\") " pod="ai-tenant-e2e-inf-b-3daba5/test-model-3daba5-kserve-6f59d8687-qp658"
Nov 02 15:25:37.843688 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:37.843586    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wl2ht\" (UniqueName: \"kubernetes.io/projected/33fc6836-e163-4b43-8a3b-b04ce4e6f4bc-kube-api-access-wl2ht\") pod \"test-model-3daba5-kserve-6f59d8687-qp658\" (UID: \"33fc6836-e163-4b43-8a3b-b04ce4e6f4bc\") " pod="ai-tenant-e2e-inf-b-3daba5/test-model-3daba5-kserve-6f59d8687-qp658"
Nov 02 15:25:37.843688 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:37.843610    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/33fc6836-e163-4b43-8a3b-b04ce4e6f4bc-model-cache\") pod \"test-model-3daba5-kserve-6f59d8687-qp658\" (UID: \"33fc6836-e163-4b43-8a3b-b04ce4e6f4bc\") " pod="ai-tenant-e2e-inf-b-3daba5/test-model-3daba5-kserve-6f59d8687-qp658"
Nov 02 15:25:37.843871 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:37.843719    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/33fc6836-e163-4b43-8a3b-b04ce4e6f4bc-tls-certs\") pod \"test-model-3daba5-kserve-6f59d8687-qp658\" (UID: \"33fc6836-e163-4b43-8a3b-b04ce4e6f4bc\") " pod="ai-tenant-e2e-inf-b-3daba5/test-model-3daba5-kserve-6f59d8687-qp658"
Nov 02 15:25:37.843871 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:37.843768    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/33fc6836-e163-4b43-8a3b-b04ce4e6f4bc-home\") pod \"test-model-3daba5-kserve-6f59d8687-qp658\" (UID: \"33fc6836-e163-4b43-8a3b-b04ce4e6f4bc\") " pod="ai-tenant-e2e-inf-b-3daba5/test-model-3daba5-kserve-6f59d8687-qp658"
Nov 02 15:25:37.844065 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:37.844034    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/33fc6836-e163-4b43-8a3b-b04ce4e6f4bc-tmp-dir\") pod \"test-model-3daba5-kserve-6f59d8687-qp658\" (UID: \"33fc6836-e163-4b43-8a3b-b04ce4e6f4bc\") " pod="ai-tenant-e2e-inf-b-3daba5/test-model-3daba5-kserve-6f59d8687-qp658"
Nov 02 15:25:37.844220 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:37.844038    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/33fc6836-e163-4b43-8a3b-b04ce4e6f4bc-model-cache\") pod \"test-model-3daba5-kserve-6f59d8687-qp658\" (UID: \"33fc6836-e163-4b43-8a3b-b04ce4e6f4bc\") " pod="ai-tenant-e2e-inf-b-3daba5/test-model-3daba5-kserve-6f59d8687-qp658"
Nov 02 15:25:37.844220 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:37.844092    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/33fc6836-e163-4b43-8a3b-b04ce4e6f4bc-home\") pod \"test-model-3daba5-kserve-6f59d8687-qp658\" (UID: \"33fc6836-e163-4b43-8a3b-b04ce4e6f4bc\") " pod="ai-tenant-e2e-inf-b-3daba5/test-model-3daba5-kserve-6f59d8687-qp658"
Nov 02 15:25:37.845882 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:37.845856    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/33fc6836-e163-4b43-8a3b-b04ce4e6f4bc-dshm\") pod \"test-model-3daba5-kserve-6f59d8687-qp658\" (UID: \"33fc6836-e163-4b43-8a3b-b04ce4e6f4bc\") " pod="ai-tenant-e2e-inf-b-3daba5/test-model-3daba5-kserve-6f59d8687-qp658"
Nov 02 15:25:37.846271 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:37.846255    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/33fc6836-e163-4b43-8a3b-b04ce4e6f4bc-tls-certs\") pod \"test-model-3daba5-kserve-6f59d8687-qp658\" (UID: \"33fc6836-e163-4b43-8a3b-b04ce4e6f4bc\") " pod="ai-tenant-e2e-inf-b-3daba5/test-model-3daba5-kserve-6f59d8687-qp658"
Nov 02 15:25:37.869583 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:37.869556    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wl2ht\" (UniqueName: \"kubernetes.io/projected/33fc6836-e163-4b43-8a3b-b04ce4e6f4bc-kube-api-access-wl2ht\") pod \"test-model-3daba5-kserve-6f59d8687-qp658\" (UID: \"33fc6836-e163-4b43-8a3b-b04ce4e6f4bc\") " pod="ai-tenant-e2e-inf-b-3daba5/test-model-3daba5-kserve-6f59d8687-qp658"
Nov 02 15:25:37.991881 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:37.991485    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-3daba5/test-model-3daba5-kserve-6f59d8687-qp658"
Nov 02 15:25:38.111615 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:25:38.111577    5304 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/system.slice/NetworkManager-dispatcher.service\": RecentStats: unable to find data in memory cache]"
Nov 02 15:25:38.352820 ip-10-0-135-138 kubenswrapper[5304]: W1102 15:25:38.352787    5304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod33fc6836_e163_4b43_8a3b_b04ce4e6f4bc.slice/crio-fd2d28715baa23d76d42bef7f21accead8c7491b3ea2d9bdd272b5d0db4930a2 WatchSource:0}: Error finding container fd2d28715baa23d76d42bef7f21accead8c7491b3ea2d9bdd272b5d0db4930a2: Status 404 returned error can't find the container with id fd2d28715baa23d76d42bef7f21accead8c7491b3ea2d9bdd272b5d0db4930a2
Nov 02 15:25:38.356496 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:38.356474    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-3daba5/test-model-3daba5-kserve-6f59d8687-qp658"]
Nov 02 15:25:39.254829 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:39.254794    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-3daba5/test-model-3daba5-kserve-6f59d8687-qp658" event={"ID":"33fc6836-e163-4b43-8a3b-b04ce4e6f4bc","Type":"ContainerStarted","Data":"4f2309313dd11f79152f30c171cf02a7c24d76179f07bd06e3923d0f3ec584e1"}
Nov 02 15:25:39.254829 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:39.254836    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-3daba5/test-model-3daba5-kserve-6f59d8687-qp658" event={"ID":"33fc6836-e163-4b43-8a3b-b04ce4e6f4bc","Type":"ContainerStarted","Data":"fd2d28715baa23d76d42bef7f21accead8c7491b3ea2d9bdd272b5d0db4930a2"}
Nov 02 15:25:39.304441 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:39.304383    5304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-3daba5/test-model-3daba5-kserve-6f59d8687-qp658" podStartSLOduration=2.304365963 podStartE2EDuration="2.304365963s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:25:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:25:39.300713676 +0000 UTC m=+1739.294538085" watchObservedRunningTime="2026-11-02 15:25:39.304365963 +0000 UTC m=+1739.298190384"
Nov 02 15:25:39.993575 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:39.993536    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-3daba5/test-model-3daba5-kserve-6f59d8687-qp658"
Nov 02 15:25:41.492988 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:41.492947    5304 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-21ece2-85dc7bbc8c-59ktr" podUID="13e8c2b9-05fb-44e4-99b7-fd3104272bef" containerName="payload-pre-processing" containerID="cri-o://949da75c2c36820110111ee72f86d794110509e2f7985044c0667f32d3c89766" gracePeriod=25
Nov 02 15:25:41.793345 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:41.793313    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-21ece2-85dc7bbc8c-59ktr"
Nov 02 15:25:41.922318 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:41.922285    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/13e8c2b9-05fb-44e4-99b7-fd3104272bef-kube-api-access-fl4x7\" (UniqueName: \"kubernetes.io/projected/13e8c2b9-05fb-44e4-99b7-fd3104272bef-kube-api-access-fl4x7\") pod \"13e8c2b9-05fb-44e4-99b7-fd3104272bef\" (UID: \"13e8c2b9-05fb-44e4-99b7-fd3104272bef\") "
Nov 02 15:25:41.922517 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:41.922500    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/13e8c2b9-05fb-44e4-99b7-fd3104272bef-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13e8c2b9-05fb-44e4-99b7-fd3104272bef-plugins-config-volume\") pod \"13e8c2b9-05fb-44e4-99b7-fd3104272bef\" (UID: \"13e8c2b9-05fb-44e4-99b7-fd3104272bef\") "
Nov 02 15:25:41.922981 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:41.922943    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13e8c2b9-05fb-44e4-99b7-fd3104272bef-plugins-config-volume" pod "13e8c2b9-05fb-44e4-99b7-fd3104272bef" (UID: "13e8c2b9-05fb-44e4-99b7-fd3104272bef"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:25:41.923244 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:41.923221    5304 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13e8c2b9-05fb-44e4-99b7-fd3104272bef-plugins-config-volume\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:25:41.924789 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:41.924756    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13e8c2b9-05fb-44e4-99b7-fd3104272bef-kube-api-access-fl4x7" pod "13e8c2b9-05fb-44e4-99b7-fd3104272bef" (UID: "13e8c2b9-05fb-44e4-99b7-fd3104272bef"). InnerVolumeSpecName "kube-api-access-fl4x7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:25:42.024750 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:42.024654    5304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fl4x7\" (UniqueName: \"kubernetes.io/projected/13e8c2b9-05fb-44e4-99b7-fd3104272bef-kube-api-access-fl4x7\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:25:42.276673 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:42.276580    5304 generic.go:356] "Generic (PLEG): container finished" podID="13e8c2b9-05fb-44e4-99b7-fd3104272bef" containerID="949da75c2c36820110111ee72f86d794110509e2f7985044c0667f32d3c89766" exitCode=0
Nov 02 15:25:42.276830 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:42.276739    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-21ece2-85dc7bbc8c-59ktr" event={"ID":"13e8c2b9-05fb-44e4-99b7-fd3104272bef","Type":"ContainerDied","Data":"949da75c2c36820110111ee72f86d794110509e2f7985044c0667f32d3c89766"}
Nov 02 15:25:42.276830 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:42.276793    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-21ece2-85dc7bbc8c-59ktr" event={"ID":"13e8c2b9-05fb-44e4-99b7-fd3104272bef","Type":"ContainerDied","Data":"41898dac39eaeb5538a0248d87c4fbe17aad453eaf0f7484f63a497882ea5cb7"}
Nov 02 15:25:42.276830 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:42.276822    5304 scope.go:122] "RemoveContainer" containerID="949da75c2c36820110111ee72f86d794110509e2f7985044c0667f32d3c89766"
Nov 02 15:25:42.277004 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:42.276990    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-21ece2-85dc7bbc8c-59ktr"
Nov 02 15:25:42.289326 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:42.289265    5304 scope.go:122] "RemoveContainer" containerID="949da75c2c36820110111ee72f86d794110509e2f7985044c0667f32d3c89766"
Nov 02 15:25:42.289638 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:25:42.289608    5304 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"949da75c2c36820110111ee72f86d794110509e2f7985044c0667f32d3c89766\": container with ID starting with 949da75c2c36820110111ee72f86d794110509e2f7985044c0667f32d3c89766 not found: ID does not exist" containerID="949da75c2c36820110111ee72f86d794110509e2f7985044c0667f32d3c89766"
Nov 02 15:25:42.289714 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:42.289648    5304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"949da75c2c36820110111ee72f86d794110509e2f7985044c0667f32d3c89766"} err="failed to get container status \"949da75c2c36820110111ee72f86d794110509e2f7985044c0667f32d3c89766\": rpc error: code = NotFound desc = could not find container \"949da75c2c36820110111ee72f86d794110509e2f7985044c0667f32d3c89766\": container with ID starting with 949da75c2c36820110111ee72f86d794110509e2f7985044c0667f32d3c89766 not found: ID does not exist"
Nov 02 15:25:42.342898 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:42.342860    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-21ece2-85dc7bbc8c-59ktr"]
Nov 02 15:25:42.356569 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:42.356535    5304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-21ece2-85dc7bbc8c-59ktr"]
Nov 02 15:25:44.123671 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:44.123634    5304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="13e8c2b9-05fb-44e4-99b7-fd3104272bef" path="/var/lib/kubelet/pods/13e8c2b9-05fb-44e4-99b7-fd3104272bef/volumes"
Nov 02 15:25:44.983149 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:44.980184    5304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-28f621-dd989bdc-9x66m"]
Nov 02 15:25:44.983149 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:44.981244    5304 cpu_manager.go:414] "containerMap: removing container" podUID="13e8c2b9-05fb-44e4-99b7-fd3104272bef" containerName="payload-pre-processing"
Nov 02 15:25:44.983149 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:44.981266    5304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="13e8c2b9-05fb-44e4-99b7-fd3104272bef" containerName="payload-pre-processing"
Nov 02 15:25:44.983149 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:44.981490    5304 memory_manager.go:365] "RemoveStaleState removing state" podUID="13e8c2b9-05fb-44e4-99b7-fd3104272bef" containerName="payload-pre-processing"
Nov 02 15:25:44.983149 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:44.981505    5304 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ecb8e24-cbc7-4808-a2aa-6536d94590b5" containerName="istio-proxy"
Nov 02 15:25:45.017150 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:45.016781    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-28f621-dd989bdc-9x66m"
Nov 02 15:25:45.028149 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:45.025748    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-28f621-dd989bdc-9x66m"]
Nov 02 15:25:45.083373 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:45.082463    5304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-28f621-dockercfg-7cvsd\""
Nov 02 15:25:45.083373 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:45.083329    5304 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-28f621\""
Nov 02 15:25:45.088537 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:45.087722    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g8xlb\" (UniqueName: \"kubernetes.io/projected/dc422942-41f9-4cee-938f-e9df1b05d0c7-kube-api-access-g8xlb\") pod \"payload-processing-e2e-inf-a-28f621-dd989bdc-9x66m\" (UID: \"dc422942-41f9-4cee-938f-e9df1b05d0c7\") " pod="openshift-ingress/payload-processing-e2e-inf-a-28f621-dd989bdc-9x66m"
Nov 02 15:25:45.088537 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:45.087811    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc422942-41f9-4cee-938f-e9df1b05d0c7-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-28f621-dd989bdc-9x66m\" (UID: \"dc422942-41f9-4cee-938f-e9df1b05d0c7\") " pod="openshift-ingress/payload-processing-e2e-inf-a-28f621-dd989bdc-9x66m"
Nov 02 15:25:45.189652 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:45.189619    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g8xlb\" (UniqueName: \"kubernetes.io/projected/dc422942-41f9-4cee-938f-e9df1b05d0c7-kube-api-access-g8xlb\") pod \"payload-processing-e2e-inf-a-28f621-dd989bdc-9x66m\" (UID: \"dc422942-41f9-4cee-938f-e9df1b05d0c7\") " pod="openshift-ingress/payload-processing-e2e-inf-a-28f621-dd989bdc-9x66m"
Nov 02 15:25:45.190168 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:45.189671    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc422942-41f9-4cee-938f-e9df1b05d0c7-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-28f621-dd989bdc-9x66m\" (UID: \"dc422942-41f9-4cee-938f-e9df1b05d0c7\") " pod="openshift-ingress/payload-processing-e2e-inf-a-28f621-dd989bdc-9x66m"
Nov 02 15:25:45.190477 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:45.190454    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc422942-41f9-4cee-938f-e9df1b05d0c7-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-28f621-dd989bdc-9x66m\" (UID: \"dc422942-41f9-4cee-938f-e9df1b05d0c7\") " pod="openshift-ingress/payload-processing-e2e-inf-a-28f621-dd989bdc-9x66m"
Nov 02 15:25:45.223009 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:45.222971    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g8xlb\" (UniqueName: \"kubernetes.io/projected/dc422942-41f9-4cee-938f-e9df1b05d0c7-kube-api-access-g8xlb\") pod \"payload-processing-e2e-inf-a-28f621-dd989bdc-9x66m\" (UID: \"dc422942-41f9-4cee-938f-e9df1b05d0c7\") " pod="openshift-ingress/payload-processing-e2e-inf-a-28f621-dd989bdc-9x66m"
Nov 02 15:25:45.432277 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:45.432235    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-28f621-dd989bdc-9x66m"
Nov 02 15:25:45.749878 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:45.749850    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-28f621-dd989bdc-9x66m"]
Nov 02 15:25:45.751897 ip-10-0-135-138 kubenswrapper[5304]: W1102 15:25:45.751867    5304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc422942_41f9_4cee_938f_e9df1b05d0c7.slice/crio-acdfe4af7fa49549631af69f5bfd19d95832844d8433670d6ad076c79109f9c5 WatchSource:0}: Error finding container acdfe4af7fa49549631af69f5bfd19d95832844d8433670d6ad076c79109f9c5: Status 404 returned error can't find the container with id acdfe4af7fa49549631af69f5bfd19d95832844d8433670d6ad076c79109f9c5
Nov 02 15:25:46.310329 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:46.310286    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-28f621-dd989bdc-9x66m" event={"ID":"dc422942-41f9-4cee-938f-e9df1b05d0c7","Type":"ContainerStarted","Data":"6cf882f73d0856520ef43074ea73feb0d1d6276d15f6807cdc1c4dde65dbe01a"}
Nov 02 15:25:46.310765 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:46.310361    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-28f621-dd989bdc-9x66m"
Nov 02 15:25:46.310765 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:46.310383    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-28f621-dd989bdc-9x66m" event={"ID":"dc422942-41f9-4cee-938f-e9df1b05d0c7","Type":"ContainerStarted","Data":"acdfe4af7fa49549631af69f5bfd19d95832844d8433670d6ad076c79109f9c5"}
Nov 02 15:25:46.359322 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:46.359260    5304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-28f621-dd989bdc-9x66m" podStartSLOduration=2.359245308 podStartE2EDuration="2.359245308s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:25:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:25:46.355298055 +0000 UTC m=+1746.349122467" watchObservedRunningTime="2026-11-02 15:25:46.359245308 +0000 UTC m=+1746.353069717"
Nov 02 15:25:47.993588 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:47.993540    5304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-3daba5/test-model-3daba5-kserve-6f59d8687-qp658"
Nov 02 15:25:48.007014 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:48.006983    5304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-3daba5/test-model-3daba5-kserve-6f59d8687-qp658"
Nov 02 15:25:48.347320 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:48.347291    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-3daba5/test-model-3daba5-kserve-6f59d8687-qp658"
Nov 02 15:25:54.801119 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:25:54.801078    5304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k6fjk" containerName="extractor"
Nov 02 15:25:56.193328 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:56.193296    5304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-2abeaa-openshift-default-ccdd5d949-6fmzt"]
Nov 02 15:25:56.198226 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:56.198199    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-2abeaa-openshift-default-ccdd5d949-6fmzt"
Nov 02 15:25:56.228785 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:56.228752    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-2abeaa-openshift-default-ccdd5d949-6fmzt"]
Nov 02 15:25:56.237440 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:56.237406    5304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-2abeaa-openshift-default-dockercfg-smrsn\""
Nov 02 15:25:56.276372 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:56.276340    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa31b68f-cd14-41d9-92ca-2d1df613f263-istio-token\") pod \"e2e-inf-b-2abeaa-openshift-default-ccdd5d949-6fmzt\" (UID: \"aa31b68f-cd14-41d9-92ca-2d1df613f263\") " pod="openshift-ingress/e2e-inf-b-2abeaa-openshift-default-ccdd5d949-6fmzt"
Nov 02 15:25:56.276576 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:56.276381    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa31b68f-cd14-41d9-92ca-2d1df613f263-istio-data\") pod \"e2e-inf-b-2abeaa-openshift-default-ccdd5d949-6fmzt\" (UID: \"aa31b68f-cd14-41d9-92ca-2d1df613f263\") " pod="openshift-ingress/e2e-inf-b-2abeaa-openshift-default-ccdd5d949-6fmzt"
Nov 02 15:25:56.276576 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:56.276502    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa31b68f-cd14-41d9-92ca-2d1df613f263-workload-socket\") pod \"e2e-inf-b-2abeaa-openshift-default-ccdd5d949-6fmzt\" (UID: \"aa31b68f-cd14-41d9-92ca-2d1df613f263\") " pod="openshift-ingress/e2e-inf-b-2abeaa-openshift-default-ccdd5d949-6fmzt"
Nov 02 15:25:56.276576 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:56.276523    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gdscp\" (UniqueName: \"kubernetes.io/projected/aa31b68f-cd14-41d9-92ca-2d1df613f263-kube-api-access-gdscp\") pod \"e2e-inf-b-2abeaa-openshift-default-ccdd5d949-6fmzt\" (UID: \"aa31b68f-cd14-41d9-92ca-2d1df613f263\") " pod="openshift-ingress/e2e-inf-b-2abeaa-openshift-default-ccdd5d949-6fmzt"
Nov 02 15:25:56.276751 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:56.276723    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa31b68f-cd14-41d9-92ca-2d1df613f263-workload-certs\") pod \"e2e-inf-b-2abeaa-openshift-default-ccdd5d949-6fmzt\" (UID: \"aa31b68f-cd14-41d9-92ca-2d1df613f263\") " pod="openshift-ingress/e2e-inf-b-2abeaa-openshift-default-ccdd5d949-6fmzt"
Nov 02 15:25:56.276855 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:56.276836    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa31b68f-cd14-41d9-92ca-2d1df613f263-istio-podinfo\") pod \"e2e-inf-b-2abeaa-openshift-default-ccdd5d949-6fmzt\" (UID: \"aa31b68f-cd14-41d9-92ca-2d1df613f263\") " pod="openshift-ingress/e2e-inf-b-2abeaa-openshift-default-ccdd5d949-6fmzt"
Nov 02 15:25:56.276920 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:56.276905    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa31b68f-cd14-41d9-92ca-2d1df613f263-istiod-ca-cert\") pod \"e2e-inf-b-2abeaa-openshift-default-ccdd5d949-6fmzt\" (UID: \"aa31b68f-cd14-41d9-92ca-2d1df613f263\") " pod="openshift-ingress/e2e-inf-b-2abeaa-openshift-default-ccdd5d949-6fmzt"
Nov 02 15:25:56.276981 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:56.276960    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa31b68f-cd14-41d9-92ca-2d1df613f263-credential-socket\") pod \"e2e-inf-b-2abeaa-openshift-default-ccdd5d949-6fmzt\" (UID: \"aa31b68f-cd14-41d9-92ca-2d1df613f263\") " pod="openshift-ingress/e2e-inf-b-2abeaa-openshift-default-ccdd5d949-6fmzt"
Nov 02 15:25:56.277221 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:56.277195    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa31b68f-cd14-41d9-92ca-2d1df613f263-istio-envoy\") pod \"e2e-inf-b-2abeaa-openshift-default-ccdd5d949-6fmzt\" (UID: \"aa31b68f-cd14-41d9-92ca-2d1df613f263\") " pod="openshift-ingress/e2e-inf-b-2abeaa-openshift-default-ccdd5d949-6fmzt"
Nov 02 15:25:56.378242 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:56.378195    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa31b68f-cd14-41d9-92ca-2d1df613f263-workload-certs\") pod \"e2e-inf-b-2abeaa-openshift-default-ccdd5d949-6fmzt\" (UID: \"aa31b68f-cd14-41d9-92ca-2d1df613f263\") " pod="openshift-ingress/e2e-inf-b-2abeaa-openshift-default-ccdd5d949-6fmzt"
Nov 02 15:25:56.378472 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:56.378283    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa31b68f-cd14-41d9-92ca-2d1df613f263-istio-podinfo\") pod \"e2e-inf-b-2abeaa-openshift-default-ccdd5d949-6fmzt\" (UID: \"aa31b68f-cd14-41d9-92ca-2d1df613f263\") " pod="openshift-ingress/e2e-inf-b-2abeaa-openshift-default-ccdd5d949-6fmzt"
Nov 02 15:25:56.378472 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:56.378329    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa31b68f-cd14-41d9-92ca-2d1df613f263-istiod-ca-cert\") pod \"e2e-inf-b-2abeaa-openshift-default-ccdd5d949-6fmzt\" (UID: \"aa31b68f-cd14-41d9-92ca-2d1df613f263\") " pod="openshift-ingress/e2e-inf-b-2abeaa-openshift-default-ccdd5d949-6fmzt"
Nov 02 15:25:56.378472 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:56.378377    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa31b68f-cd14-41d9-92ca-2d1df613f263-credential-socket\") pod \"e2e-inf-b-2abeaa-openshift-default-ccdd5d949-6fmzt\" (UID: \"aa31b68f-cd14-41d9-92ca-2d1df613f263\") " pod="openshift-ingress/e2e-inf-b-2abeaa-openshift-default-ccdd5d949-6fmzt"
Nov 02 15:25:56.378472 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:56.378420    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa31b68f-cd14-41d9-92ca-2d1df613f263-istio-envoy\") pod \"e2e-inf-b-2abeaa-openshift-default-ccdd5d949-6fmzt\" (UID: \"aa31b68f-cd14-41d9-92ca-2d1df613f263\") " pod="openshift-ingress/e2e-inf-b-2abeaa-openshift-default-ccdd5d949-6fmzt"
Nov 02 15:25:56.378805 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:56.378478    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa31b68f-cd14-41d9-92ca-2d1df613f263-istio-token\") pod \"e2e-inf-b-2abeaa-openshift-default-ccdd5d949-6fmzt\" (UID: \"aa31b68f-cd14-41d9-92ca-2d1df613f263\") " pod="openshift-ingress/e2e-inf-b-2abeaa-openshift-default-ccdd5d949-6fmzt"
Nov 02 15:25:56.378805 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:56.378519    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa31b68f-cd14-41d9-92ca-2d1df613f263-istio-data\") pod \"e2e-inf-b-2abeaa-openshift-default-ccdd5d949-6fmzt\" (UID: \"aa31b68f-cd14-41d9-92ca-2d1df613f263\") " pod="openshift-ingress/e2e-inf-b-2abeaa-openshift-default-ccdd5d949-6fmzt"
Nov 02 15:25:56.378805 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:56.378589    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa31b68f-cd14-41d9-92ca-2d1df613f263-workload-socket\") pod \"e2e-inf-b-2abeaa-openshift-default-ccdd5d949-6fmzt\" (UID: \"aa31b68f-cd14-41d9-92ca-2d1df613f263\") " pod="openshift-ingress/e2e-inf-b-2abeaa-openshift-default-ccdd5d949-6fmzt"
Nov 02 15:25:56.378805 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:56.378620    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gdscp\" (UniqueName: \"kubernetes.io/projected/aa31b68f-cd14-41d9-92ca-2d1df613f263-kube-api-access-gdscp\") pod \"e2e-inf-b-2abeaa-openshift-default-ccdd5d949-6fmzt\" (UID: \"aa31b68f-cd14-41d9-92ca-2d1df613f263\") " pod="openshift-ingress/e2e-inf-b-2abeaa-openshift-default-ccdd5d949-6fmzt"
Nov 02 15:25:56.378805 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:56.378631    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa31b68f-cd14-41d9-92ca-2d1df613f263-workload-certs\") pod \"e2e-inf-b-2abeaa-openshift-default-ccdd5d949-6fmzt\" (UID: \"aa31b68f-cd14-41d9-92ca-2d1df613f263\") " pod="openshift-ingress/e2e-inf-b-2abeaa-openshift-default-ccdd5d949-6fmzt"
Nov 02 15:25:56.378805 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:56.378750    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa31b68f-cd14-41d9-92ca-2d1df613f263-credential-socket\") pod \"e2e-inf-b-2abeaa-openshift-default-ccdd5d949-6fmzt\" (UID: \"aa31b68f-cd14-41d9-92ca-2d1df613f263\") " pod="openshift-ingress/e2e-inf-b-2abeaa-openshift-default-ccdd5d949-6fmzt"
Nov 02 15:25:56.379159 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:56.378974    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa31b68f-cd14-41d9-92ca-2d1df613f263-istio-data\") pod \"e2e-inf-b-2abeaa-openshift-default-ccdd5d949-6fmzt\" (UID: \"aa31b68f-cd14-41d9-92ca-2d1df613f263\") " pod="openshift-ingress/e2e-inf-b-2abeaa-openshift-default-ccdd5d949-6fmzt"
Nov 02 15:25:56.379222 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:56.379163    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa31b68f-cd14-41d9-92ca-2d1df613f263-workload-socket\") pod \"e2e-inf-b-2abeaa-openshift-default-ccdd5d949-6fmzt\" (UID: \"aa31b68f-cd14-41d9-92ca-2d1df613f263\") " pod="openshift-ingress/e2e-inf-b-2abeaa-openshift-default-ccdd5d949-6fmzt"
Nov 02 15:25:56.379364 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:56.379344    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa31b68f-cd14-41d9-92ca-2d1df613f263-istiod-ca-cert\") pod \"e2e-inf-b-2abeaa-openshift-default-ccdd5d949-6fmzt\" (UID: \"aa31b68f-cd14-41d9-92ca-2d1df613f263\") " pod="openshift-ingress/e2e-inf-b-2abeaa-openshift-default-ccdd5d949-6fmzt"
Nov 02 15:25:56.380649 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:56.380628    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa31b68f-cd14-41d9-92ca-2d1df613f263-istio-envoy\") pod \"e2e-inf-b-2abeaa-openshift-default-ccdd5d949-6fmzt\" (UID: \"aa31b68f-cd14-41d9-92ca-2d1df613f263\") " pod="openshift-ingress/e2e-inf-b-2abeaa-openshift-default-ccdd5d949-6fmzt"
Nov 02 15:25:56.380962 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:56.380943    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa31b68f-cd14-41d9-92ca-2d1df613f263-istio-podinfo\") pod \"e2e-inf-b-2abeaa-openshift-default-ccdd5d949-6fmzt\" (UID: \"aa31b68f-cd14-41d9-92ca-2d1df613f263\") " pod="openshift-ingress/e2e-inf-b-2abeaa-openshift-default-ccdd5d949-6fmzt"
Nov 02 15:25:56.403822 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:56.403796    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa31b68f-cd14-41d9-92ca-2d1df613f263-istio-token\") pod \"e2e-inf-b-2abeaa-openshift-default-ccdd5d949-6fmzt\" (UID: \"aa31b68f-cd14-41d9-92ca-2d1df613f263\") " pod="openshift-ingress/e2e-inf-b-2abeaa-openshift-default-ccdd5d949-6fmzt"
Nov 02 15:25:56.404103 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:56.404084    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gdscp\" (UniqueName: \"kubernetes.io/projected/aa31b68f-cd14-41d9-92ca-2d1df613f263-kube-api-access-gdscp\") pod \"e2e-inf-b-2abeaa-openshift-default-ccdd5d949-6fmzt\" (UID: \"aa31b68f-cd14-41d9-92ca-2d1df613f263\") " pod="openshift-ingress/e2e-inf-b-2abeaa-openshift-default-ccdd5d949-6fmzt"
Nov 02 15:25:56.514391 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:56.514309    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-2abeaa-openshift-default-ccdd5d949-6fmzt"
Nov 02 15:25:56.855331 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:56.852084    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-2abeaa-openshift-default-ccdd5d949-6fmzt"]
Nov 02 15:25:56.855331 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:56.854759    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:25:56.855331 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:56.854910    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:25:56.855331 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:56.854999    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:25:57.322308 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:57.322227    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-28f621-dd989bdc-9x66m"
Nov 02 15:25:57.408072 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:57.408033    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-2abeaa-openshift-default-ccdd5d949-6fmzt" event={"ID":"aa31b68f-cd14-41d9-92ca-2d1df613f263","Type":"ContainerStarted","Data":"cacbcf86b32c08d520cac3f78168b566738508a647c8224c74f5c07655bf3b7e"}
Nov 02 15:25:57.408295 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:57.408209    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-2abeaa-openshift-default-ccdd5d949-6fmzt" event={"ID":"aa31b68f-cd14-41d9-92ca-2d1df613f263","Type":"ContainerStarted","Data":"d34aee5b205baeb0b7a7c30e51622fdd30adc7e9d6f88fca828b4a7b38041b2a"}
Nov 02 15:25:57.459000 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:57.458943    5304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-2abeaa-openshift-default-ccdd5d949-6fmzt" podStartSLOduration=1.458929645 podStartE2EDuration="1.458929645s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:25:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:25:57.457871992 +0000 UTC m=+1757.451696401" watchObservedRunningTime="2026-11-02 15:25:57.458929645 +0000 UTC m=+1757.452754054"
Nov 02 15:25:57.519728 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:57.519431    5304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-2abeaa-openshift-default-ccdd5d949-6fmzt"
Nov 02 15:25:57.523853 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:57.523825    5304 patch_prober.go:29] interesting pod/e2e-inf-b-2abeaa-openshift-default-ccdd5d949-6fmzt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" start-of-body=
Nov 02 15:25:57.524022 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:57.523879    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-2abeaa-openshift-default-ccdd5d949-6fmzt" podUID="aa31b68f-cd14-41d9-92ca-2d1df613f263" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused"
Nov 02 15:25:58.514945 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:58.514913    5304 patch_prober.go:29] interesting pod/e2e-inf-b-2abeaa-openshift-default-ccdd5d949-6fmzt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" start-of-body=
Nov 02 15:25:58.515356 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:58.514978    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-2abeaa-openshift-default-ccdd5d949-6fmzt" podUID="aa31b68f-cd14-41d9-92ca-2d1df613f263" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused"
Nov 02 15:25:59.514992 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:59.514953    5304 patch_prober.go:29] interesting pod/e2e-inf-b-2abeaa-openshift-default-ccdd5d949-6fmzt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" start-of-body=
Nov 02 15:25:59.515438 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:25:59.515008    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-2abeaa-openshift-default-ccdd5d949-6fmzt" podUID="aa31b68f-cd14-41d9-92ca-2d1df613f263" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused"
Nov 02 15:26:00.515185 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:26:00.515147    5304 patch_prober.go:29] interesting pod/e2e-inf-b-2abeaa-openshift-default-ccdd5d949-6fmzt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" start-of-body=
Nov 02 15:26:00.515608 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:26:00.515206    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-2abeaa-openshift-default-ccdd5d949-6fmzt" podUID="aa31b68f-cd14-41d9-92ca-2d1df613f263" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused"
Nov 02 15:26:01.515318 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:26:01.515280    5304 patch_prober.go:29] interesting pod/e2e-inf-b-2abeaa-openshift-default-ccdd5d949-6fmzt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" start-of-body=
Nov 02 15:26:01.515814 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:26:01.515336    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-2abeaa-openshift-default-ccdd5d949-6fmzt" podUID="aa31b68f-cd14-41d9-92ca-2d1df613f263" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused"
Nov 02 15:26:02.514936 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:26:02.514893    5304 patch_prober.go:29] interesting pod/e2e-inf-b-2abeaa-openshift-default-ccdd5d949-6fmzt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" start-of-body=
Nov 02 15:26:02.515164 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:26:02.514955    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-2abeaa-openshift-default-ccdd5d949-6fmzt" podUID="aa31b68f-cd14-41d9-92ca-2d1df613f263" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused"
Nov 02 15:26:03.576438 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:26:03.575572    5304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-2abeaa-openshift-default-ccdd5d949-6fmzt"
Nov 02 15:26:03.576438 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:26:03.576143    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-2abeaa-openshift-default-ccdd5d949-6fmzt"
Nov 02 15:26:03.578728 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:26:03.578705    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-2abeaa-openshift-default-ccdd5d949-6fmzt"
Nov 02 15:26:24.800312 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:26:24.800267    5304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k6fjk" containerName="extractor"
Nov 02 15:26:24.963888 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:26:24.963852    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-4ec02d-openshift-default-86955cc5c9-2j2bw"]
Nov 02 15:26:24.964177 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:26:24.964145    5304 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-4ec02d-openshift-default-86955cc5c9-2j2bw" podUID="de0eb9f2-e614-454a-a294-77fc9830a610" containerName="istio-proxy" containerID="cri-o://8f6743d57906831a955d6819bf57b7df840590afd90ae35e0879082811e3ecc3" gracePeriod=30
Nov 02 15:26:26.918961 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:26:26.918886    5304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-80ec03-openshift-default-66dfdbcf85-b4ltb"]
Nov 02 15:26:26.926726 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:26:26.926695    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-80ec03-openshift-default-66dfdbcf85-b4ltb"
Nov 02 15:26:26.946939 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:26:26.946889    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-80ec03-openshift-default-66dfdbcf85-b4ltb"]
Nov 02 15:26:26.953932 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:26:26.953905    5304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-80ec03-openshift-default-dockercfg-fnn49\""
Nov 02 15:26:27.082744 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:26:27.082701    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f9d68a5-7f9c-489b-9289-2f85b7889c2c-credential-socket\") pod \"e2e-inf-a-80ec03-openshift-default-66dfdbcf85-b4ltb\" (UID: \"1f9d68a5-7f9c-489b-9289-2f85b7889c2c\") " pod="openshift-ingress/e2e-inf-a-80ec03-openshift-default-66dfdbcf85-b4ltb"
Nov 02 15:26:27.094879 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:26:27.094832    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f9d68a5-7f9c-489b-9289-2f85b7889c2c-istio-envoy\") pod \"e2e-inf-a-80ec03-openshift-default-66dfdbcf85-b4ltb\" (UID: \"1f9d68a5-7f9c-489b-9289-2f85b7889c2c\") " pod="openshift-ingress/e2e-inf-a-80ec03-openshift-default-66dfdbcf85-b4ltb"
Nov 02 15:26:27.095257 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:26:27.095236    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f9d68a5-7f9c-489b-9289-2f85b7889c2c-workload-certs\") pod \"e2e-inf-a-80ec03-openshift-default-66dfdbcf85-b4ltb\" (UID: \"1f9d68a5-7f9c-489b-9289-2f85b7889c2c\") " pod="openshift-ingress/e2e-inf-a-80ec03-openshift-default-66dfdbcf85-b4ltb"
Nov 02 15:26:27.095542 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:26:27.095523    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f9d68a5-7f9c-489b-9289-2f85b7889c2c-istio-token\") pod \"e2e-inf-a-80ec03-openshift-default-66dfdbcf85-b4ltb\" (UID: \"1f9d68a5-7f9c-489b-9289-2f85b7889c2c\") " pod="openshift-ingress/e2e-inf-a-80ec03-openshift-default-66dfdbcf85-b4ltb"
Nov 02 15:26:27.095759 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:26:27.095742    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f9d68a5-7f9c-489b-9289-2f85b7889c2c-istio-podinfo\") pod \"e2e-inf-a-80ec03-openshift-default-66dfdbcf85-b4ltb\" (UID: \"1f9d68a5-7f9c-489b-9289-2f85b7889c2c\") " pod="openshift-ingress/e2e-inf-a-80ec03-openshift-default-66dfdbcf85-b4ltb"
Nov 02 15:26:27.095934 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:26:27.095918    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f9d68a5-7f9c-489b-9289-2f85b7889c2c-istio-data\") pod \"e2e-inf-a-80ec03-openshift-default-66dfdbcf85-b4ltb\" (UID: \"1f9d68a5-7f9c-489b-9289-2f85b7889c2c\") " pod="openshift-ingress/e2e-inf-a-80ec03-openshift-default-66dfdbcf85-b4ltb"
Nov 02 15:26:27.096203 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:26:27.096183    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tt84c\" (UniqueName: \"kubernetes.io/projected/1f9d68a5-7f9c-489b-9289-2f85b7889c2c-kube-api-access-tt84c\") pod \"e2e-inf-a-80ec03-openshift-default-66dfdbcf85-b4ltb\" (UID: \"1f9d68a5-7f9c-489b-9289-2f85b7889c2c\") " pod="openshift-ingress/e2e-inf-a-80ec03-openshift-default-66dfdbcf85-b4ltb"
Nov 02 15:26:27.096436 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:26:27.096410    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f9d68a5-7f9c-489b-9289-2f85b7889c2c-istiod-ca-cert\") pod \"e2e-inf-a-80ec03-openshift-default-66dfdbcf85-b4ltb\" (UID: \"1f9d68a5-7f9c-489b-9289-2f85b7889c2c\") " pod="openshift-ingress/e2e-inf-a-80ec03-openshift-default-66dfdbcf85-b4ltb"
Nov 02 15:26:27.096646 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:26:27.096627    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f9d68a5-7f9c-489b-9289-2f85b7889c2c-workload-socket\") pod \"e2e-inf-a-80ec03-openshift-default-66dfdbcf85-b4ltb\" (UID: \"1f9d68a5-7f9c-489b-9289-2f85b7889c2c\") " pod="openshift-ingress/e2e-inf-a-80ec03-openshift-default-66dfdbcf85-b4ltb"
Nov 02 15:26:27.219206 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:26:27.218870    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f9d68a5-7f9c-489b-9289-2f85b7889c2c-workload-certs\") pod \"e2e-inf-a-80ec03-openshift-default-66dfdbcf85-b4ltb\" (UID: \"1f9d68a5-7f9c-489b-9289-2f85b7889c2c\") " pod="openshift-ingress/e2e-inf-a-80ec03-openshift-default-66dfdbcf85-b4ltb"
Nov 02 15:26:27.219206 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:26:27.219015    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f9d68a5-7f9c-489b-9289-2f85b7889c2c-istio-token\") pod \"e2e-inf-a-80ec03-openshift-default-66dfdbcf85-b4ltb\" (UID: \"1f9d68a5-7f9c-489b-9289-2f85b7889c2c\") " pod="openshift-ingress/e2e-inf-a-80ec03-openshift-default-66dfdbcf85-b4ltb"
Nov 02 15:26:27.219206 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:26:27.219070    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f9d68a5-7f9c-489b-9289-2f85b7889c2c-istio-podinfo\") pod \"e2e-inf-a-80ec03-openshift-default-66dfdbcf85-b4ltb\" (UID: \"1f9d68a5-7f9c-489b-9289-2f85b7889c2c\") " pod="openshift-ingress/e2e-inf-a-80ec03-openshift-default-66dfdbcf85-b4ltb"
Nov 02 15:26:27.219206 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:26:27.219097    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f9d68a5-7f9c-489b-9289-2f85b7889c2c-istio-data\") pod \"e2e-inf-a-80ec03-openshift-default-66dfdbcf85-b4ltb\" (UID: \"1f9d68a5-7f9c-489b-9289-2f85b7889c2c\") " pod="openshift-ingress/e2e-inf-a-80ec03-openshift-default-66dfdbcf85-b4ltb"
Nov 02 15:26:27.219206 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:26:27.219197    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tt84c\" (UniqueName: \"kubernetes.io/projected/1f9d68a5-7f9c-489b-9289-2f85b7889c2c-kube-api-access-tt84c\") pod \"e2e-inf-a-80ec03-openshift-default-66dfdbcf85-b4ltb\" (UID: \"1f9d68a5-7f9c-489b-9289-2f85b7889c2c\") " pod="openshift-ingress/e2e-inf-a-80ec03-openshift-default-66dfdbcf85-b4ltb"
Nov 02 15:26:27.219601 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:26:27.219248    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f9d68a5-7f9c-489b-9289-2f85b7889c2c-istiod-ca-cert\") pod \"e2e-inf-a-80ec03-openshift-default-66dfdbcf85-b4ltb\" (UID: \"1f9d68a5-7f9c-489b-9289-2f85b7889c2c\") " pod="openshift-ingress/e2e-inf-a-80ec03-openshift-default-66dfdbcf85-b4ltb"
Nov 02 15:26:27.219601 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:26:27.219295    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f9d68a5-7f9c-489b-9289-2f85b7889c2c-workload-socket\") pod \"e2e-inf-a-80ec03-openshift-default-66dfdbcf85-b4ltb\" (UID: \"1f9d68a5-7f9c-489b-9289-2f85b7889c2c\") " pod="openshift-ingress/e2e-inf-a-80ec03-openshift-default-66dfdbcf85-b4ltb"
Nov 02 15:26:27.219601 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:26:27.219528    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f9d68a5-7f9c-489b-9289-2f85b7889c2c-credential-socket\") pod \"e2e-inf-a-80ec03-openshift-default-66dfdbcf85-b4ltb\" (UID: \"1f9d68a5-7f9c-489b-9289-2f85b7889c2c\") " pod="openshift-ingress/e2e-inf-a-80ec03-openshift-default-66dfdbcf85-b4ltb"
Nov 02 15:26:27.219601 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:26:27.219578    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f9d68a5-7f9c-489b-9289-2f85b7889c2c-istio-envoy\") pod \"e2e-inf-a-80ec03-openshift-default-66dfdbcf85-b4ltb\" (UID: \"1f9d68a5-7f9c-489b-9289-2f85b7889c2c\") " pod="openshift-ingress/e2e-inf-a-80ec03-openshift-default-66dfdbcf85-b4ltb"
Nov 02 15:26:27.220992 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:26:27.220913    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f9d68a5-7f9c-489b-9289-2f85b7889c2c-istio-data\") pod \"e2e-inf-a-80ec03-openshift-default-66dfdbcf85-b4ltb\" (UID: \"1f9d68a5-7f9c-489b-9289-2f85b7889c2c\") " pod="openshift-ingress/e2e-inf-a-80ec03-openshift-default-66dfdbcf85-b4ltb"
Nov 02 15:26:27.226010 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:26:27.221455    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f9d68a5-7f9c-489b-9289-2f85b7889c2c-istiod-ca-cert\") pod \"e2e-inf-a-80ec03-openshift-default-66dfdbcf85-b4ltb\" (UID: \"1f9d68a5-7f9c-489b-9289-2f85b7889c2c\") " pod="openshift-ingress/e2e-inf-a-80ec03-openshift-default-66dfdbcf85-b4ltb"
Nov 02 15:26:27.226010 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:26:27.222463    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f9d68a5-7f9c-489b-9289-2f85b7889c2c-workload-certs\") pod \"e2e-inf-a-80ec03-openshift-default-66dfdbcf85-b4ltb\" (UID: \"1f9d68a5-7f9c-489b-9289-2f85b7889c2c\") " pod="openshift-ingress/e2e-inf-a-80ec03-openshift-default-66dfdbcf85-b4ltb"
Nov 02 15:26:27.226010 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:26:27.222695    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f9d68a5-7f9c-489b-9289-2f85b7889c2c-workload-socket\") pod \"e2e-inf-a-80ec03-openshift-default-66dfdbcf85-b4ltb\" (UID: \"1f9d68a5-7f9c-489b-9289-2f85b7889c2c\") " pod="openshift-ingress/e2e-inf-a-80ec03-openshift-default-66dfdbcf85-b4ltb"
Nov 02 15:26:27.226010 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:26:27.222738    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f9d68a5-7f9c-489b-9289-2f85b7889c2c-credential-socket\") pod \"e2e-inf-a-80ec03-openshift-default-66dfdbcf85-b4ltb\" (UID: \"1f9d68a5-7f9c-489b-9289-2f85b7889c2c\") " pod="openshift-ingress/e2e-inf-a-80ec03-openshift-default-66dfdbcf85-b4ltb"
Nov 02 15:26:27.239110 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:26:27.235429    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f9d68a5-7f9c-489b-9289-2f85b7889c2c-istio-podinfo\") pod \"e2e-inf-a-80ec03-openshift-default-66dfdbcf85-b4ltb\" (UID: \"1f9d68a5-7f9c-489b-9289-2f85b7889c2c\") " pod="openshift-ingress/e2e-inf-a-80ec03-openshift-default-66dfdbcf85-b4ltb"
Nov 02 15:26:27.239110 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:26:27.237282    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f9d68a5-7f9c-489b-9289-2f85b7889c2c-istio-envoy\") pod \"e2e-inf-a-80ec03-openshift-default-66dfdbcf85-b4ltb\" (UID: \"1f9d68a5-7f9c-489b-9289-2f85b7889c2c\") " pod="openshift-ingress/e2e-inf-a-80ec03-openshift-default-66dfdbcf85-b4ltb"
Nov 02 15:26:27.273236 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:26:27.268871    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f9d68a5-7f9c-489b-9289-2f85b7889c2c-istio-token\") pod \"e2e-inf-a-80ec03-openshift-default-66dfdbcf85-b4ltb\" (UID: \"1f9d68a5-7f9c-489b-9289-2f85b7889c2c\") " pod="openshift-ingress/e2e-inf-a-80ec03-openshift-default-66dfdbcf85-b4ltb"
Nov 02 15:26:27.289149 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:26:27.283331    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tt84c\" (UniqueName: \"kubernetes.io/projected/1f9d68a5-7f9c-489b-9289-2f85b7889c2c-kube-api-access-tt84c\") pod \"e2e-inf-a-80ec03-openshift-default-66dfdbcf85-b4ltb\" (UID: \"1f9d68a5-7f9c-489b-9289-2f85b7889c2c\") " pod="openshift-ingress/e2e-inf-a-80ec03-openshift-default-66dfdbcf85-b4ltb"
Nov 02 15:26:27.557740 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:26:27.557652    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-80ec03-openshift-default-66dfdbcf85-b4ltb"
Nov 02 15:26:27.889049 ip-10-0-135-138 kubenswrapper[5304]: W1102 15:26:27.889020    5304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f9d68a5_7f9c_489b_9289_2f85b7889c2c.slice/crio-c2318f9bfe896666b0892b745d6798be109ccd624bea6c987bcc744f05d042e6 WatchSource:0}: Error finding container c2318f9bfe896666b0892b745d6798be109ccd624bea6c987bcc744f05d042e6: Status 404 returned error can't find the container with id c2318f9bfe896666b0892b745d6798be109ccd624bea6c987bcc744f05d042e6
Nov 02 15:26:27.889784 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:26:27.889762    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-80ec03-openshift-default-66dfdbcf85-b4ltb"]
Nov 02 15:26:27.892017 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:26:27.891984    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:26:27.892072 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:26:27.892062    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:26:27.892110 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:26:27.892095    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:26:28.747029 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:26:28.746991    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-80ec03-openshift-default-66dfdbcf85-b4ltb" event={"ID":"1f9d68a5-7f9c-489b-9289-2f85b7889c2c","Type":"ContainerStarted","Data":"ca4848c3f3277089ca035016514b458f2450114dc6a79e6b0ef5191a3518abc8"}
Nov 02 15:26:28.747542 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:26:28.747054    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-80ec03-openshift-default-66dfdbcf85-b4ltb" event={"ID":"1f9d68a5-7f9c-489b-9289-2f85b7889c2c","Type":"ContainerStarted","Data":"c2318f9bfe896666b0892b745d6798be109ccd624bea6c987bcc744f05d042e6"}
Nov 02 15:26:28.813066 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:26:28.813010    5304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-80ec03-openshift-default-66dfdbcf85-b4ltb" podStartSLOduration=2.81299271 podStartE2EDuration="2.81299271s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:26:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:26:28.811538514 +0000 UTC m=+1788.805362925" watchObservedRunningTime="2026-11-02 15:26:28.81299271 +0000 UTC m=+1788.806817120"
Nov 02 15:26:29.558790 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:26:29.558754    5304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-80ec03-openshift-default-66dfdbcf85-b4ltb"
Nov 02 15:26:29.565406 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:26:29.565379    5304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-80ec03-openshift-default-66dfdbcf85-b4ltb"
Nov 02 15:26:29.753709 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:26:29.753672    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-80ec03-openshift-default-66dfdbcf85-b4ltb"
Nov 02 15:26:29.755289 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:26:29.755265    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-80ec03-openshift-default-66dfdbcf85-b4ltb"
Nov 02 15:26:30.254250 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:26:30.254224    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-4ec02d-openshift-default-86955cc5c9-2j2bw"
Nov 02 15:26:30.373074 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:26:30.373036    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de0eb9f2-e614-454a-a294-77fc9830a610-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de0eb9f2-e614-454a-a294-77fc9830a610-istio-envoy\") pod \"de0eb9f2-e614-454a-a294-77fc9830a610\" (UID: \"de0eb9f2-e614-454a-a294-77fc9830a610\") "
Nov 02 15:26:30.373293 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:26:30.373154    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de0eb9f2-e614-454a-a294-77fc9830a610-istio-token\" (UniqueName: \"kubernetes.io/projected/de0eb9f2-e614-454a-a294-77fc9830a610-istio-token\") pod \"de0eb9f2-e614-454a-a294-77fc9830a610\" (UID: \"de0eb9f2-e614-454a-a294-77fc9830a610\") "
Nov 02 15:26:30.373293 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:26:30.373281    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/de0eb9f2-e614-454a-a294-77fc9830a610-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de0eb9f2-e614-454a-a294-77fc9830a610-istio-podinfo\") pod \"de0eb9f2-e614-454a-a294-77fc9830a610\" (UID: \"de0eb9f2-e614-454a-a294-77fc9830a610\") "
Nov 02 15:26:30.373430 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:26:30.373415    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de0eb9f2-e614-454a-a294-77fc9830a610-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de0eb9f2-e614-454a-a294-77fc9830a610-workload-certs\") pod \"de0eb9f2-e614-454a-a294-77fc9830a610\" (UID: \"de0eb9f2-e614-454a-a294-77fc9830a610\") "
Nov 02 15:26:30.373584 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:26:30.373570    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de0eb9f2-e614-454a-a294-77fc9830a610-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de0eb9f2-e614-454a-a294-77fc9830a610-credential-socket\") pod \"de0eb9f2-e614-454a-a294-77fc9830a610\" (UID: \"de0eb9f2-e614-454a-a294-77fc9830a610\") "
Nov 02 15:26:30.373679 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:26:30.373649    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de0eb9f2-e614-454a-a294-77fc9830a610-workload-certs" pod "de0eb9f2-e614-454a-a294-77fc9830a610" (UID: "de0eb9f2-e614-454a-a294-77fc9830a610"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:26:30.373739 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:26:30.373678    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de0eb9f2-e614-454a-a294-77fc9830a610-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de0eb9f2-e614-454a-a294-77fc9830a610-workload-socket\") pod \"de0eb9f2-e614-454a-a294-77fc9830a610\" (UID: \"de0eb9f2-e614-454a-a294-77fc9830a610\") "
Nov 02 15:26:30.373797 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:26:30.373740    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de0eb9f2-e614-454a-a294-77fc9830a610-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de0eb9f2-e614-454a-a294-77fc9830a610-istio-data\") pod \"de0eb9f2-e614-454a-a294-77fc9830a610\" (UID: \"de0eb9f2-e614-454a-a294-77fc9830a610\") "
Nov 02 15:26:30.373797 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:26:30.373775    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de0eb9f2-e614-454a-a294-77fc9830a610-kube-api-access-zfj5n\" (UniqueName: \"kubernetes.io/projected/de0eb9f2-e614-454a-a294-77fc9830a610-kube-api-access-zfj5n\") pod \"de0eb9f2-e614-454a-a294-77fc9830a610\" (UID: \"de0eb9f2-e614-454a-a294-77fc9830a610\") "
Nov 02 15:26:30.373902 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:26:30.373871    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de0eb9f2-e614-454a-a294-77fc9830a610-credential-socket" pod "de0eb9f2-e614-454a-a294-77fc9830a610" (UID: "de0eb9f2-e614-454a-a294-77fc9830a610"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:26:30.373955 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:26:30.373894    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/de0eb9f2-e614-454a-a294-77fc9830a610-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de0eb9f2-e614-454a-a294-77fc9830a610-istiod-ca-cert\") pod \"de0eb9f2-e614-454a-a294-77fc9830a610\" (UID: \"de0eb9f2-e614-454a-a294-77fc9830a610\") "
Nov 02 15:26:30.373955 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:26:30.373916    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de0eb9f2-e614-454a-a294-77fc9830a610-workload-socket" pod "de0eb9f2-e614-454a-a294-77fc9830a610" (UID: "de0eb9f2-e614-454a-a294-77fc9830a610"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:26:30.374084 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:26:30.374059    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de0eb9f2-e614-454a-a294-77fc9830a610-istio-data" pod "de0eb9f2-e614-454a-a294-77fc9830a610" (UID: "de0eb9f2-e614-454a-a294-77fc9830a610"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:26:30.374386 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:26:30.374356    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de0eb9f2-e614-454a-a294-77fc9830a610-istiod-ca-cert" pod "de0eb9f2-e614-454a-a294-77fc9830a610" (UID: "de0eb9f2-e614-454a-a294-77fc9830a610"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:26:30.374720 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:26:30.374696    5304 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de0eb9f2-e614-454a-a294-77fc9830a610-workload-certs\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:26:30.374803 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:26:30.374726    5304 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de0eb9f2-e614-454a-a294-77fc9830a610-credential-socket\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:26:30.374803 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:26:30.374742    5304 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de0eb9f2-e614-454a-a294-77fc9830a610-workload-socket\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:26:30.374803 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:26:30.374757    5304 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de0eb9f2-e614-454a-a294-77fc9830a610-istio-data\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:26:30.374803 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:26:30.374770    5304 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de0eb9f2-e614-454a-a294-77fc9830a610-istiod-ca-cert\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:26:30.375612 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:26:30.375588    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de0eb9f2-e614-454a-a294-77fc9830a610-istio-token" pod "de0eb9f2-e614-454a-a294-77fc9830a610" (UID: "de0eb9f2-e614-454a-a294-77fc9830a610"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:26:30.375695 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:26:30.375656    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/de0eb9f2-e614-454a-a294-77fc9830a610-istio-podinfo" pod "de0eb9f2-e614-454a-a294-77fc9830a610" (UID: "de0eb9f2-e614-454a-a294-77fc9830a610"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 15:26:30.375761 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:26:30.375744    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de0eb9f2-e614-454a-a294-77fc9830a610-istio-envoy" pod "de0eb9f2-e614-454a-a294-77fc9830a610" (UID: "de0eb9f2-e614-454a-a294-77fc9830a610"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:26:30.375945 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:26:30.375928    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de0eb9f2-e614-454a-a294-77fc9830a610-kube-api-access-zfj5n" pod "de0eb9f2-e614-454a-a294-77fc9830a610" (UID: "de0eb9f2-e614-454a-a294-77fc9830a610"). InnerVolumeSpecName "kube-api-access-zfj5n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:26:30.476159 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:26:30.476090    5304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zfj5n\" (UniqueName: \"kubernetes.io/projected/de0eb9f2-e614-454a-a294-77fc9830a610-kube-api-access-zfj5n\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:26:30.476159 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:26:30.476122    5304 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de0eb9f2-e614-454a-a294-77fc9830a610-istio-envoy\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:26:30.476159 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:26:30.476144    5304 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de0eb9f2-e614-454a-a294-77fc9830a610-istio-token\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:26:30.476159 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:26:30.476153    5304 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de0eb9f2-e614-454a-a294-77fc9830a610-istio-podinfo\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:26:30.762018 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:26:30.761983    5304 generic.go:356] "Generic (PLEG): container finished" podID="de0eb9f2-e614-454a-a294-77fc9830a610" containerID="8f6743d57906831a955d6819bf57b7df840590afd90ae35e0879082811e3ecc3" exitCode=0
Nov 02 15:26:30.762527 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:26:30.762079    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-4ec02d-openshift-default-86955cc5c9-2j2bw"
Nov 02 15:26:30.762527 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:26:30.762107    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-4ec02d-openshift-default-86955cc5c9-2j2bw" event={"ID":"de0eb9f2-e614-454a-a294-77fc9830a610","Type":"ContainerDied","Data":"8f6743d57906831a955d6819bf57b7df840590afd90ae35e0879082811e3ecc3"}
Nov 02 15:26:30.762527 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:26:30.762180    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-4ec02d-openshift-default-86955cc5c9-2j2bw" event={"ID":"de0eb9f2-e614-454a-a294-77fc9830a610","Type":"ContainerDied","Data":"2347819a82b76a47c127de3314c770b3dff2a928f35aaa3eb03b8bfb9d28063d"}
Nov 02 15:26:30.762527 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:26:30.762212    5304 scope.go:122] "RemoveContainer" containerID="8f6743d57906831a955d6819bf57b7df840590afd90ae35e0879082811e3ecc3"
Nov 02 15:26:30.774763 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:26:30.774733    5304 scope.go:122] "RemoveContainer" containerID="9f2169beab40a8765fdfcbb363eb57d18bd15f131072d9c4e777d228c1dfabff"
Nov 02 15:26:30.784745 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:26:30.784721    5304 scope.go:122] "RemoveContainer" containerID="8f6743d57906831a955d6819bf57b7df840590afd90ae35e0879082811e3ecc3"
Nov 02 15:26:30.785113 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:26:30.785083    5304 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f6743d57906831a955d6819bf57b7df840590afd90ae35e0879082811e3ecc3\": container with ID starting with 8f6743d57906831a955d6819bf57b7df840590afd90ae35e0879082811e3ecc3 not found: ID does not exist" containerID="8f6743d57906831a955d6819bf57b7df840590afd90ae35e0879082811e3ecc3"
Nov 02 15:26:30.785245 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:26:30.785192    5304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f6743d57906831a955d6819bf57b7df840590afd90ae35e0879082811e3ecc3"} err="failed to get container status \"8f6743d57906831a955d6819bf57b7df840590afd90ae35e0879082811e3ecc3\": rpc error: code = NotFound desc = could not find container \"8f6743d57906831a955d6819bf57b7df840590afd90ae35e0879082811e3ecc3\": container with ID starting with 8f6743d57906831a955d6819bf57b7df840590afd90ae35e0879082811e3ecc3 not found: ID does not exist"
Nov 02 15:26:30.785245 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:26:30.785223    5304 scope.go:122] "RemoveContainer" containerID="9f2169beab40a8765fdfcbb363eb57d18bd15f131072d9c4e777d228c1dfabff"
Nov 02 15:26:30.785492 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:26:30.785469    5304 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f2169beab40a8765fdfcbb363eb57d18bd15f131072d9c4e777d228c1dfabff\": container with ID starting with 9f2169beab40a8765fdfcbb363eb57d18bd15f131072d9c4e777d228c1dfabff not found: ID does not exist" containerID="9f2169beab40a8765fdfcbb363eb57d18bd15f131072d9c4e777d228c1dfabff"
Nov 02 15:26:30.785553 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:26:30.785504    5304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f2169beab40a8765fdfcbb363eb57d18bd15f131072d9c4e777d228c1dfabff"} err="failed to get container status \"9f2169beab40a8765fdfcbb363eb57d18bd15f131072d9c4e777d228c1dfabff\": rpc error: code = NotFound desc = could not find container \"9f2169beab40a8765fdfcbb363eb57d18bd15f131072d9c4e777d228c1dfabff\": container with ID starting with 9f2169beab40a8765fdfcbb363eb57d18bd15f131072d9c4e777d228c1dfabff not found: ID does not exist"
Nov 02 15:26:30.837058 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:26:30.837032    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-4ec02d-openshift-default-86955cc5c9-2j2bw"]
Nov 02 15:26:30.851079 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:26:30.851047    5304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-4ec02d-openshift-default-86955cc5c9-2j2bw"]
Nov 02 15:26:32.120496 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:26:32.120463    5304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="de0eb9f2-e614-454a-a294-77fc9830a610" path="/var/lib/kubelet/pods/de0eb9f2-e614-454a-a294-77fc9830a610/volumes"
Nov 02 15:26:38.388138 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:26:38.386220    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-624254/distinct-w3-624254-kserve-5465dc4765-dmq67" podUID="1a5aa910-a6de-417b-8638-81e4ec5c1825" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:26:38.707204 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:26:38.706440    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-f975ed/test-model-f975ed-kserve-54cff7ccbc-9w5br" podUID="02bfbeb4-6e08-4839-b1e3-822a84cbb9c9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.123:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:26:38.741297 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:26:38.741238    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-739bf1/simulator-w2-739bf1-kserve-7c74f85465-ltk4t" podUID="cf6a1ded-28a3-49eb-878b-b4ad77a0bb5c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:26:39.000600 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:26:39.000503    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-2bbcb5/test-model-2bbcb5-kserve-9597db49f-g6cdn" podUID="f3c417d3-fdf3-4951-8246-dcd24d7e3820" containerName="main" probeResult="failure" output="Get \"https://10.133.0.119:8000/ready\": context deadline exceeded"
Nov 02 15:26:39.058355 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:26:39.058291    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-a03e84/test-model-a03e84-kserve-554589b56f-tq4bb" podUID="917f905c-457e-401b-af26-2af1201b5d28" containerName="main" probeResult="failure" output="Get \"https://10.133.0.124:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:26:54.801516 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:26:54.801437    5304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k6fjk" containerName="extractor"
Nov 02 15:26:56.698824 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:26:56.698785    5304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-3daba5-openshift-default-59d44c74df-gjqj2"]
Nov 02 15:26:56.699559 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:26:56.699541    5304 cpu_manager.go:414] "containerMap: removing container" podUID="de0eb9f2-e614-454a-a294-77fc9830a610" containerName="istio-proxy"
Nov 02 15:26:56.699616 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:26:56.699560    5304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de0eb9f2-e614-454a-a294-77fc9830a610" containerName="istio-proxy"
Nov 02 15:26:56.699616 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:26:56.699605    5304 cpu_manager.go:414] "containerMap: removing container" podUID="de0eb9f2-e614-454a-a294-77fc9830a610" containerName="istio-proxy"
Nov 02 15:26:56.699616 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:26:56.699613    5304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de0eb9f2-e614-454a-a294-77fc9830a610" containerName="istio-proxy"
Nov 02 15:26:56.699749 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:26:56.699740    5304 memory_manager.go:365] "RemoveStaleState removing state" podUID="de0eb9f2-e614-454a-a294-77fc9830a610" containerName="istio-proxy"
Nov 02 15:26:56.704828 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:26:56.704807    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-3daba5-openshift-default-59d44c74df-gjqj2"
Nov 02 15:26:56.725676 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:26:56.725650    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-3daba5-openshift-default-59d44c74df-gjqj2"]
Nov 02 15:26:56.753150 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:26:56.741170    5304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-3daba5-openshift-default-dockercfg-gcznk\""
Nov 02 15:26:56.817186 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:26:56.817147    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9jzrp\" (UniqueName: \"kubernetes.io/projected/ea04b74e-0059-4462-9b89-45069a0093cd-kube-api-access-9jzrp\") pod \"e2e-inf-b-3daba5-openshift-default-59d44c74df-gjqj2\" (UID: \"ea04b74e-0059-4462-9b89-45069a0093cd\") " pod="openshift-ingress/e2e-inf-b-3daba5-openshift-default-59d44c74df-gjqj2"
Nov 02 15:26:56.817359 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:26:56.817264    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea04b74e-0059-4462-9b89-45069a0093cd-istio-data\") pod \"e2e-inf-b-3daba5-openshift-default-59d44c74df-gjqj2\" (UID: \"ea04b74e-0059-4462-9b89-45069a0093cd\") " pod="openshift-ingress/e2e-inf-b-3daba5-openshift-default-59d44c74df-gjqj2"
Nov 02 15:26:56.817359 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:26:56.817345    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea04b74e-0059-4462-9b89-45069a0093cd-workload-socket\") pod \"e2e-inf-b-3daba5-openshift-default-59d44c74df-gjqj2\" (UID: \"ea04b74e-0059-4462-9b89-45069a0093cd\") " pod="openshift-ingress/e2e-inf-b-3daba5-openshift-default-59d44c74df-gjqj2"
Nov 02 15:26:56.817445 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:26:56.817431    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea04b74e-0059-4462-9b89-45069a0093cd-istio-token\") pod \"e2e-inf-b-3daba5-openshift-default-59d44c74df-gjqj2\" (UID: \"ea04b74e-0059-4462-9b89-45069a0093cd\") " pod="openshift-ingress/e2e-inf-b-3daba5-openshift-default-59d44c74df-gjqj2"
Nov 02 15:26:56.817497 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:26:56.817483    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea04b74e-0059-4462-9b89-45069a0093cd-istio-envoy\") pod \"e2e-inf-b-3daba5-openshift-default-59d44c74df-gjqj2\" (UID: \"ea04b74e-0059-4462-9b89-45069a0093cd\") " pod="openshift-ingress/e2e-inf-b-3daba5-openshift-default-59d44c74df-gjqj2"
Nov 02 15:26:56.817611 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:26:56.817589    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea04b74e-0059-4462-9b89-45069a0093cd-workload-certs\") pod \"e2e-inf-b-3daba5-openshift-default-59d44c74df-gjqj2\" (UID: \"ea04b74e-0059-4462-9b89-45069a0093cd\") " pod="openshift-ingress/e2e-inf-b-3daba5-openshift-default-59d44c74df-gjqj2"
Nov 02 15:26:56.817704 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:26:56.817691    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea04b74e-0059-4462-9b89-45069a0093cd-istio-podinfo\") pod \"e2e-inf-b-3daba5-openshift-default-59d44c74df-gjqj2\" (UID: \"ea04b74e-0059-4462-9b89-45069a0093cd\") " pod="openshift-ingress/e2e-inf-b-3daba5-openshift-default-59d44c74df-gjqj2"
Nov 02 15:26:56.817804 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:26:56.817791    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea04b74e-0059-4462-9b89-45069a0093cd-credential-socket\") pod \"e2e-inf-b-3daba5-openshift-default-59d44c74df-gjqj2\" (UID: \"ea04b74e-0059-4462-9b89-45069a0093cd\") " pod="openshift-ingress/e2e-inf-b-3daba5-openshift-default-59d44c74df-gjqj2"
Nov 02 15:26:56.817882 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:26:56.817868    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea04b74e-0059-4462-9b89-45069a0093cd-istiod-ca-cert\") pod \"e2e-inf-b-3daba5-openshift-default-59d44c74df-gjqj2\" (UID: \"ea04b74e-0059-4462-9b89-45069a0093cd\") " pod="openshift-ingress/e2e-inf-b-3daba5-openshift-default-59d44c74df-gjqj2"
Nov 02 15:26:56.919674 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:26:56.919638    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea04b74e-0059-4462-9b89-45069a0093cd-istio-podinfo\") pod \"e2e-inf-b-3daba5-openshift-default-59d44c74df-gjqj2\" (UID: \"ea04b74e-0059-4462-9b89-45069a0093cd\") " pod="openshift-ingress/e2e-inf-b-3daba5-openshift-default-59d44c74df-gjqj2"
Nov 02 15:26:56.919853 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:26:56.919741    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea04b74e-0059-4462-9b89-45069a0093cd-credential-socket\") pod \"e2e-inf-b-3daba5-openshift-default-59d44c74df-gjqj2\" (UID: \"ea04b74e-0059-4462-9b89-45069a0093cd\") " pod="openshift-ingress/e2e-inf-b-3daba5-openshift-default-59d44c74df-gjqj2"
Nov 02 15:26:56.919853 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:26:56.919800    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea04b74e-0059-4462-9b89-45069a0093cd-istiod-ca-cert\") pod \"e2e-inf-b-3daba5-openshift-default-59d44c74df-gjqj2\" (UID: \"ea04b74e-0059-4462-9b89-45069a0093cd\") " pod="openshift-ingress/e2e-inf-b-3daba5-openshift-default-59d44c74df-gjqj2"
Nov 02 15:26:56.919853 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:26:56.919828    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9jzrp\" (UniqueName: \"kubernetes.io/projected/ea04b74e-0059-4462-9b89-45069a0093cd-kube-api-access-9jzrp\") pod \"e2e-inf-b-3daba5-openshift-default-59d44c74df-gjqj2\" (UID: \"ea04b74e-0059-4462-9b89-45069a0093cd\") " pod="openshift-ingress/e2e-inf-b-3daba5-openshift-default-59d44c74df-gjqj2"
Nov 02 15:26:56.920016 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:26:56.919870    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea04b74e-0059-4462-9b89-45069a0093cd-istio-data\") pod \"e2e-inf-b-3daba5-openshift-default-59d44c74df-gjqj2\" (UID: \"ea04b74e-0059-4462-9b89-45069a0093cd\") " pod="openshift-ingress/e2e-inf-b-3daba5-openshift-default-59d44c74df-gjqj2"
Nov 02 15:26:56.920016 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:26:56.919916    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea04b74e-0059-4462-9b89-45069a0093cd-workload-socket\") pod \"e2e-inf-b-3daba5-openshift-default-59d44c74df-gjqj2\" (UID: \"ea04b74e-0059-4462-9b89-45069a0093cd\") " pod="openshift-ingress/e2e-inf-b-3daba5-openshift-default-59d44c74df-gjqj2"
Nov 02 15:26:56.920016 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:26:56.919980    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea04b74e-0059-4462-9b89-45069a0093cd-istio-token\") pod \"e2e-inf-b-3daba5-openshift-default-59d44c74df-gjqj2\" (UID: \"ea04b74e-0059-4462-9b89-45069a0093cd\") " pod="openshift-ingress/e2e-inf-b-3daba5-openshift-default-59d44c74df-gjqj2"
Nov 02 15:26:56.920291 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:26:56.920045    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea04b74e-0059-4462-9b89-45069a0093cd-istio-envoy\") pod \"e2e-inf-b-3daba5-openshift-default-59d44c74df-gjqj2\" (UID: \"ea04b74e-0059-4462-9b89-45069a0093cd\") " pod="openshift-ingress/e2e-inf-b-3daba5-openshift-default-59d44c74df-gjqj2"
Nov 02 15:26:56.920291 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:26:56.920099    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea04b74e-0059-4462-9b89-45069a0093cd-workload-certs\") pod \"e2e-inf-b-3daba5-openshift-default-59d44c74df-gjqj2\" (UID: \"ea04b74e-0059-4462-9b89-45069a0093cd\") " pod="openshift-ingress/e2e-inf-b-3daba5-openshift-default-59d44c74df-gjqj2"
Nov 02 15:26:56.920401 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:26:56.920340    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea04b74e-0059-4462-9b89-45069a0093cd-istio-data\") pod \"e2e-inf-b-3daba5-openshift-default-59d44c74df-gjqj2\" (UID: \"ea04b74e-0059-4462-9b89-45069a0093cd\") " pod="openshift-ingress/e2e-inf-b-3daba5-openshift-default-59d44c74df-gjqj2"
Nov 02 15:26:56.920401 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:26:56.920380    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea04b74e-0059-4462-9b89-45069a0093cd-workload-socket\") pod \"e2e-inf-b-3daba5-openshift-default-59d44c74df-gjqj2\" (UID: \"ea04b74e-0059-4462-9b89-45069a0093cd\") " pod="openshift-ingress/e2e-inf-b-3daba5-openshift-default-59d44c74df-gjqj2"
Nov 02 15:26:56.920529 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:26:56.920485    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea04b74e-0059-4462-9b89-45069a0093cd-workload-certs\") pod \"e2e-inf-b-3daba5-openshift-default-59d44c74df-gjqj2\" (UID: \"ea04b74e-0059-4462-9b89-45069a0093cd\") " pod="openshift-ingress/e2e-inf-b-3daba5-openshift-default-59d44c74df-gjqj2"
Nov 02 15:26:56.920575 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:26:56.920539    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea04b74e-0059-4462-9b89-45069a0093cd-credential-socket\") pod \"e2e-inf-b-3daba5-openshift-default-59d44c74df-gjqj2\" (UID: \"ea04b74e-0059-4462-9b89-45069a0093cd\") " pod="openshift-ingress/e2e-inf-b-3daba5-openshift-default-59d44c74df-gjqj2"
Nov 02 15:26:56.920633 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:26:56.920608    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea04b74e-0059-4462-9b89-45069a0093cd-istiod-ca-cert\") pod \"e2e-inf-b-3daba5-openshift-default-59d44c74df-gjqj2\" (UID: \"ea04b74e-0059-4462-9b89-45069a0093cd\") " pod="openshift-ingress/e2e-inf-b-3daba5-openshift-default-59d44c74df-gjqj2"
Nov 02 15:26:56.922601 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:26:56.922578    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea04b74e-0059-4462-9b89-45069a0093cd-istio-envoy\") pod \"e2e-inf-b-3daba5-openshift-default-59d44c74df-gjqj2\" (UID: \"ea04b74e-0059-4462-9b89-45069a0093cd\") " pod="openshift-ingress/e2e-inf-b-3daba5-openshift-default-59d44c74df-gjqj2"
Nov 02 15:26:56.922932 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:26:56.922913    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea04b74e-0059-4462-9b89-45069a0093cd-istio-podinfo\") pod \"e2e-inf-b-3daba5-openshift-default-59d44c74df-gjqj2\" (UID: \"ea04b74e-0059-4462-9b89-45069a0093cd\") " pod="openshift-ingress/e2e-inf-b-3daba5-openshift-default-59d44c74df-gjqj2"
Nov 02 15:26:56.957861 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:26:56.957768    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea04b74e-0059-4462-9b89-45069a0093cd-istio-token\") pod \"e2e-inf-b-3daba5-openshift-default-59d44c74df-gjqj2\" (UID: \"ea04b74e-0059-4462-9b89-45069a0093cd\") " pod="openshift-ingress/e2e-inf-b-3daba5-openshift-default-59d44c74df-gjqj2"
Nov 02 15:26:56.958835 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:26:56.958798    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9jzrp\" (UniqueName: \"kubernetes.io/projected/ea04b74e-0059-4462-9b89-45069a0093cd-kube-api-access-9jzrp\") pod \"e2e-inf-b-3daba5-openshift-default-59d44c74df-gjqj2\" (UID: \"ea04b74e-0059-4462-9b89-45069a0093cd\") " pod="openshift-ingress/e2e-inf-b-3daba5-openshift-default-59d44c74df-gjqj2"
Nov 02 15:26:57.018691 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:26:57.018653    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-3daba5-openshift-default-59d44c74df-gjqj2"
Nov 02 15:26:57.385900 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:26:57.385857    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-3daba5-openshift-default-59d44c74df-gjqj2"]
Nov 02 15:26:57.386091 ip-10-0-135-138 kubenswrapper[5304]: W1102 15:26:57.386060    5304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea04b74e_0059_4462_9b89_45069a0093cd.slice/crio-09f3b38ad06ace0567c1dde1b3b549309be38ed0ae51339dc960e1a7c653df90 WatchSource:0}: Error finding container 09f3b38ad06ace0567c1dde1b3b549309be38ed0ae51339dc960e1a7c653df90: Status 404 returned error can't find the container with id 09f3b38ad06ace0567c1dde1b3b549309be38ed0ae51339dc960e1a7c653df90
Nov 02 15:26:57.389277 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:26:57.389242    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:26:57.389378 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:26:57.389320    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:26:57.389378 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:26:57.389351    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:26:58.119001 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:26:58.118964    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-3daba5-openshift-default-59d44c74df-gjqj2" event={"ID":"ea04b74e-0059-4462-9b89-45069a0093cd","Type":"ContainerStarted","Data":"787771bdd1a185240d2b2b835d094eb6060e35a7a71bffd9517d61f8e4fa8c81"}
Nov 02 15:26:58.119420 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:26:58.119022    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-3daba5-openshift-default-59d44c74df-gjqj2" event={"ID":"ea04b74e-0059-4462-9b89-45069a0093cd","Type":"ContainerStarted","Data":"09f3b38ad06ace0567c1dde1b3b549309be38ed0ae51339dc960e1a7c653df90"}
Nov 02 15:26:58.198201 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:26:58.198141    5304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-3daba5-openshift-default-59d44c74df-gjqj2" podStartSLOduration=2.19811415 podStartE2EDuration="2.19811415s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:26:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:26:58.185672456 +0000 UTC m=+1818.179496868" watchObservedRunningTime="2026-11-02 15:26:58.19811415 +0000 UTC m=+1818.191938559"
Nov 02 15:26:59.018891 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:26:59.018851    5304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-3daba5-openshift-default-59d44c74df-gjqj2"
Nov 02 15:26:59.020157 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:26:59.020111    5304 patch_prober.go:29] interesting pod/e2e-inf-b-3daba5-openshift-default-59d44c74df-gjqj2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" start-of-body=
Nov 02 15:26:59.020234 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:26:59.020184    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-3daba5-openshift-default-59d44c74df-gjqj2" podUID="ea04b74e-0059-4462-9b89-45069a0093cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused"
Nov 02 15:27:00.019260 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:27:00.019213    5304 patch_prober.go:29] interesting pod/e2e-inf-b-3daba5-openshift-default-59d44c74df-gjqj2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" start-of-body=
Nov 02 15:27:00.019659 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:27:00.019276    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-3daba5-openshift-default-59d44c74df-gjqj2" podUID="ea04b74e-0059-4462-9b89-45069a0093cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused"
Nov 02 15:27:01.019634 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:27:01.019592    5304 patch_prober.go:29] interesting pod/e2e-inf-b-3daba5-openshift-default-59d44c74df-gjqj2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" start-of-body=
Nov 02 15:27:01.020237 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:27:01.019655    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-3daba5-openshift-default-59d44c74df-gjqj2" podUID="ea04b74e-0059-4462-9b89-45069a0093cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused"
Nov 02 15:27:02.019394 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:27:02.019361    5304 patch_prober.go:29] interesting pod/e2e-inf-b-3daba5-openshift-default-59d44c74df-gjqj2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" start-of-body=
Nov 02 15:27:02.019571 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:27:02.019427    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-3daba5-openshift-default-59d44c74df-gjqj2" podUID="ea04b74e-0059-4462-9b89-45069a0093cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused"
Nov 02 15:27:02.202271 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:27:02.202233    5304 patch_prober.go:29] interesting pod/e2e-inf-a-2f8b7c-openshift-default-769cb98f9-xm9dm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 15:27:02.202680 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:27:02.202297    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-2f8b7c-openshift-default-769cb98f9-xm9dm" podUID="6f8cf102-3f23-49fe-804f-694c50509462" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:27:03.019202 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:27:03.019167    5304 patch_prober.go:29] interesting pod/e2e-inf-b-3daba5-openshift-default-59d44c74df-gjqj2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" start-of-body=
Nov 02 15:27:03.019410 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:27:03.019233    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-3daba5-openshift-default-59d44c74df-gjqj2" podUID="ea04b74e-0059-4462-9b89-45069a0093cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused"
Nov 02 15:27:04.019262 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:27:04.019225    5304 patch_prober.go:29] interesting pod/e2e-inf-b-3daba5-openshift-default-59d44c74df-gjqj2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" start-of-body=
Nov 02 15:27:04.019685 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:27:04.019285    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-3daba5-openshift-default-59d44c74df-gjqj2" podUID="ea04b74e-0059-4462-9b89-45069a0093cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused"
Nov 02 15:27:05.026358 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:27:05.026328    5304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-3daba5-openshift-default-59d44c74df-gjqj2"
Nov 02 15:27:05.026849 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:27:05.026730    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-3daba5-openshift-default-59d44c74df-gjqj2"
Nov 02 15:27:05.027439 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:27:05.027418    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-3daba5-openshift-default-59d44c74df-gjqj2"
Nov 02 15:27:07.224197 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:27:07.224160    5304 generic.go:356] "Generic (PLEG): container finished" podID="6f8cf102-3f23-49fe-804f-694c50509462" containerID="21a548dc52c0aa6d983371c2a530a668823ee88279a6b33b6df874dfb1cedcfc" exitCode=0
Nov 02 15:27:07.224679 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:27:07.224170    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-2f8b7c-openshift-default-769cb98f9-xm9dm" event={"ID":"6f8cf102-3f23-49fe-804f-694c50509462","Type":"ContainerDied","Data":"21a548dc52c0aa6d983371c2a530a668823ee88279a6b33b6df874dfb1cedcfc"}
Nov 02 15:27:07.224791 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:27:07.224775    5304 scope.go:122] "RemoveContainer" containerID="21a548dc52c0aa6d983371c2a530a668823ee88279a6b33b6df874dfb1cedcfc"
Nov 02 15:27:07.226732 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:27:07.226697    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:27:07.226844 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:27:07.226784    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:27:07.226844 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:27:07.226838    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:27:08.236565 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:27:08.236524    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-2f8b7c-openshift-default-769cb98f9-xm9dm" event={"ID":"6f8cf102-3f23-49fe-804f-694c50509462","Type":"ContainerStarted","Data":"8471ba32241bf0c43c3d85ff57b9f5078b985aa12a1d1d918fde5941cf321089"}
Nov 02 15:27:08.868233 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:27:08.868190    5304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-d35a86-75458f47fc-hd67p"]
Nov 02 15:27:08.881151 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:27:08.869679    5304 memory_manager.go:365] "RemoveStaleState removing state" podUID="de0eb9f2-e614-454a-a294-77fc9830a610" containerName="istio-proxy"
Nov 02 15:27:08.907437 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:27:08.907408    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-d35a86-75458f47fc-hd67p"
Nov 02 15:27:08.955302 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:27:08.949650    5304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-d35a86-dockercfg-p7mcw\""
Nov 02 15:27:08.955302 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:27:08.949729    5304 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-d35a86\""
Nov 02 15:27:08.973232 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:27:08.966207    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-d35a86-75458f47fc-hd67p"]
Nov 02 15:27:09.128659 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:27:09.128580    5304 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b81a985_2325_4b54_80a7_8dc2034e5596.slice\": RecentStats: unable to find data in memory cache]"
Nov 02 15:27:09.162474 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:27:09.162434    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b81a985-2325-4b54-80a7-8dc2034e5596-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-d35a86-75458f47fc-hd67p\" (UID: \"5b81a985-2325-4b54-80a7-8dc2034e5596\") " pod="openshift-ingress/payload-processing-e2e-inf-b-d35a86-75458f47fc-hd67p"
Nov 02 15:27:09.163036 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:27:09.163011    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-trkwg\" (UniqueName: \"kubernetes.io/projected/5b81a985-2325-4b54-80a7-8dc2034e5596-kube-api-access-trkwg\") pod \"payload-processing-e2e-inf-b-d35a86-75458f47fc-hd67p\" (UID: \"5b81a985-2325-4b54-80a7-8dc2034e5596\") " pod="openshift-ingress/payload-processing-e2e-inf-b-d35a86-75458f47fc-hd67p"
Nov 02 15:27:09.165842 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:27:09.165824    5304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-2f8b7c-openshift-default-769cb98f9-xm9dm"
Nov 02 15:27:09.166035 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:27:09.166021    5304 patch_prober.go:29] interesting pod/e2e-inf-a-2f8b7c-openshift-default-769cb98f9-xm9dm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body=
Nov 02 15:27:09.166195 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:27:09.166118    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-2f8b7c-openshift-default-769cb98f9-xm9dm" podUID="6f8cf102-3f23-49fe-804f-694c50509462" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused"
Nov 02 15:27:09.265056 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:27:09.265025    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-trkwg\" (UniqueName: \"kubernetes.io/projected/5b81a985-2325-4b54-80a7-8dc2034e5596-kube-api-access-trkwg\") pod \"payload-processing-e2e-inf-b-d35a86-75458f47fc-hd67p\" (UID: \"5b81a985-2325-4b54-80a7-8dc2034e5596\") " pod="openshift-ingress/payload-processing-e2e-inf-b-d35a86-75458f47fc-hd67p"
Nov 02 15:27:09.265694 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:27:09.265668    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b81a985-2325-4b54-80a7-8dc2034e5596-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-d35a86-75458f47fc-hd67p\" (UID: \"5b81a985-2325-4b54-80a7-8dc2034e5596\") " pod="openshift-ingress/payload-processing-e2e-inf-b-d35a86-75458f47fc-hd67p"
Nov 02 15:27:09.266386 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:27:09.266364    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b81a985-2325-4b54-80a7-8dc2034e5596-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-d35a86-75458f47fc-hd67p\" (UID: \"5b81a985-2325-4b54-80a7-8dc2034e5596\") " pod="openshift-ingress/payload-processing-e2e-inf-b-d35a86-75458f47fc-hd67p"
Nov 02 15:27:09.291530 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:27:09.291493    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-trkwg\" (UniqueName: \"kubernetes.io/projected/5b81a985-2325-4b54-80a7-8dc2034e5596-kube-api-access-trkwg\") pod \"payload-processing-e2e-inf-b-d35a86-75458f47fc-hd67p\" (UID: \"5b81a985-2325-4b54-80a7-8dc2034e5596\") " pod="openshift-ingress/payload-processing-e2e-inf-b-d35a86-75458f47fc-hd67p"
Nov 02 15:27:09.374360 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:27:09.374305    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-d35a86-75458f47fc-hd67p"
Nov 02 15:27:09.723424 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:27:09.723390    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-d35a86-75458f47fc-hd67p"]
Nov 02 15:27:09.724185 ip-10-0-135-138 kubenswrapper[5304]: W1102 15:27:09.724152    5304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b81a985_2325_4b54_80a7_8dc2034e5596.slice/crio-4d6373d72b57fa10989ad7bdc6956ced50e8a2deb1bf86b0902d53b2775c624f WatchSource:0}: Error finding container 4d6373d72b57fa10989ad7bdc6956ced50e8a2deb1bf86b0902d53b2775c624f: Status 404 returned error can't find the container with id 4d6373d72b57fa10989ad7bdc6956ced50e8a2deb1bf86b0902d53b2775c624f
Nov 02 15:27:10.165870 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:27:10.165834    5304 patch_prober.go:29] interesting pod/e2e-inf-a-2f8b7c-openshift-default-769cb98f9-xm9dm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body=
Nov 02 15:27:10.166064 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:27:10.165889    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-2f8b7c-openshift-default-769cb98f9-xm9dm" podUID="6f8cf102-3f23-49fe-804f-694c50509462" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused"
Nov 02 15:27:10.274401 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:27:10.274368    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-d35a86-75458f47fc-hd67p" event={"ID":"5b81a985-2325-4b54-80a7-8dc2034e5596","Type":"ContainerStarted","Data":"c12b7ea1b85d12e31c42bef5c27266eeceda95d0a931309285a4b0f3bb9e10fc"}
Nov 02 15:27:10.275007 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:27:10.274986    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-d35a86-75458f47fc-hd67p"
Nov 02 15:27:10.275215 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:27:10.275195    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-d35a86-75458f47fc-hd67p" event={"ID":"5b81a985-2325-4b54-80a7-8dc2034e5596","Type":"ContainerStarted","Data":"4d6373d72b57fa10989ad7bdc6956ced50e8a2deb1bf86b0902d53b2775c624f"}
Nov 02 15:27:11.166397 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:27:11.166351    5304 patch_prober.go:29] interesting pod/e2e-inf-a-2f8b7c-openshift-default-769cb98f9-xm9dm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body=
Nov 02 15:27:11.166606 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:27:11.166411    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-2f8b7c-openshift-default-769cb98f9-xm9dm" podUID="6f8cf102-3f23-49fe-804f-694c50509462" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused"
Nov 02 15:27:12.166804 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:27:12.166764    5304 patch_prober.go:29] interesting pod/e2e-inf-a-2f8b7c-openshift-default-769cb98f9-xm9dm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body=
Nov 02 15:27:12.167403 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:27:12.166838    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-2f8b7c-openshift-default-769cb98f9-xm9dm" podUID="6f8cf102-3f23-49fe-804f-694c50509462" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused"
Nov 02 15:27:12.557739 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:27:12.557606    5304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-d35a86-75458f47fc-hd67p" podStartSLOduration=4.557587949 podStartE2EDuration="4.557587949s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:27:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:27:10.328325179 +0000 UTC m=+1830.322149589" watchObservedRunningTime="2026-11-02 15:27:12.557587949 +0000 UTC m=+1832.551412359"
Nov 02 15:27:12.561372 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:27:12.561340    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-cmbdw"]
Nov 02 15:27:12.561787 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:27:12.561754    5304 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-cmbdw" podUID="8d335628-6e81-44d0-809a-e6ccb181679c" containerName="istio-proxy" containerID="cri-o://50dd14419792d2806f01a0855390a3119fff13cd513b170c410d8c2e77a557ec" gracePeriod=30
Nov 02 15:27:12.573019 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:27:12.572989    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-6q5dn"]
Nov 02 15:27:12.573361 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:27:12.573286    5304 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-6q5dn" podUID="aee20b2e-e86a-44d0-a69c-069a9ca35892" containerName="istio-proxy" containerID="cri-o://f631f140ca6152560e503207d387e6965b8c0c336f4e093e33b9da1ed3f9c7c8" gracePeriod=30
Nov 02 15:27:13.166210 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:27:13.166178    5304 patch_prober.go:29] interesting pod/e2e-inf-a-2f8b7c-openshift-default-769cb98f9-xm9dm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body=
Nov 02 15:27:13.166387 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:27:13.166231    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-2f8b7c-openshift-default-769cb98f9-xm9dm" podUID="6f8cf102-3f23-49fe-804f-694c50509462" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused"
Nov 02 15:27:14.171776 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:27:14.171744    5304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-2f8b7c-openshift-default-769cb98f9-xm9dm"
Nov 02 15:27:14.172284 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:27:14.172167    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-2f8b7c-openshift-default-769cb98f9-xm9dm"
Nov 02 15:27:14.173065 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:27:14.173044    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-2f8b7c-openshift-default-769cb98f9-xm9dm"
Nov 02 15:27:17.855708 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:27:17.855674    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-w9hxk"]
Nov 02 15:27:17.856231 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:27:17.855933    5304 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-w9hxk" podUID="c41d5c79-8a9b-4ad5-995c-818fd3faf115" containerName="istio-proxy" containerID="cri-o://d5135bb67ac3138b8ad8647cec43c2b60d9644c5642d5ea923599b45f37d7719" gracePeriod=30
Nov 02 15:27:17.912199 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:27:17.912152    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-cmbdw"
Nov 02 15:27:17.951487 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:27:17.951459    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d335628-6e81-44d0-809a-e6ccb181679c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d335628-6e81-44d0-809a-e6ccb181679c-istio-data\") pod \"8d335628-6e81-44d0-809a-e6ccb181679c\" (UID: \"8d335628-6e81-44d0-809a-e6ccb181679c\") "
Nov 02 15:27:17.951632 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:27:17.951618    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d335628-6e81-44d0-809a-e6ccb181679c-kube-api-access-2fmsn\" (UniqueName: \"kubernetes.io/projected/8d335628-6e81-44d0-809a-e6ccb181679c-kube-api-access-2fmsn\") pod \"8d335628-6e81-44d0-809a-e6ccb181679c\" (UID: \"8d335628-6e81-44d0-809a-e6ccb181679c\") "
Nov 02 15:27:17.951691 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:27:17.951680    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d335628-6e81-44d0-809a-e6ccb181679c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d335628-6e81-44d0-809a-e6ccb181679c-workload-certs\") pod \"8d335628-6e81-44d0-809a-e6ccb181679c\" (UID: \"8d335628-6e81-44d0-809a-e6ccb181679c\") "
Nov 02 15:27:17.951734 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:27:17.951709    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d335628-6e81-44d0-809a-e6ccb181679c-istio-data" pod "8d335628-6e81-44d0-809a-e6ccb181679c" (UID: "8d335628-6e81-44d0-809a-e6ccb181679c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:27:17.951734 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:27:17.951716    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d335628-6e81-44d0-809a-e6ccb181679c-istio-token\" (UniqueName: \"kubernetes.io/projected/8d335628-6e81-44d0-809a-e6ccb181679c-istio-token\") pod \"8d335628-6e81-44d0-809a-e6ccb181679c\" (UID: \"8d335628-6e81-44d0-809a-e6ccb181679c\") "
Nov 02 15:27:17.951830 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:27:17.951780    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d335628-6e81-44d0-809a-e6ccb181679c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d335628-6e81-44d0-809a-e6ccb181679c-workload-socket\") pod \"8d335628-6e81-44d0-809a-e6ccb181679c\" (UID: \"8d335628-6e81-44d0-809a-e6ccb181679c\") "
Nov 02 15:27:17.951830 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:27:17.951802    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d335628-6e81-44d0-809a-e6ccb181679c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d335628-6e81-44d0-809a-e6ccb181679c-credential-socket\") pod \"8d335628-6e81-44d0-809a-e6ccb181679c\" (UID: \"8d335628-6e81-44d0-809a-e6ccb181679c\") "
Nov 02 15:27:17.951934 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:27:17.951898    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d335628-6e81-44d0-809a-e6ccb181679c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d335628-6e81-44d0-809a-e6ccb181679c-istio-envoy\") pod \"8d335628-6e81-44d0-809a-e6ccb181679c\" (UID: \"8d335628-6e81-44d0-809a-e6ccb181679c\") "
Nov 02 15:27:17.951934 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:27:17.951919    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8d335628-6e81-44d0-809a-e6ccb181679c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d335628-6e81-44d0-809a-e6ccb181679c-istio-podinfo\") pod \"8d335628-6e81-44d0-809a-e6ccb181679c\" (UID: \"8d335628-6e81-44d0-809a-e6ccb181679c\") "
Nov 02 15:27:17.952036 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:27:17.951945    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8d335628-6e81-44d0-809a-e6ccb181679c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d335628-6e81-44d0-809a-e6ccb181679c-istiod-ca-cert\") pod \"8d335628-6e81-44d0-809a-e6ccb181679c\" (UID: \"8d335628-6e81-44d0-809a-e6ccb181679c\") "
Nov 02 15:27:17.952096 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:27:17.952037    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d335628-6e81-44d0-809a-e6ccb181679c-workload-socket" pod "8d335628-6e81-44d0-809a-e6ccb181679c" (UID: "8d335628-6e81-44d0-809a-e6ccb181679c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:27:17.952096 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:27:17.952052    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d335628-6e81-44d0-809a-e6ccb181679c-workload-certs" pod "8d335628-6e81-44d0-809a-e6ccb181679c" (UID: "8d335628-6e81-44d0-809a-e6ccb181679c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:27:17.952219 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:27:17.952156    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d335628-6e81-44d0-809a-e6ccb181679c-credential-socket" pod "8d335628-6e81-44d0-809a-e6ccb181679c" (UID: "8d335628-6e81-44d0-809a-e6ccb181679c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:27:17.952457 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:27:17.952433    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d335628-6e81-44d0-809a-e6ccb181679c-istiod-ca-cert" pod "8d335628-6e81-44d0-809a-e6ccb181679c" (UID: "8d335628-6e81-44d0-809a-e6ccb181679c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:27:17.952834 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:27:17.952814    5304 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d335628-6e81-44d0-809a-e6ccb181679c-istiod-ca-cert\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:27:17.952935 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:27:17.952838    5304 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d335628-6e81-44d0-809a-e6ccb181679c-istio-data\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:27:17.952935 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:27:17.952852    5304 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d335628-6e81-44d0-809a-e6ccb181679c-workload-certs\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:27:17.952935 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:27:17.952863    5304 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d335628-6e81-44d0-809a-e6ccb181679c-workload-socket\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:27:17.952935 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:27:17.952876    5304 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d335628-6e81-44d0-809a-e6ccb181679c-credential-socket\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:27:17.954106 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:27:17.954087    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d335628-6e81-44d0-809a-e6ccb181679c-istio-token" pod "8d335628-6e81-44d0-809a-e6ccb181679c" (UID: "8d335628-6e81-44d0-809a-e6ccb181679c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:27:17.954359 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:27:17.954332    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d335628-6e81-44d0-809a-e6ccb181679c-kube-api-access-2fmsn" pod "8d335628-6e81-44d0-809a-e6ccb181679c" (UID: "8d335628-6e81-44d0-809a-e6ccb181679c"). InnerVolumeSpecName "kube-api-access-2fmsn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:27:17.954571 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:27:17.954543    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d335628-6e81-44d0-809a-e6ccb181679c-istio-envoy" pod "8d335628-6e81-44d0-809a-e6ccb181679c" (UID: "8d335628-6e81-44d0-809a-e6ccb181679c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:27:17.954653 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:27:17.954582    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8d335628-6e81-44d0-809a-e6ccb181679c-istio-podinfo" pod "8d335628-6e81-44d0-809a-e6ccb181679c" (UID: "8d335628-6e81-44d0-809a-e6ccb181679c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 15:27:17.981699 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:27:17.981670    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-6q5dn"
Nov 02 15:27:18.053615 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:27:18.053581    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aee20b2e-e86a-44d0-a69c-069a9ca35892-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aee20b2e-e86a-44d0-a69c-069a9ca35892-credential-socket\") pod \"aee20b2e-e86a-44d0-a69c-069a9ca35892\" (UID: \"aee20b2e-e86a-44d0-a69c-069a9ca35892\") "
Nov 02 15:27:18.053811 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:27:18.053665    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aee20b2e-e86a-44d0-a69c-069a9ca35892-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aee20b2e-e86a-44d0-a69c-069a9ca35892-istiod-ca-cert\") pod \"aee20b2e-e86a-44d0-a69c-069a9ca35892\" (UID: \"aee20b2e-e86a-44d0-a69c-069a9ca35892\") "
Nov 02 15:27:18.053811 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:27:18.053716    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aee20b2e-e86a-44d0-a69c-069a9ca35892-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aee20b2e-e86a-44d0-a69c-069a9ca35892-istio-data\") pod \"aee20b2e-e86a-44d0-a69c-069a9ca35892\" (UID: \"aee20b2e-e86a-44d0-a69c-069a9ca35892\") "
Nov 02 15:27:18.053977 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:27:18.053816    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aee20b2e-e86a-44d0-a69c-069a9ca35892-istio-token\" (UniqueName: \"kubernetes.io/projected/aee20b2e-e86a-44d0-a69c-069a9ca35892-istio-token\") pod \"aee20b2e-e86a-44d0-a69c-069a9ca35892\" (UID: \"aee20b2e-e86a-44d0-a69c-069a9ca35892\") "
Nov 02 15:27:18.053977 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:27:18.053852    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aee20b2e-e86a-44d0-a69c-069a9ca35892-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aee20b2e-e86a-44d0-a69c-069a9ca35892-workload-socket\") pod \"aee20b2e-e86a-44d0-a69c-069a9ca35892\" (UID: \"aee20b2e-e86a-44d0-a69c-069a9ca35892\") "
Nov 02 15:27:18.053977 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:27:18.053922    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aee20b2e-e86a-44d0-a69c-069a9ca35892-credential-socket" pod "aee20b2e-e86a-44d0-a69c-069a9ca35892" (UID: "aee20b2e-e86a-44d0-a69c-069a9ca35892"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:27:18.054177 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:27:18.054024    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aee20b2e-e86a-44d0-a69c-069a9ca35892-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aee20b2e-e86a-44d0-a69c-069a9ca35892-istio-envoy\") pod \"aee20b2e-e86a-44d0-a69c-069a9ca35892\" (UID: \"aee20b2e-e86a-44d0-a69c-069a9ca35892\") "
Nov 02 15:27:18.054177 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:27:18.054062    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aee20b2e-e86a-44d0-a69c-069a9ca35892-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aee20b2e-e86a-44d0-a69c-069a9ca35892-workload-certs\") pod \"aee20b2e-e86a-44d0-a69c-069a9ca35892\" (UID: \"aee20b2e-e86a-44d0-a69c-069a9ca35892\") "
Nov 02 15:27:18.054177 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:27:18.054161    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aee20b2e-e86a-44d0-a69c-069a9ca35892-istio-data" pod "aee20b2e-e86a-44d0-a69c-069a9ca35892" (UID: "aee20b2e-e86a-44d0-a69c-069a9ca35892"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:27:18.054357 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:27:18.054162    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aee20b2e-e86a-44d0-a69c-069a9ca35892-istiod-ca-cert" pod "aee20b2e-e86a-44d0-a69c-069a9ca35892" (UID: "aee20b2e-e86a-44d0-a69c-069a9ca35892"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:27:18.054357 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:27:18.054163    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aee20b2e-e86a-44d0-a69c-069a9ca35892-kube-api-access-8jfzm\" (UniqueName: \"kubernetes.io/projected/aee20b2e-e86a-44d0-a69c-069a9ca35892-kube-api-access-8jfzm\") pod \"aee20b2e-e86a-44d0-a69c-069a9ca35892\" (UID: \"aee20b2e-e86a-44d0-a69c-069a9ca35892\") "
Nov 02 15:27:18.054357 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:27:18.054251    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/aee20b2e-e86a-44d0-a69c-069a9ca35892-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aee20b2e-e86a-44d0-a69c-069a9ca35892-istio-podinfo\") pod \"aee20b2e-e86a-44d0-a69c-069a9ca35892\" (UID: \"aee20b2e-e86a-44d0-a69c-069a9ca35892\") "
Nov 02 15:27:18.054520 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:27:18.054420    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aee20b2e-e86a-44d0-a69c-069a9ca35892-workload-socket" pod "aee20b2e-e86a-44d0-a69c-069a9ca35892" (UID: "aee20b2e-e86a-44d0-a69c-069a9ca35892"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:27:18.055109 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:27:18.055078    5304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2fmsn\" (UniqueName: \"kubernetes.io/projected/8d335628-6e81-44d0-809a-e6ccb181679c-kube-api-access-2fmsn\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:27:18.055282 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:27:18.055112    5304 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d335628-6e81-44d0-809a-e6ccb181679c-istio-token\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:27:18.055282 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:27:18.055142    5304 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aee20b2e-e86a-44d0-a69c-069a9ca35892-credential-socket\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:27:18.055282 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:27:18.055155    5304 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aee20b2e-e86a-44d0-a69c-069a9ca35892-istiod-ca-cert\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:27:18.055282 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:27:18.055171    5304 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d335628-6e81-44d0-809a-e6ccb181679c-istio-envoy\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:27:18.055282 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:27:18.055183    5304 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d335628-6e81-44d0-809a-e6ccb181679c-istio-podinfo\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:27:18.055282 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:27:18.055196    5304 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aee20b2e-e86a-44d0-a69c-069a9ca35892-istio-data\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:27:18.055282 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:27:18.055209    5304 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aee20b2e-e86a-44d0-a69c-069a9ca35892-workload-socket\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:27:18.055683 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:27:18.055379    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aee20b2e-e86a-44d0-a69c-069a9ca35892-workload-certs" pod "aee20b2e-e86a-44d0-a69c-069a9ca35892" (UID: "aee20b2e-e86a-44d0-a69c-069a9ca35892"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:27:18.057101 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:27:18.057069    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aee20b2e-e86a-44d0-a69c-069a9ca35892-istio-envoy" pod "aee20b2e-e86a-44d0-a69c-069a9ca35892" (UID: "aee20b2e-e86a-44d0-a69c-069a9ca35892"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:27:18.057439 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:27:18.057407    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aee20b2e-e86a-44d0-a69c-069a9ca35892-kube-api-access-8jfzm" pod "aee20b2e-e86a-44d0-a69c-069a9ca35892" (UID: "aee20b2e-e86a-44d0-a69c-069a9ca35892"). InnerVolumeSpecName "kube-api-access-8jfzm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:27:18.057538 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:27:18.057515    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aee20b2e-e86a-44d0-a69c-069a9ca35892-istio-token" pod "aee20b2e-e86a-44d0-a69c-069a9ca35892" (UID: "aee20b2e-e86a-44d0-a69c-069a9ca35892"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:27:18.058447 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:27:18.058412    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/aee20b2e-e86a-44d0-a69c-069a9ca35892-istio-podinfo" pod "aee20b2e-e86a-44d0-a69c-069a9ca35892" (UID: "aee20b2e-e86a-44d0-a69c-069a9ca35892"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 15:27:18.156972 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:27:18.156939    5304 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aee20b2e-e86a-44d0-a69c-069a9ca35892-istio-envoy\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:27:18.156972 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:27:18.156969    5304 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aee20b2e-e86a-44d0-a69c-069a9ca35892-workload-certs\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:27:18.156972 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:27:18.156978    5304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8jfzm\" (UniqueName: \"kubernetes.io/projected/aee20b2e-e86a-44d0-a69c-069a9ca35892-kube-api-access-8jfzm\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:27:18.157236 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:27:18.156988    5304 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aee20b2e-e86a-44d0-a69c-069a9ca35892-istio-podinfo\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:27:18.157236 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:27:18.156996    5304 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aee20b2e-e86a-44d0-a69c-069a9ca35892-istio-token\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:27:18.336996 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:27:18.336961    5304 generic.go:356] "Generic (PLEG): container finished" podID="8d335628-6e81-44d0-809a-e6ccb181679c" containerID="50dd14419792d2806f01a0855390a3119fff13cd513b170c410d8c2e77a557ec" exitCode=0
Nov 02 15:27:18.337236 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:27:18.337017    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-cmbdw"
Nov 02 15:27:18.337236 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:27:18.337024    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-cmbdw" event={"ID":"8d335628-6e81-44d0-809a-e6ccb181679c","Type":"ContainerDied","Data":"50dd14419792d2806f01a0855390a3119fff13cd513b170c410d8c2e77a557ec"}
Nov 02 15:27:18.337236 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:27:18.337084    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-cmbdw" event={"ID":"8d335628-6e81-44d0-809a-e6ccb181679c","Type":"ContainerDied","Data":"7b1e1d9dfb177254163c7d93544adcec57c764fb449b35b6457e49ab4cfa6c85"}
Nov 02 15:27:18.337236 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:27:18.337118    5304 scope.go:122] "RemoveContainer" containerID="50dd14419792d2806f01a0855390a3119fff13cd513b170c410d8c2e77a557ec"
Nov 02 15:27:18.338841 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:27:18.338815    5304 generic.go:356] "Generic (PLEG): container finished" podID="aee20b2e-e86a-44d0-a69c-069a9ca35892" containerID="f631f140ca6152560e503207d387e6965b8c0c336f4e093e33b9da1ed3f9c7c8" exitCode=0
Nov 02 15:27:18.338942 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:27:18.338877    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-6q5dn"
Nov 02 15:27:18.338942 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:27:18.338930    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-6q5dn" event={"ID":"aee20b2e-e86a-44d0-a69c-069a9ca35892","Type":"ContainerDied","Data":"f631f140ca6152560e503207d387e6965b8c0c336f4e093e33b9da1ed3f9c7c8"}
Nov 02 15:27:18.339021 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:27:18.338977    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-6q5dn" event={"ID":"aee20b2e-e86a-44d0-a69c-069a9ca35892","Type":"ContainerDied","Data":"89f96020de1fe121d1e64096c344b79654da577fab9453c06be8ab0a11eb9955"}
Nov 02 15:27:18.348914 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:27:18.348888    5304 scope.go:122] "RemoveContainer" containerID="50dd14419792d2806f01a0855390a3119fff13cd513b170c410d8c2e77a557ec"
Nov 02 15:27:18.349239 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:27:18.349218    5304 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"50dd14419792d2806f01a0855390a3119fff13cd513b170c410d8c2e77a557ec\": container with ID starting with 50dd14419792d2806f01a0855390a3119fff13cd513b170c410d8c2e77a557ec not found: ID does not exist" containerID="50dd14419792d2806f01a0855390a3119fff13cd513b170c410d8c2e77a557ec"
Nov 02 15:27:18.349292 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:27:18.349248    5304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"50dd14419792d2806f01a0855390a3119fff13cd513b170c410d8c2e77a557ec"} err="failed to get container status \"50dd14419792d2806f01a0855390a3119fff13cd513b170c410d8c2e77a557ec\": rpc error: code = NotFound desc = could not find container \"50dd14419792d2806f01a0855390a3119fff13cd513b170c410d8c2e77a557ec\": container with ID starting with 50dd14419792d2806f01a0855390a3119fff13cd513b170c410d8c2e77a557ec not found: ID does not exist"
Nov 02 15:27:18.349292 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:27:18.349270    5304 scope.go:122] "RemoveContainer" containerID="f631f140ca6152560e503207d387e6965b8c0c336f4e093e33b9da1ed3f9c7c8"
Nov 02 15:27:18.359976 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:27:18.359955    5304 scope.go:122] "RemoveContainer" containerID="f631f140ca6152560e503207d387e6965b8c0c336f4e093e33b9da1ed3f9c7c8"
Nov 02 15:27:18.360267 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:27:18.360245    5304 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f631f140ca6152560e503207d387e6965b8c0c336f4e093e33b9da1ed3f9c7c8\": container with ID starting with f631f140ca6152560e503207d387e6965b8c0c336f4e093e33b9da1ed3f9c7c8 not found: ID does not exist" containerID="f631f140ca6152560e503207d387e6965b8c0c336f4e093e33b9da1ed3f9c7c8"
Nov 02 15:27:18.360321 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:27:18.360275    5304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f631f140ca6152560e503207d387e6965b8c0c336f4e093e33b9da1ed3f9c7c8"} err="failed to get container status \"f631f140ca6152560e503207d387e6965b8c0c336f4e093e33b9da1ed3f9c7c8\": rpc error: code = NotFound desc = could not find container \"f631f140ca6152560e503207d387e6965b8c0c336f4e093e33b9da1ed3f9c7c8\": container with ID starting with f631f140ca6152560e503207d387e6965b8c0c336f4e093e33b9da1ed3f9c7c8 not found: ID does not exist"
Nov 02 15:27:18.398875 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:27:18.398794    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-cmbdw"]
Nov 02 15:27:18.411867 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:27:18.411827    5304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-cmbdw"]
Nov 02 15:27:18.443834 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:27:18.443797    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-6q5dn"]
Nov 02 15:27:18.452818 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:27:18.452783    5304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-6q5dn"]
Nov 02 15:27:20.121834 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:27:20.121799    5304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8d335628-6e81-44d0-809a-e6ccb181679c" path="/var/lib/kubelet/pods/8d335628-6e81-44d0-809a-e6ccb181679c/volumes"
Nov 02 15:27:20.122430 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:27:20.122413    5304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aee20b2e-e86a-44d0-a69c-069a9ca35892" path="/var/lib/kubelet/pods/aee20b2e-e86a-44d0-a69c-069a9ca35892/volumes"
Nov 02 15:27:21.285437 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:27:21.285408    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-d35a86-75458f47fc-hd67p"
Nov 02 15:27:23.161361 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:27:23.161328    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-w9hxk"
Nov 02 15:27:23.261791 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:27:23.261758    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c41d5c79-8a9b-4ad5-995c-818fd3faf115-kube-api-access-q5fjx\" (UniqueName: \"kubernetes.io/projected/c41d5c79-8a9b-4ad5-995c-818fd3faf115-kube-api-access-q5fjx\") pod \"c41d5c79-8a9b-4ad5-995c-818fd3faf115\" (UID: \"c41d5c79-8a9b-4ad5-995c-818fd3faf115\") "
Nov 02 15:27:23.261988 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:27:23.261801    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c41d5c79-8a9b-4ad5-995c-818fd3faf115-istio-token\" (UniqueName: \"kubernetes.io/projected/c41d5c79-8a9b-4ad5-995c-818fd3faf115-istio-token\") pod \"c41d5c79-8a9b-4ad5-995c-818fd3faf115\" (UID: \"c41d5c79-8a9b-4ad5-995c-818fd3faf115\") "
Nov 02 15:27:23.261988 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:27:23.261946    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c41d5c79-8a9b-4ad5-995c-818fd3faf115-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c41d5c79-8a9b-4ad5-995c-818fd3faf115-istiod-ca-cert\") pod \"c41d5c79-8a9b-4ad5-995c-818fd3faf115\" (UID: \"c41d5c79-8a9b-4ad5-995c-818fd3faf115\") "
Nov 02 15:27:23.262213 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:27:23.262046    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c41d5c79-8a9b-4ad5-995c-818fd3faf115-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c41d5c79-8a9b-4ad5-995c-818fd3faf115-workload-certs\") pod \"c41d5c79-8a9b-4ad5-995c-818fd3faf115\" (UID: \"c41d5c79-8a9b-4ad5-995c-818fd3faf115\") "
Nov 02 15:27:23.262213 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:27:23.262089    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c41d5c79-8a9b-4ad5-995c-818fd3faf115-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c41d5c79-8a9b-4ad5-995c-818fd3faf115-workload-socket\") pod \"c41d5c79-8a9b-4ad5-995c-818fd3faf115\" (UID: \"c41d5c79-8a9b-4ad5-995c-818fd3faf115\") "
Nov 02 15:27:23.262345 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:27:23.262219    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c41d5c79-8a9b-4ad5-995c-818fd3faf115-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c41d5c79-8a9b-4ad5-995c-818fd3faf115-istio-envoy\") pod \"c41d5c79-8a9b-4ad5-995c-818fd3faf115\" (UID: \"c41d5c79-8a9b-4ad5-995c-818fd3faf115\") "
Nov 02 15:27:23.262345 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:27:23.262334    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c41d5c79-8a9b-4ad5-995c-818fd3faf115-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c41d5c79-8a9b-4ad5-995c-818fd3faf115-istio-data\") pod \"c41d5c79-8a9b-4ad5-995c-818fd3faf115\" (UID: \"c41d5c79-8a9b-4ad5-995c-818fd3faf115\") "
Nov 02 15:27:23.262457 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:27:23.262349    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c41d5c79-8a9b-4ad5-995c-818fd3faf115-istiod-ca-cert" pod "c41d5c79-8a9b-4ad5-995c-818fd3faf115" (UID: "c41d5c79-8a9b-4ad5-995c-818fd3faf115"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:27:23.262457 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:27:23.262384    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c41d5c79-8a9b-4ad5-995c-818fd3faf115-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c41d5c79-8a9b-4ad5-995c-818fd3faf115-credential-socket\") pod \"c41d5c79-8a9b-4ad5-995c-818fd3faf115\" (UID: \"c41d5c79-8a9b-4ad5-995c-818fd3faf115\") "
Nov 02 15:27:23.262457 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:27:23.262413    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c41d5c79-8a9b-4ad5-995c-818fd3faf115-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c41d5c79-8a9b-4ad5-995c-818fd3faf115-istio-podinfo\") pod \"c41d5c79-8a9b-4ad5-995c-818fd3faf115\" (UID: \"c41d5c79-8a9b-4ad5-995c-818fd3faf115\") "
Nov 02 15:27:23.262626 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:27:23.262464    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c41d5c79-8a9b-4ad5-995c-818fd3faf115-workload-certs" pod "c41d5c79-8a9b-4ad5-995c-818fd3faf115" (UID: "c41d5c79-8a9b-4ad5-995c-818fd3faf115"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:27:23.262685 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:27:23.262669    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c41d5c79-8a9b-4ad5-995c-818fd3faf115-istio-data" pod "c41d5c79-8a9b-4ad5-995c-818fd3faf115" (UID: "c41d5c79-8a9b-4ad5-995c-818fd3faf115"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:27:23.262956 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:27:23.262927    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c41d5c79-8a9b-4ad5-995c-818fd3faf115-credential-socket" pod "c41d5c79-8a9b-4ad5-995c-818fd3faf115" (UID: "c41d5c79-8a9b-4ad5-995c-818fd3faf115"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:27:23.263515 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:27:23.263488    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c41d5c79-8a9b-4ad5-995c-818fd3faf115-workload-socket" pod "c41d5c79-8a9b-4ad5-995c-818fd3faf115" (UID: "c41d5c79-8a9b-4ad5-995c-818fd3faf115"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:27:23.264678 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:27:23.264650    5304 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c41d5c79-8a9b-4ad5-995c-818fd3faf115-istiod-ca-cert\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:27:23.264786 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:27:23.264679    5304 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c41d5c79-8a9b-4ad5-995c-818fd3faf115-workload-certs\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:27:23.264786 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:27:23.264694    5304 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c41d5c79-8a9b-4ad5-995c-818fd3faf115-istio-data\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:27:23.264786 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:27:23.264706    5304 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c41d5c79-8a9b-4ad5-995c-818fd3faf115-credential-socket\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:27:23.265401 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:27:23.265347    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c41d5c79-8a9b-4ad5-995c-818fd3faf115-istio-token" pod "c41d5c79-8a9b-4ad5-995c-818fd3faf115" (UID: "c41d5c79-8a9b-4ad5-995c-818fd3faf115"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:27:23.267221 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:27:23.267188    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c41d5c79-8a9b-4ad5-995c-818fd3faf115-istio-envoy" pod "c41d5c79-8a9b-4ad5-995c-818fd3faf115" (UID: "c41d5c79-8a9b-4ad5-995c-818fd3faf115"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:27:23.267221 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:27:23.267208    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c41d5c79-8a9b-4ad5-995c-818fd3faf115-kube-api-access-q5fjx" pod "c41d5c79-8a9b-4ad5-995c-818fd3faf115" (UID: "c41d5c79-8a9b-4ad5-995c-818fd3faf115"). InnerVolumeSpecName "kube-api-access-q5fjx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:27:23.267962 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:27:23.267944    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c41d5c79-8a9b-4ad5-995c-818fd3faf115-istio-podinfo" pod "c41d5c79-8a9b-4ad5-995c-818fd3faf115" (UID: "c41d5c79-8a9b-4ad5-995c-818fd3faf115"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 15:27:23.365933 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:27:23.365899    5304 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c41d5c79-8a9b-4ad5-995c-818fd3faf115-workload-socket\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:27:23.365933 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:27:23.365930    5304 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c41d5c79-8a9b-4ad5-995c-818fd3faf115-istio-envoy\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:27:23.365933 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:27:23.365940    5304 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c41d5c79-8a9b-4ad5-995c-818fd3faf115-istio-podinfo\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:27:23.366215 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:27:23.365948    5304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q5fjx\" (UniqueName: \"kubernetes.io/projected/c41d5c79-8a9b-4ad5-995c-818fd3faf115-kube-api-access-q5fjx\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:27:23.366215 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:27:23.365958    5304 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c41d5c79-8a9b-4ad5-995c-818fd3faf115-istio-token\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:27:23.385999 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:27:23.385910    5304 generic.go:356] "Generic (PLEG): container finished" podID="c41d5c79-8a9b-4ad5-995c-818fd3faf115" containerID="d5135bb67ac3138b8ad8647cec43c2b60d9644c5642d5ea923599b45f37d7719" exitCode=0
Nov 02 15:27:23.385999 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:27:23.385967    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-w9hxk"
Nov 02 15:27:23.386203 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:27:23.385993    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-w9hxk" event={"ID":"c41d5c79-8a9b-4ad5-995c-818fd3faf115","Type":"ContainerDied","Data":"d5135bb67ac3138b8ad8647cec43c2b60d9644c5642d5ea923599b45f37d7719"}
Nov 02 15:27:23.386203 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:27:23.386082    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-w9hxk" event={"ID":"c41d5c79-8a9b-4ad5-995c-818fd3faf115","Type":"ContainerDied","Data":"9b7f4a9f7b3ddeffaa0f8005230c2c54bddebe1fe582fa089c0dff8f69eb8d91"}
Nov 02 15:27:23.386203 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:27:23.386112    5304 scope.go:122] "RemoveContainer" containerID="d5135bb67ac3138b8ad8647cec43c2b60d9644c5642d5ea923599b45f37d7719"
Nov 02 15:27:23.398668 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:27:23.398649    5304 scope.go:122] "RemoveContainer" containerID="5726645d5a40c50de38f15f9278fddfc748e28e031f3a911b0055733b1c724f2"
Nov 02 15:27:23.409921 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:27:23.409899    5304 scope.go:122] "RemoveContainer" containerID="d5135bb67ac3138b8ad8647cec43c2b60d9644c5642d5ea923599b45f37d7719"
Nov 02 15:27:23.410327 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:27:23.410292    5304 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d5135bb67ac3138b8ad8647cec43c2b60d9644c5642d5ea923599b45f37d7719\": container with ID starting with d5135bb67ac3138b8ad8647cec43c2b60d9644c5642d5ea923599b45f37d7719 not found: ID does not exist" containerID="d5135bb67ac3138b8ad8647cec43c2b60d9644c5642d5ea923599b45f37d7719"
Nov 02 15:27:23.410426 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:27:23.410333    5304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5135bb67ac3138b8ad8647cec43c2b60d9644c5642d5ea923599b45f37d7719"} err="failed to get container status \"d5135bb67ac3138b8ad8647cec43c2b60d9644c5642d5ea923599b45f37d7719\": rpc error: code = NotFound desc = could not find container \"d5135bb67ac3138b8ad8647cec43c2b60d9644c5642d5ea923599b45f37d7719\": container with ID starting with d5135bb67ac3138b8ad8647cec43c2b60d9644c5642d5ea923599b45f37d7719 not found: ID does not exist"
Nov 02 15:27:23.410426 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:27:23.410359    5304 scope.go:122] "RemoveContainer" containerID="5726645d5a40c50de38f15f9278fddfc748e28e031f3a911b0055733b1c724f2"
Nov 02 15:27:23.410683 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:27:23.410661    5304 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5726645d5a40c50de38f15f9278fddfc748e28e031f3a911b0055733b1c724f2\": container with ID starting with 5726645d5a40c50de38f15f9278fddfc748e28e031f3a911b0055733b1c724f2 not found: ID does not exist" containerID="5726645d5a40c50de38f15f9278fddfc748e28e031f3a911b0055733b1c724f2"
Nov 02 15:27:23.410770 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:27:23.410686    5304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5726645d5a40c50de38f15f9278fddfc748e28e031f3a911b0055733b1c724f2"} err="failed to get container status \"5726645d5a40c50de38f15f9278fddfc748e28e031f3a911b0055733b1c724f2\": rpc error: code = NotFound desc = could not find container \"5726645d5a40c50de38f15f9278fddfc748e28e031f3a911b0055733b1c724f2\": container with ID starting with 5726645d5a40c50de38f15f9278fddfc748e28e031f3a911b0055733b1c724f2 not found: ID does not exist"
Nov 02 15:27:23.446082 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:27:23.446047    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-w9hxk"]
Nov 02 15:27:23.450978 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:27:23.450954    5304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-w9hxk"]
Nov 02 15:27:24.121383 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:27:24.121351    5304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c41d5c79-8a9b-4ad5-995c-818fd3faf115" path="/var/lib/kubelet/pods/c41d5c79-8a9b-4ad5-995c-818fd3faf115/volumes"
Nov 02 15:27:24.799864 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:27:24.799830    5304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k6fjk" containerName="extractor"
Nov 02 15:27:43.329956 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:27:43.329916    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-2bbcb5/test-model-2bbcb5-kserve-9597db49f-g6cdn"]
Nov 02 15:27:54.142228 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:27:54.142191    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-a713aa-6cd9667b9-dk2qh"]
Nov 02 15:27:54.967388 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:27:54.967346    5304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k6fjk" containerName="extractor"
Nov 02 15:27:58.360500 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:27:58.360430    5304 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-2bbcb5/test-model-2bbcb5-kserve-9597db49f-g6cdn" podUID="f3c417d3-fdf3-4951-8246-dcd24d7e3820" containerName="main" containerID="cri-o://07b8f2c2ee423e3be8d9bc7e255be104ae0ecbf94fcc09a6d097aed75f6237c3" gracePeriod=45
Nov 02 15:27:58.672633 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:27:58.672604    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-2bbcb5/test-model-2bbcb5-kserve-9597db49f-g6cdn"
Nov 02 15:27:58.723072 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:27:58.723036    5304 generic.go:356] "Generic (PLEG): container finished" podID="f3c417d3-fdf3-4951-8246-dcd24d7e3820" containerID="07b8f2c2ee423e3be8d9bc7e255be104ae0ecbf94fcc09a6d097aed75f6237c3" exitCode=0
Nov 02 15:27:58.723297 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:27:58.723088    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-2bbcb5/test-model-2bbcb5-kserve-9597db49f-g6cdn" event={"ID":"f3c417d3-fdf3-4951-8246-dcd24d7e3820","Type":"ContainerDied","Data":"07b8f2c2ee423e3be8d9bc7e255be104ae0ecbf94fcc09a6d097aed75f6237c3"}
Nov 02 15:27:58.723297 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:27:58.723091    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-2bbcb5/test-model-2bbcb5-kserve-9597db49f-g6cdn"
Nov 02 15:27:58.723297 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:27:58.723151    5304 scope.go:122] "RemoveContainer" containerID="07b8f2c2ee423e3be8d9bc7e255be104ae0ecbf94fcc09a6d097aed75f6237c3"
Nov 02 15:27:58.723297 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:27:58.723138    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-2bbcb5/test-model-2bbcb5-kserve-9597db49f-g6cdn" event={"ID":"f3c417d3-fdf3-4951-8246-dcd24d7e3820","Type":"ContainerDied","Data":"28c4ae9555e439eac40a696ca76cfb3de24695bcebb271d3d1fad9372677b756"}
Nov 02 15:27:58.740485 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:27:58.740456    5304 scope.go:122] "RemoveContainer" containerID="07b8f2c2ee423e3be8d9bc7e255be104ae0ecbf94fcc09a6d097aed75f6237c3"
Nov 02 15:27:58.740859 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:27:58.740832    5304 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"07b8f2c2ee423e3be8d9bc7e255be104ae0ecbf94fcc09a6d097aed75f6237c3\": container with ID starting with 07b8f2c2ee423e3be8d9bc7e255be104ae0ecbf94fcc09a6d097aed75f6237c3 not found: ID does not exist" containerID="07b8f2c2ee423e3be8d9bc7e255be104ae0ecbf94fcc09a6d097aed75f6237c3"
Nov 02 15:27:58.740929 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:27:58.740871    5304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07b8f2c2ee423e3be8d9bc7e255be104ae0ecbf94fcc09a6d097aed75f6237c3"} err="failed to get container status \"07b8f2c2ee423e3be8d9bc7e255be104ae0ecbf94fcc09a6d097aed75f6237c3\": rpc error: code = NotFound desc = could not find container \"07b8f2c2ee423e3be8d9bc7e255be104ae0ecbf94fcc09a6d097aed75f6237c3\": container with ID starting with 07b8f2c2ee423e3be8d9bc7e255be104ae0ecbf94fcc09a6d097aed75f6237c3 not found: ID does not exist"
Nov 02 15:27:58.771299 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:27:58.771260    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3c417d3-fdf3-4951-8246-dcd24d7e3820-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f3c417d3-fdf3-4951-8246-dcd24d7e3820-model-cache\") pod \"f3c417d3-fdf3-4951-8246-dcd24d7e3820\" (UID: \"f3c417d3-fdf3-4951-8246-dcd24d7e3820\") "
Nov 02 15:27:58.771472 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:27:58.771316    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3c417d3-fdf3-4951-8246-dcd24d7e3820-home\" (UniqueName: \"kubernetes.io/empty-dir/f3c417d3-fdf3-4951-8246-dcd24d7e3820-home\") pod \"f3c417d3-fdf3-4951-8246-dcd24d7e3820\" (UID: \"f3c417d3-fdf3-4951-8246-dcd24d7e3820\") "
Nov 02 15:27:58.771472 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:27:58.771358    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f3c417d3-fdf3-4951-8246-dcd24d7e3820-tls-certs\" (UniqueName: \"kubernetes.io/secret/f3c417d3-fdf3-4951-8246-dcd24d7e3820-tls-certs\") pod \"f3c417d3-fdf3-4951-8246-dcd24d7e3820\" (UID: \"f3c417d3-fdf3-4951-8246-dcd24d7e3820\") "
Nov 02 15:27:58.771472 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:27:58.771404    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3c417d3-fdf3-4951-8246-dcd24d7e3820-kube-api-access-vq67h\" (UniqueName: \"kubernetes.io/projected/f3c417d3-fdf3-4951-8246-dcd24d7e3820-kube-api-access-vq67h\") pod \"f3c417d3-fdf3-4951-8246-dcd24d7e3820\" (UID: \"f3c417d3-fdf3-4951-8246-dcd24d7e3820\") "
Nov 02 15:27:58.771648 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:27:58.771571    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3c417d3-fdf3-4951-8246-dcd24d7e3820-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f3c417d3-fdf3-4951-8246-dcd24d7e3820-dshm\") pod \"f3c417d3-fdf3-4951-8246-dcd24d7e3820\" (UID: \"f3c417d3-fdf3-4951-8246-dcd24d7e3820\") "
Nov 02 15:27:58.771648 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:27:58.771599    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3c417d3-fdf3-4951-8246-dcd24d7e3820-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f3c417d3-fdf3-4951-8246-dcd24d7e3820-tmp-dir\") pod \"f3c417d3-fdf3-4951-8246-dcd24d7e3820\" (UID: \"f3c417d3-fdf3-4951-8246-dcd24d7e3820\") "
Nov 02 15:27:58.771803 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:27:58.771566    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3c417d3-fdf3-4951-8246-dcd24d7e3820-home" pod "f3c417d3-fdf3-4951-8246-dcd24d7e3820" (UID: "f3c417d3-fdf3-4951-8246-dcd24d7e3820"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:27:58.771859 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:27:58.771573    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3c417d3-fdf3-4951-8246-dcd24d7e3820-model-cache" pod "f3c417d3-fdf3-4951-8246-dcd24d7e3820" (UID: "f3c417d3-fdf3-4951-8246-dcd24d7e3820"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:27:58.772536 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:27:58.772078    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3c417d3-fdf3-4951-8246-dcd24d7e3820-tmp-dir" pod "f3c417d3-fdf3-4951-8246-dcd24d7e3820" (UID: "f3c417d3-fdf3-4951-8246-dcd24d7e3820"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:27:58.772536 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:27:58.772474    5304 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f3c417d3-fdf3-4951-8246-dcd24d7e3820-tmp-dir\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:27:58.772536 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:27:58.772501    5304 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f3c417d3-fdf3-4951-8246-dcd24d7e3820-model-cache\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:27:58.772536 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:27:58.772514    5304 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f3c417d3-fdf3-4951-8246-dcd24d7e3820-home\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:27:58.774571 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:27:58.774533    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f3c417d3-fdf3-4951-8246-dcd24d7e3820-tls-certs" pod "f3c417d3-fdf3-4951-8246-dcd24d7e3820" (UID: "f3c417d3-fdf3-4951-8246-dcd24d7e3820"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:27:58.774694 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:27:58.774612    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3c417d3-fdf3-4951-8246-dcd24d7e3820-dshm" pod "f3c417d3-fdf3-4951-8246-dcd24d7e3820" (UID: "f3c417d3-fdf3-4951-8246-dcd24d7e3820"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:27:58.774694 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:27:58.774647    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3c417d3-fdf3-4951-8246-dcd24d7e3820-kube-api-access-vq67h" pod "f3c417d3-fdf3-4951-8246-dcd24d7e3820" (UID: "f3c417d3-fdf3-4951-8246-dcd24d7e3820"). InnerVolumeSpecName "kube-api-access-vq67h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:27:58.873679 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:27:58.873648    5304 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f3c417d3-fdf3-4951-8246-dcd24d7e3820-tls-certs\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:27:58.873679 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:27:58.873676    5304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vq67h\" (UniqueName: \"kubernetes.io/projected/f3c417d3-fdf3-4951-8246-dcd24d7e3820-kube-api-access-vq67h\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:27:58.873894 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:27:58.873688    5304 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f3c417d3-fdf3-4951-8246-dcd24d7e3820-dshm\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:27:59.071629 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:27:59.071593    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-2bbcb5/test-model-2bbcb5-kserve-9597db49f-g6cdn"]
Nov 02 15:27:59.075200 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:27:59.075171    5304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-2bbcb5/test-model-2bbcb5-kserve-9597db49f-g6cdn"]
Nov 02 15:27:59.142726 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:27:59.142675    5304 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-a713aa-6cd9667b9-dk2qh" podUID="864c95a0-3f74-41bc-8d00-2dd60f1d5cdc" containerName="payload-processing" containerID="cri-o://9b27d8ffdacf02d5ca788b9ab3d2a7a55cbf3e203851d0eca1a9694aca6816b6" gracePeriod=25
Nov 02 15:27:59.526418 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:27:59.526384    5304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-cfc63a-6cc5b9494c-scfrq"]
Nov 02 15:27:59.527035 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:27:59.527020    5304 cpu_manager.go:414] "containerMap: removing container" podUID="8d335628-6e81-44d0-809a-e6ccb181679c" containerName="istio-proxy"
Nov 02 15:27:59.527086 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:27:59.527036    5304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d335628-6e81-44d0-809a-e6ccb181679c" containerName="istio-proxy"
Nov 02 15:27:59.527086 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:27:59.527053    5304 cpu_manager.go:414] "containerMap: removing container" podUID="c41d5c79-8a9b-4ad5-995c-818fd3faf115" containerName="istio-proxy"
Nov 02 15:27:59.527086 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:27:59.527058    5304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c41d5c79-8a9b-4ad5-995c-818fd3faf115" containerName="istio-proxy"
Nov 02 15:27:59.527086 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:27:59.527066    5304 cpu_manager.go:414] "containerMap: removing container" podUID="aee20b2e-e86a-44d0-a69c-069a9ca35892" containerName="istio-proxy"
Nov 02 15:27:59.527086 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:27:59.527071    5304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aee20b2e-e86a-44d0-a69c-069a9ca35892" containerName="istio-proxy"
Nov 02 15:27:59.527086 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:27:59.527081    5304 cpu_manager.go:414] "containerMap: removing container" podUID="f3c417d3-fdf3-4951-8246-dcd24d7e3820" containerName="main"
Nov 02 15:27:59.527086 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:27:59.527086    5304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3c417d3-fdf3-4951-8246-dcd24d7e3820" containerName="main"
Nov 02 15:27:59.527343 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:27:59.527240    5304 memory_manager.go:365] "RemoveStaleState removing state" podUID="c41d5c79-8a9b-4ad5-995c-818fd3faf115" containerName="istio-proxy"
Nov 02 15:27:59.527343 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:27:59.527255    5304 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d335628-6e81-44d0-809a-e6ccb181679c" containerName="istio-proxy"
Nov 02 15:27:59.527343 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:27:59.527262    5304 memory_manager.go:365] "RemoveStaleState removing state" podUID="aee20b2e-e86a-44d0-a69c-069a9ca35892" containerName="istio-proxy"
Nov 02 15:27:59.527343 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:27:59.527270    5304 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3c417d3-fdf3-4951-8246-dcd24d7e3820" containerName="main"
Nov 02 15:27:59.532262 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:27:59.532240    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cfc63a-6cc5b9494c-scfrq"
Nov 02 15:27:59.548021 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:27:59.547992    5304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-cfc63a-serving-cert\""
Nov 02 15:27:59.568303 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:27:59.568265    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-cfc63a-6cc5b9494c-scfrq"]
Nov 02 15:27:59.583427 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:27:59.583388    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/36c430b0-d26d-46cb-8b8d-f2f2a89aedbd-openshift-service-ca\") pod \"maas-api-e2e-inf-b-cfc63a-6cc5b9494c-scfrq\" (UID: \"36c430b0-d26d-46cb-8b8d-f2f2a89aedbd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cfc63a-6cc5b9494c-scfrq"
Nov 02 15:27:59.583912 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:27:59.583885    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j8bp4\" (UniqueName: \"kubernetes.io/projected/36c430b0-d26d-46cb-8b8d-f2f2a89aedbd-kube-api-access-j8bp4\") pod \"maas-api-e2e-inf-b-cfc63a-6cc5b9494c-scfrq\" (UID: \"36c430b0-d26d-46cb-8b8d-f2f2a89aedbd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cfc63a-6cc5b9494c-scfrq"
Nov 02 15:27:59.584160 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:27:59.584120    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/36c430b0-d26d-46cb-8b8d-f2f2a89aedbd-metrics-certs\") pod \"maas-api-e2e-inf-b-cfc63a-6cc5b9494c-scfrq\" (UID: \"36c430b0-d26d-46cb-8b8d-f2f2a89aedbd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cfc63a-6cc5b9494c-scfrq"
Nov 02 15:27:59.584267 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:27:59.584195    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/36c430b0-d26d-46cb-8b8d-f2f2a89aedbd-maas-api-tls\") pod \"maas-api-e2e-inf-b-cfc63a-6cc5b9494c-scfrq\" (UID: \"36c430b0-d26d-46cb-8b8d-f2f2a89aedbd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cfc63a-6cc5b9494c-scfrq"
Nov 02 15:27:59.685205 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:27:59.685167    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j8bp4\" (UniqueName: \"kubernetes.io/projected/36c430b0-d26d-46cb-8b8d-f2f2a89aedbd-kube-api-access-j8bp4\") pod \"maas-api-e2e-inf-b-cfc63a-6cc5b9494c-scfrq\" (UID: \"36c430b0-d26d-46cb-8b8d-f2f2a89aedbd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cfc63a-6cc5b9494c-scfrq"
Nov 02 15:27:59.685415 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:27:59.685242    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/36c430b0-d26d-46cb-8b8d-f2f2a89aedbd-metrics-certs\") pod \"maas-api-e2e-inf-b-cfc63a-6cc5b9494c-scfrq\" (UID: \"36c430b0-d26d-46cb-8b8d-f2f2a89aedbd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cfc63a-6cc5b9494c-scfrq"
Nov 02 15:27:59.685415 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:27:59.685295    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/36c430b0-d26d-46cb-8b8d-f2f2a89aedbd-maas-api-tls\") pod \"maas-api-e2e-inf-b-cfc63a-6cc5b9494c-scfrq\" (UID: \"36c430b0-d26d-46cb-8b8d-f2f2a89aedbd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cfc63a-6cc5b9494c-scfrq"
Nov 02 15:27:59.685415 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:27:59.685391    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/36c430b0-d26d-46cb-8b8d-f2f2a89aedbd-openshift-service-ca\") pod \"maas-api-e2e-inf-b-cfc63a-6cc5b9494c-scfrq\" (UID: \"36c430b0-d26d-46cb-8b8d-f2f2a89aedbd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cfc63a-6cc5b9494c-scfrq"
Nov 02 15:27:59.686063 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:27:59.686036    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/36c430b0-d26d-46cb-8b8d-f2f2a89aedbd-openshift-service-ca\") pod \"maas-api-e2e-inf-b-cfc63a-6cc5b9494c-scfrq\" (UID: \"36c430b0-d26d-46cb-8b8d-f2f2a89aedbd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cfc63a-6cc5b9494c-scfrq"
Nov 02 15:27:59.688113 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:27:59.688089    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/36c430b0-d26d-46cb-8b8d-f2f2a89aedbd-metrics-certs\") pod \"maas-api-e2e-inf-b-cfc63a-6cc5b9494c-scfrq\" (UID: \"36c430b0-d26d-46cb-8b8d-f2f2a89aedbd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cfc63a-6cc5b9494c-scfrq"
Nov 02 15:27:59.688283 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:27:59.688155    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/36c430b0-d26d-46cb-8b8d-f2f2a89aedbd-maas-api-tls\") pod \"maas-api-e2e-inf-b-cfc63a-6cc5b9494c-scfrq\" (UID: \"36c430b0-d26d-46cb-8b8d-f2f2a89aedbd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cfc63a-6cc5b9494c-scfrq"
Nov 02 15:27:59.708902 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:27:59.708870    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j8bp4\" (UniqueName: \"kubernetes.io/projected/36c430b0-d26d-46cb-8b8d-f2f2a89aedbd-kube-api-access-j8bp4\") pod \"maas-api-e2e-inf-b-cfc63a-6cc5b9494c-scfrq\" (UID: \"36c430b0-d26d-46cb-8b8d-f2f2a89aedbd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cfc63a-6cc5b9494c-scfrq"
Nov 02 15:27:59.846608 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:27:59.846578    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cfc63a-6cc5b9494c-scfrq"
Nov 02 15:28:00.120401 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:28:00.120319    5304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f3c417d3-fdf3-4951-8246-dcd24d7e3820" path="/var/lib/kubelet/pods/f3c417d3-fdf3-4951-8246-dcd24d7e3820/volumes"
Nov 02 15:28:00.161860 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:28:00.161834    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-cfc63a-6cc5b9494c-scfrq"]
Nov 02 15:28:00.164591 ip-10-0-135-138 kubenswrapper[5304]: W1102 15:28:00.164559    5304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36c430b0_d26d_46cb_8b8d_f2f2a89aedbd.slice/crio-dfac31c453482b16200fed5103f78d274ed0413a093a7f3492f98d4716bd4972 WatchSource:0}: Error finding container dfac31c453482b16200fed5103f78d274ed0413a093a7f3492f98d4716bd4972: Status 404 returned error can't find the container with id dfac31c453482b16200fed5103f78d274ed0413a093a7f3492f98d4716bd4972
Nov 02 15:28:00.779396 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:28:00.779302    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cfc63a-6cc5b9494c-scfrq" event={"ID":"36c430b0-d26d-46cb-8b8d-f2f2a89aedbd","Type":"ContainerStarted","Data":"4d3c47b359719df31679348e0fc7abed238a6562ef2a0f9f8ce25e242c60f161"}
Nov 02 15:28:00.779396 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:28:00.779370    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cfc63a-6cc5b9494c-scfrq" event={"ID":"36c430b0-d26d-46cb-8b8d-f2f2a89aedbd","Type":"ContainerStarted","Data":"dfac31c453482b16200fed5103f78d274ed0413a093a7f3492f98d4716bd4972"}
Nov 02 15:28:00.779910 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:28:00.779482    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cfc63a-6cc5b9494c-scfrq"
Nov 02 15:28:00.823843 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:28:00.823794    5304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cfc63a-6cc5b9494c-scfrq" podStartSLOduration=1.468926447 podStartE2EDuration="1.82377799s" totalImagesPullingTime="354.851543ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:27:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:28:00.821057236 +0000 UTC m=+1880.814881667" watchObservedRunningTime="2026-11-02 15:28:00.82377799 +0000 UTC m=+1880.817602399"
Nov 02 15:28:03.733966 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:28:03.733911    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-a713aa-6cd9667b9-dk2qh" podUID="864c95a0-3f74-41bc-8d00-2dd60f1d5cdc" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 15:28:06.791001 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:28:06.790972    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cfc63a-6cc5b9494c-scfrq"
Nov 02 15:28:13.734342 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:28:13.734298    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-a713aa-6cd9667b9-dk2qh" podUID="864c95a0-3f74-41bc-8d00-2dd60f1d5cdc" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 15:28:19.432633 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:28:19.432603    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-a713aa-6cd9667b9-dk2qh"
Nov 02 15:28:19.622225 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:28:19.622191    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/864c95a0-3f74-41bc-8d00-2dd60f1d5cdc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/864c95a0-3f74-41bc-8d00-2dd60f1d5cdc-plugins-config-volume\") pod \"864c95a0-3f74-41bc-8d00-2dd60f1d5cdc\" (UID: \"864c95a0-3f74-41bc-8d00-2dd60f1d5cdc\") "
Nov 02 15:28:19.622407 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:28:19.622362    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/864c95a0-3f74-41bc-8d00-2dd60f1d5cdc-kube-api-access-szc9m\" (UniqueName: \"kubernetes.io/projected/864c95a0-3f74-41bc-8d00-2dd60f1d5cdc-kube-api-access-szc9m\") pod \"864c95a0-3f74-41bc-8d00-2dd60f1d5cdc\" (UID: \"864c95a0-3f74-41bc-8d00-2dd60f1d5cdc\") "
Nov 02 15:28:19.622698 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:28:19.622672    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/864c95a0-3f74-41bc-8d00-2dd60f1d5cdc-plugins-config-volume" pod "864c95a0-3f74-41bc-8d00-2dd60f1d5cdc" (UID: "864c95a0-3f74-41bc-8d00-2dd60f1d5cdc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:28:19.623084 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:28:19.623065    5304 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/864c95a0-3f74-41bc-8d00-2dd60f1d5cdc-plugins-config-volume\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:28:19.624707 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:28:19.624681    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/864c95a0-3f74-41bc-8d00-2dd60f1d5cdc-kube-api-access-szc9m" pod "864c95a0-3f74-41bc-8d00-2dd60f1d5cdc" (UID: "864c95a0-3f74-41bc-8d00-2dd60f1d5cdc"). InnerVolumeSpecName "kube-api-access-szc9m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:28:19.724720 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:28:19.724638    5304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-szc9m\" (UniqueName: \"kubernetes.io/projected/864c95a0-3f74-41bc-8d00-2dd60f1d5cdc-kube-api-access-szc9m\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:28:19.954215 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:28:19.954121    5304 generic.go:356] "Generic (PLEG): container finished" podID="864c95a0-3f74-41bc-8d00-2dd60f1d5cdc" containerID="9b27d8ffdacf02d5ca788b9ab3d2a7a55cbf3e203851d0eca1a9694aca6816b6" exitCode=0
Nov 02 15:28:19.954389 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:28:19.954366    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-a713aa-6cd9667b9-dk2qh" event={"ID":"864c95a0-3f74-41bc-8d00-2dd60f1d5cdc","Type":"ContainerDied","Data":"9b27d8ffdacf02d5ca788b9ab3d2a7a55cbf3e203851d0eca1a9694aca6816b6"}
Nov 02 15:28:19.954433 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:28:19.954423    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-a713aa-6cd9667b9-dk2qh" event={"ID":"864c95a0-3f74-41bc-8d00-2dd60f1d5cdc","Type":"ContainerDied","Data":"b1f65333cf2d4fc7bdf5d423ab16455988dd6264ebe913c83ddb787b9881d1a7"}
Nov 02 15:28:19.954473 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:28:19.954457    5304 scope.go:122] "RemoveContainer" containerID="9b27d8ffdacf02d5ca788b9ab3d2a7a55cbf3e203851d0eca1a9694aca6816b6"
Nov 02 15:28:19.954646 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:28:19.954630    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-a713aa-6cd9667b9-dk2qh"
Nov 02 15:28:19.972676 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:28:19.972647    5304 scope.go:122] "RemoveContainer" containerID="9b27d8ffdacf02d5ca788b9ab3d2a7a55cbf3e203851d0eca1a9694aca6816b6"
Nov 02 15:28:19.973031 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:28:19.973008    5304 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b27d8ffdacf02d5ca788b9ab3d2a7a55cbf3e203851d0eca1a9694aca6816b6\": container with ID starting with 9b27d8ffdacf02d5ca788b9ab3d2a7a55cbf3e203851d0eca1a9694aca6816b6 not found: ID does not exist" containerID="9b27d8ffdacf02d5ca788b9ab3d2a7a55cbf3e203851d0eca1a9694aca6816b6"
Nov 02 15:28:19.973113 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:28:19.973041    5304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b27d8ffdacf02d5ca788b9ab3d2a7a55cbf3e203851d0eca1a9694aca6816b6"} err="failed to get container status \"9b27d8ffdacf02d5ca788b9ab3d2a7a55cbf3e203851d0eca1a9694aca6816b6\": rpc error: code = NotFound desc = could not find container \"9b27d8ffdacf02d5ca788b9ab3d2a7a55cbf3e203851d0eca1a9694aca6816b6\": container with ID starting with 9b27d8ffdacf02d5ca788b9ab3d2a7a55cbf3e203851d0eca1a9694aca6816b6 not found: ID does not exist"
Nov 02 15:28:20.008250 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:28:20.008159    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-a713aa-6cd9667b9-dk2qh"]
Nov 02 15:28:20.020114 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:28:20.020081    5304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-a713aa-6cd9667b9-dk2qh"]
Nov 02 15:28:20.123531 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:28:20.123491    5304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="864c95a0-3f74-41bc-8d00-2dd60f1d5cdc" path="/var/lib/kubelet/pods/864c95a0-3f74-41bc-8d00-2dd60f1d5cdc/volumes"
Nov 02 15:28:24.837207 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:28:24.834269    5304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k6fjk" containerName="extractor"
Nov 02 15:28:34.523064 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:28:34.523027    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-a713aa-openshift-default-7554947cc9-trs9t"]
Nov 02 15:28:34.523601 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:28:34.523303    5304 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-a713aa-openshift-default-7554947cc9-trs9t" podUID="9b33eaba-53ea-47b6-bdfe-3d3d9ffd8a36" containerName="istio-proxy" containerID="cri-o://386621ec229173080d14b76e5daaeefd5b35a1b33aeca89d3fecf66ae38a29bf" gracePeriod=30
Nov 02 15:28:38.112484 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:28:38.112448    5304 generic.go:356] "Generic (PLEG): container finished" podID="1f9d68a5-7f9c-489b-9289-2f85b7889c2c" containerID="ca4848c3f3277089ca035016514b458f2450114dc6a79e6b0ef5191a3518abc8" exitCode=0
Nov 02 15:28:38.119689 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:28:38.119658    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-80ec03-openshift-default-66dfdbcf85-b4ltb" event={"ID":"1f9d68a5-7f9c-489b-9289-2f85b7889c2c","Type":"ContainerDied","Data":"ca4848c3f3277089ca035016514b458f2450114dc6a79e6b0ef5191a3518abc8"}
Nov 02 15:28:38.120076 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:28:38.120057    5304 scope.go:122] "RemoveContainer" containerID="ca4848c3f3277089ca035016514b458f2450114dc6a79e6b0ef5191a3518abc8"
Nov 02 15:28:38.121863 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:28:38.121828    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:28:38.121952 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:28:38.121922    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:28:38.122017 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:28:38.121998    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:28:38.123557 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:28:38.123530    5304 patch_prober.go:29] interesting pod/e2e-inf-a-a713aa-openshift-default-7554947cc9-trs9t container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 15:28:38.123651 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:28:38.123576    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-a713aa-openshift-default-7554947cc9-trs9t" podUID="9b33eaba-53ea-47b6-bdfe-3d3d9ffd8a36" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 15:28:39.121682 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:28:39.121644    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-80ec03-openshift-default-66dfdbcf85-b4ltb" event={"ID":"1f9d68a5-7f9c-489b-9289-2f85b7889c2c","Type":"ContainerStarted","Data":"a2e6e9aa05bc75198000808de8a0a4e21fef1cc7311a0b5ccebfcd1ef4518d52"}
Nov 02 15:28:39.559405 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:28:39.558767    5304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-80ec03-openshift-default-66dfdbcf85-b4ltb"
Nov 02 15:28:39.559405 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:28:39.559069    5304 patch_prober.go:29] interesting pod/e2e-inf-a-80ec03-openshift-default-66dfdbcf85-b4ltb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" start-of-body=
Nov 02 15:28:39.559405 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:28:39.559202    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-80ec03-openshift-default-66dfdbcf85-b4ltb" podUID="1f9d68a5-7f9c-489b-9289-2f85b7889c2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused"
Nov 02 15:28:39.877638 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:28:39.877609    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-a713aa-openshift-default-7554947cc9-trs9t"
Nov 02 15:28:39.999066 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:28:39.999016    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b33eaba-53ea-47b6-bdfe-3d3d9ffd8a36-kube-api-access-fttkn\" (UniqueName: \"kubernetes.io/projected/9b33eaba-53ea-47b6-bdfe-3d3d9ffd8a36-kube-api-access-fttkn\") pod \"9b33eaba-53ea-47b6-bdfe-3d3d9ffd8a36\" (UID: \"9b33eaba-53ea-47b6-bdfe-3d3d9ffd8a36\") "
Nov 02 15:28:39.999310 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:28:39.999105    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9b33eaba-53ea-47b6-bdfe-3d3d9ffd8a36-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b33eaba-53ea-47b6-bdfe-3d3d9ffd8a36-istiod-ca-cert\") pod \"9b33eaba-53ea-47b6-bdfe-3d3d9ffd8a36\" (UID: \"9b33eaba-53ea-47b6-bdfe-3d3d9ffd8a36\") "
Nov 02 15:28:39.999310 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:28:39.999220    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b33eaba-53ea-47b6-bdfe-3d3d9ffd8a36-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b33eaba-53ea-47b6-bdfe-3d3d9ffd8a36-istio-envoy\") pod \"9b33eaba-53ea-47b6-bdfe-3d3d9ffd8a36\" (UID: \"9b33eaba-53ea-47b6-bdfe-3d3d9ffd8a36\") "
Nov 02 15:28:39.999310 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:28:39.999269    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b33eaba-53ea-47b6-bdfe-3d3d9ffd8a36-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b33eaba-53ea-47b6-bdfe-3d3d9ffd8a36-workload-socket\") pod \"9b33eaba-53ea-47b6-bdfe-3d3d9ffd8a36\" (UID: \"9b33eaba-53ea-47b6-bdfe-3d3d9ffd8a36\") "
Nov 02 15:28:39.999606 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:28:39.999556    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b33eaba-53ea-47b6-bdfe-3d3d9ffd8a36-workload-socket" pod "9b33eaba-53ea-47b6-bdfe-3d3d9ffd8a36" (UID: "9b33eaba-53ea-47b6-bdfe-3d3d9ffd8a36"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:28:39.999747 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:28:39.999610    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9b33eaba-53ea-47b6-bdfe-3d3d9ffd8a36-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b33eaba-53ea-47b6-bdfe-3d3d9ffd8a36-istio-podinfo\") pod \"9b33eaba-53ea-47b6-bdfe-3d3d9ffd8a36\" (UID: \"9b33eaba-53ea-47b6-bdfe-3d3d9ffd8a36\") "
Nov 02 15:28:39.999747 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:28:39.999628    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b33eaba-53ea-47b6-bdfe-3d3d9ffd8a36-istiod-ca-cert" pod "9b33eaba-53ea-47b6-bdfe-3d3d9ffd8a36" (UID: "9b33eaba-53ea-47b6-bdfe-3d3d9ffd8a36"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:28:40.000376 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:28:39.999641    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b33eaba-53ea-47b6-bdfe-3d3d9ffd8a36-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b33eaba-53ea-47b6-bdfe-3d3d9ffd8a36-istio-data\") pod \"9b33eaba-53ea-47b6-bdfe-3d3d9ffd8a36\" (UID: \"9b33eaba-53ea-47b6-bdfe-3d3d9ffd8a36\") "
Nov 02 15:28:40.000376 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:28:40.000000    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b33eaba-53ea-47b6-bdfe-3d3d9ffd8a36-istio-data" pod "9b33eaba-53ea-47b6-bdfe-3d3d9ffd8a36" (UID: "9b33eaba-53ea-47b6-bdfe-3d3d9ffd8a36"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:28:40.000376 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:28:40.000104    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b33eaba-53ea-47b6-bdfe-3d3d9ffd8a36-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b33eaba-53ea-47b6-bdfe-3d3d9ffd8a36-workload-certs\") pod \"9b33eaba-53ea-47b6-bdfe-3d3d9ffd8a36\" (UID: \"9b33eaba-53ea-47b6-bdfe-3d3d9ffd8a36\") "
Nov 02 15:28:40.000376 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:28:40.000244    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b33eaba-53ea-47b6-bdfe-3d3d9ffd8a36-istio-token\" (UniqueName: \"kubernetes.io/projected/9b33eaba-53ea-47b6-bdfe-3d3d9ffd8a36-istio-token\") pod \"9b33eaba-53ea-47b6-bdfe-3d3d9ffd8a36\" (UID: \"9b33eaba-53ea-47b6-bdfe-3d3d9ffd8a36\") "
Nov 02 15:28:40.000376 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:28:40.000348    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b33eaba-53ea-47b6-bdfe-3d3d9ffd8a36-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b33eaba-53ea-47b6-bdfe-3d3d9ffd8a36-credential-socket\") pod \"9b33eaba-53ea-47b6-bdfe-3d3d9ffd8a36\" (UID: \"9b33eaba-53ea-47b6-bdfe-3d3d9ffd8a36\") "
Nov 02 15:28:40.000643 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:28:40.000450    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b33eaba-53ea-47b6-bdfe-3d3d9ffd8a36-workload-certs" pod "9b33eaba-53ea-47b6-bdfe-3d3d9ffd8a36" (UID: "9b33eaba-53ea-47b6-bdfe-3d3d9ffd8a36"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:28:40.001040 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:28:40.001016    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b33eaba-53ea-47b6-bdfe-3d3d9ffd8a36-credential-socket" pod "9b33eaba-53ea-47b6-bdfe-3d3d9ffd8a36" (UID: "9b33eaba-53ea-47b6-bdfe-3d3d9ffd8a36"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:28:40.001227 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:28:40.001207    5304 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b33eaba-53ea-47b6-bdfe-3d3d9ffd8a36-istiod-ca-cert\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:28:40.001287 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:28:40.001233    5304 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b33eaba-53ea-47b6-bdfe-3d3d9ffd8a36-workload-socket\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:28:40.001287 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:28:40.001249    5304 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b33eaba-53ea-47b6-bdfe-3d3d9ffd8a36-istio-data\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:28:40.001287 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:28:40.001262    5304 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b33eaba-53ea-47b6-bdfe-3d3d9ffd8a36-workload-certs\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:28:40.001287 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:28:40.001276    5304 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b33eaba-53ea-47b6-bdfe-3d3d9ffd8a36-credential-socket\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:28:40.002023 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:28:40.001997    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b33eaba-53ea-47b6-bdfe-3d3d9ffd8a36-kube-api-access-fttkn" pod "9b33eaba-53ea-47b6-bdfe-3d3d9ffd8a36" (UID: "9b33eaba-53ea-47b6-bdfe-3d3d9ffd8a36"). InnerVolumeSpecName "kube-api-access-fttkn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:28:40.002857 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:28:40.002832    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9b33eaba-53ea-47b6-bdfe-3d3d9ffd8a36-istio-podinfo" pod "9b33eaba-53ea-47b6-bdfe-3d3d9ffd8a36" (UID: "9b33eaba-53ea-47b6-bdfe-3d3d9ffd8a36"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 15:28:40.003247 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:28:40.003225    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b33eaba-53ea-47b6-bdfe-3d3d9ffd8a36-istio-token" pod "9b33eaba-53ea-47b6-bdfe-3d3d9ffd8a36" (UID: "9b33eaba-53ea-47b6-bdfe-3d3d9ffd8a36"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:28:40.003508 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:28:40.003488    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b33eaba-53ea-47b6-bdfe-3d3d9ffd8a36-istio-envoy" pod "9b33eaba-53ea-47b6-bdfe-3d3d9ffd8a36" (UID: "9b33eaba-53ea-47b6-bdfe-3d3d9ffd8a36"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:28:40.103162 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:28:40.103093    5304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fttkn\" (UniqueName: \"kubernetes.io/projected/9b33eaba-53ea-47b6-bdfe-3d3d9ffd8a36-kube-api-access-fttkn\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:28:40.103162 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:28:40.103165    5304 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b33eaba-53ea-47b6-bdfe-3d3d9ffd8a36-istio-envoy\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:28:40.103363 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:28:40.103182    5304 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b33eaba-53ea-47b6-bdfe-3d3d9ffd8a36-istio-podinfo\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:28:40.103363 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:28:40.103194    5304 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b33eaba-53ea-47b6-bdfe-3d3d9ffd8a36-istio-token\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:28:40.148392 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:28:40.130121    5304 generic.go:356] "Generic (PLEG): container finished" podID="9b33eaba-53ea-47b6-bdfe-3d3d9ffd8a36" containerID="386621ec229173080d14b76e5daaeefd5b35a1b33aeca89d3fecf66ae38a29bf" exitCode=0
Nov 02 15:28:40.148392 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:28:40.130220    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-a713aa-openshift-default-7554947cc9-trs9t"
Nov 02 15:28:40.148392 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:28:40.130284    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a713aa-openshift-default-7554947cc9-trs9t" event={"ID":"9b33eaba-53ea-47b6-bdfe-3d3d9ffd8a36","Type":"ContainerDied","Data":"386621ec229173080d14b76e5daaeefd5b35a1b33aeca89d3fecf66ae38a29bf"}
Nov 02 15:28:40.148392 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:28:40.130333    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a713aa-openshift-default-7554947cc9-trs9t" event={"ID":"9b33eaba-53ea-47b6-bdfe-3d3d9ffd8a36","Type":"ContainerDied","Data":"e7a44dcb202db0de44db94d4b90f50351ba9bffbbb9b461beb4c6bdb5405831f"}
Nov 02 15:28:40.148392 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:28:40.130362    5304 scope.go:122] "RemoveContainer" containerID="386621ec229173080d14b76e5daaeefd5b35a1b33aeca89d3fecf66ae38a29bf"
Nov 02 15:28:40.167410 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:28:40.167388    5304 scope.go:122] "RemoveContainer" containerID="f48f93ebb35e8494ee80b0db35e6df19cf522c1d5e7a8601b57ef28149a900bb"
Nov 02 15:28:40.177987 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:28:40.177959    5304 scope.go:122] "RemoveContainer" containerID="386621ec229173080d14b76e5daaeefd5b35a1b33aeca89d3fecf66ae38a29bf"
Nov 02 15:28:40.178274 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:28:40.178251    5304 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"386621ec229173080d14b76e5daaeefd5b35a1b33aeca89d3fecf66ae38a29bf\": container with ID starting with 386621ec229173080d14b76e5daaeefd5b35a1b33aeca89d3fecf66ae38a29bf not found: ID does not exist" containerID="386621ec229173080d14b76e5daaeefd5b35a1b33aeca89d3fecf66ae38a29bf"
Nov 02 15:28:40.178353 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:28:40.178283    5304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"386621ec229173080d14b76e5daaeefd5b35a1b33aeca89d3fecf66ae38a29bf"} err="failed to get container status \"386621ec229173080d14b76e5daaeefd5b35a1b33aeca89d3fecf66ae38a29bf\": rpc error: code = NotFound desc = could not find container \"386621ec229173080d14b76e5daaeefd5b35a1b33aeca89d3fecf66ae38a29bf\": container with ID starting with 386621ec229173080d14b76e5daaeefd5b35a1b33aeca89d3fecf66ae38a29bf not found: ID does not exist"
Nov 02 15:28:40.178353 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:28:40.178301    5304 scope.go:122] "RemoveContainer" containerID="f48f93ebb35e8494ee80b0db35e6df19cf522c1d5e7a8601b57ef28149a900bb"
Nov 02 15:28:40.178574 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:28:40.178558    5304 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f48f93ebb35e8494ee80b0db35e6df19cf522c1d5e7a8601b57ef28149a900bb\": container with ID starting with f48f93ebb35e8494ee80b0db35e6df19cf522c1d5e7a8601b57ef28149a900bb not found: ID does not exist" containerID="f48f93ebb35e8494ee80b0db35e6df19cf522c1d5e7a8601b57ef28149a900bb"
Nov 02 15:28:40.178616 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:28:40.178580    5304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f48f93ebb35e8494ee80b0db35e6df19cf522c1d5e7a8601b57ef28149a900bb"} err="failed to get container status \"f48f93ebb35e8494ee80b0db35e6df19cf522c1d5e7a8601b57ef28149a900bb\": rpc error: code = NotFound desc = could not find container \"f48f93ebb35e8494ee80b0db35e6df19cf522c1d5e7a8601b57ef28149a900bb\": container with ID starting with f48f93ebb35e8494ee80b0db35e6df19cf522c1d5e7a8601b57ef28149a900bb not found: ID does not exist"
Nov 02 15:28:40.201706 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:28:40.201673    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-a713aa-openshift-default-7554947cc9-trs9t"]
Nov 02 15:28:40.205576 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:28:40.205550    5304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-a713aa-openshift-default-7554947cc9-trs9t"]
Nov 02 15:28:40.559081 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:28:40.558989    5304 patch_prober.go:29] interesting pod/e2e-inf-a-80ec03-openshift-default-66dfdbcf85-b4ltb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" start-of-body=
Nov 02 15:28:40.559081 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:28:40.559044    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-80ec03-openshift-default-66dfdbcf85-b4ltb" podUID="1f9d68a5-7f9c-489b-9289-2f85b7889c2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused"
Nov 02 15:28:41.559107 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:28:41.559066    5304 patch_prober.go:29] interesting pod/e2e-inf-a-80ec03-openshift-default-66dfdbcf85-b4ltb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" start-of-body=
Nov 02 15:28:41.559773 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:28:41.559740    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-80ec03-openshift-default-66dfdbcf85-b4ltb" podUID="1f9d68a5-7f9c-489b-9289-2f85b7889c2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused"
Nov 02 15:28:42.121020 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:28:42.120980    5304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9b33eaba-53ea-47b6-bdfe-3d3d9ffd8a36" path="/var/lib/kubelet/pods/9b33eaba-53ea-47b6-bdfe-3d3d9ffd8a36/volumes"
Nov 02 15:28:42.558293 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:28:42.558198    5304 patch_prober.go:29] interesting pod/e2e-inf-a-80ec03-openshift-default-66dfdbcf85-b4ltb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" start-of-body=
Nov 02 15:28:42.558558 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:28:42.558527    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-80ec03-openshift-default-66dfdbcf85-b4ltb" podUID="1f9d68a5-7f9c-489b-9289-2f85b7889c2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused"
Nov 02 15:28:43.559090 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:28:43.559062    5304 patch_prober.go:29] interesting pod/e2e-inf-a-80ec03-openshift-default-66dfdbcf85-b4ltb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" start-of-body=
Nov 02 15:28:43.559563 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:28:43.559539    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-80ec03-openshift-default-66dfdbcf85-b4ltb" podUID="1f9d68a5-7f9c-489b-9289-2f85b7889c2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused"
Nov 02 15:28:44.558752 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:28:44.558700    5304 patch_prober.go:29] interesting pod/e2e-inf-a-80ec03-openshift-default-66dfdbcf85-b4ltb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" start-of-body=
Nov 02 15:28:44.558958 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:28:44.558770    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-80ec03-openshift-default-66dfdbcf85-b4ltb" podUID="1f9d68a5-7f9c-489b-9289-2f85b7889c2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused"
Nov 02 15:28:44.754759 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:28:44.754713    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-80ec03-openshift-default-66dfdbcf85-b4ltb"
Nov 02 15:28:45.563779 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:28:45.563747    5304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-80ec03-openshift-default-66dfdbcf85-b4ltb"
Nov 02 15:28:45.564904 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:28:45.564882    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-80ec03-openshift-default-66dfdbcf85-b4ltb"
Nov 02 15:28:45.597004 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:28:45.596969    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-2bbcb5-67694d7bc-wzlkw"]
Nov 02 15:28:50.597583 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:28:50.597540    5304 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2bbcb5-67694d7bc-wzlkw" podUID="3567913b-db5f-4a11-9911-9d0fb8dfe57a" containerName="payload-pre-processing" containerID="cri-o://f35301c6ea2b8f1aaac76746e761fe2fcdbc8deaeec1dfea81a78caeebad0b0d" gracePeriod=25
Nov 02 15:28:54.800421 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:28:54.800381    5304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k6fjk" containerName="extractor"
Nov 02 15:29:00.551735 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:29:00.551694    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2bbcb5-67694d7bc-wzlkw" podUID="3567913b-db5f-4a11-9911-9d0fb8dfe57a" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 15:29:10.550999 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:29:10.550953    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2bbcb5-67694d7bc-wzlkw" podUID="3567913b-db5f-4a11-9911-9d0fb8dfe57a" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 15:29:10.886302 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:29:10.886274    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2bbcb5-67694d7bc-wzlkw"
Nov 02 15:29:10.962501 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:29:10.962470    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3567913b-db5f-4a11-9911-9d0fb8dfe57a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3567913b-db5f-4a11-9911-9d0fb8dfe57a-plugins-config-volume\") pod \"3567913b-db5f-4a11-9911-9d0fb8dfe57a\" (UID: \"3567913b-db5f-4a11-9911-9d0fb8dfe57a\") "
Nov 02 15:29:10.962689 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:29:10.962534    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3567913b-db5f-4a11-9911-9d0fb8dfe57a-kube-api-access-6bxn5\" (UniqueName: \"kubernetes.io/projected/3567913b-db5f-4a11-9911-9d0fb8dfe57a-kube-api-access-6bxn5\") pod \"3567913b-db5f-4a11-9911-9d0fb8dfe57a\" (UID: \"3567913b-db5f-4a11-9911-9d0fb8dfe57a\") "
Nov 02 15:29:10.962936 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:29:10.962906    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3567913b-db5f-4a11-9911-9d0fb8dfe57a-plugins-config-volume" pod "3567913b-db5f-4a11-9911-9d0fb8dfe57a" (UID: "3567913b-db5f-4a11-9911-9d0fb8dfe57a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:29:10.963287 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:29:10.963267    5304 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3567913b-db5f-4a11-9911-9d0fb8dfe57a-plugins-config-volume\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:29:10.965385 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:29:10.965356    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3567913b-db5f-4a11-9911-9d0fb8dfe57a-kube-api-access-6bxn5" pod "3567913b-db5f-4a11-9911-9d0fb8dfe57a" (UID: "3567913b-db5f-4a11-9911-9d0fb8dfe57a"). InnerVolumeSpecName "kube-api-access-6bxn5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:29:11.065005 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:29:11.064969    5304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6bxn5\" (UniqueName: \"kubernetes.io/projected/3567913b-db5f-4a11-9911-9d0fb8dfe57a-kube-api-access-6bxn5\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:29:11.420563 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:29:11.420521    5304 generic.go:356] "Generic (PLEG): container finished" podID="3567913b-db5f-4a11-9911-9d0fb8dfe57a" containerID="f35301c6ea2b8f1aaac76746e761fe2fcdbc8deaeec1dfea81a78caeebad0b0d" exitCode=0
Nov 02 15:29:11.420780 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:29:11.420565    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2bbcb5-67694d7bc-wzlkw"
Nov 02 15:29:11.420780 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:29:11.420608    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2bbcb5-67694d7bc-wzlkw" event={"ID":"3567913b-db5f-4a11-9911-9d0fb8dfe57a","Type":"ContainerDied","Data":"f35301c6ea2b8f1aaac76746e761fe2fcdbc8deaeec1dfea81a78caeebad0b0d"}
Nov 02 15:29:11.420780 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:29:11.420662    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2bbcb5-67694d7bc-wzlkw" event={"ID":"3567913b-db5f-4a11-9911-9d0fb8dfe57a","Type":"ContainerDied","Data":"f7fa0aeb6cff2864e605b03207678be4c8a4f13d060610d3203dccaa2e0b11fa"}
Nov 02 15:29:11.420780 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:29:11.420684    5304 scope.go:122] "RemoveContainer" containerID="f35301c6ea2b8f1aaac76746e761fe2fcdbc8deaeec1dfea81a78caeebad0b0d"
Nov 02 15:29:11.431398 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:29:11.431380    5304 scope.go:122] "RemoveContainer" containerID="f35301c6ea2b8f1aaac76746e761fe2fcdbc8deaeec1dfea81a78caeebad0b0d"
Nov 02 15:29:11.431683 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:29:11.431662    5304 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f35301c6ea2b8f1aaac76746e761fe2fcdbc8deaeec1dfea81a78caeebad0b0d\": container with ID starting with f35301c6ea2b8f1aaac76746e761fe2fcdbc8deaeec1dfea81a78caeebad0b0d not found: ID does not exist" containerID="f35301c6ea2b8f1aaac76746e761fe2fcdbc8deaeec1dfea81a78caeebad0b0d"
Nov 02 15:29:11.431739 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:29:11.431690    5304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f35301c6ea2b8f1aaac76746e761fe2fcdbc8deaeec1dfea81a78caeebad0b0d"} err="failed to get container status \"f35301c6ea2b8f1aaac76746e761fe2fcdbc8deaeec1dfea81a78caeebad0b0d\": rpc error: code = NotFound desc = could not find container \"f35301c6ea2b8f1aaac76746e761fe2fcdbc8deaeec1dfea81a78caeebad0b0d\": container with ID starting with f35301c6ea2b8f1aaac76746e761fe2fcdbc8deaeec1dfea81a78caeebad0b0d not found: ID does not exist"
Nov 02 15:29:11.473279 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:29:11.473249    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-2bbcb5-67694d7bc-wzlkw"]
Nov 02 15:29:11.484563 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:29:11.484537    5304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-2bbcb5-67694d7bc-wzlkw"]
Nov 02 15:29:12.120642 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:29:12.120605    5304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3567913b-db5f-4a11-9911-9d0fb8dfe57a" path="/var/lib/kubelet/pods/3567913b-db5f-4a11-9911-9d0fb8dfe57a/volumes"
Nov 02 15:29:22.991144 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:29:22.991093    5304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-2e4ebc-6d869c5c4b-7zfcq"]
Nov 02 15:29:22.991808 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:29:22.991792    5304 cpu_manager.go:414] "containerMap: removing container" podUID="864c95a0-3f74-41bc-8d00-2dd60f1d5cdc" containerName="payload-processing"
Nov 02 15:29:22.991855 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:29:22.991808    5304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="864c95a0-3f74-41bc-8d00-2dd60f1d5cdc" containerName="payload-processing"
Nov 02 15:29:22.991855 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:29:22.991825    5304 cpu_manager.go:414] "containerMap: removing container" podUID="9b33eaba-53ea-47b6-bdfe-3d3d9ffd8a36" containerName="istio-proxy"
Nov 02 15:29:22.991855 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:29:22.991830    5304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b33eaba-53ea-47b6-bdfe-3d3d9ffd8a36" containerName="istio-proxy"
Nov 02 15:29:22.991956 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:29:22.991870    5304 cpu_manager.go:414] "containerMap: removing container" podUID="c41d5c79-8a9b-4ad5-995c-818fd3faf115" containerName="istio-proxy"
Nov 02 15:29:22.991956 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:29:22.991889    5304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c41d5c79-8a9b-4ad5-995c-818fd3faf115" containerName="istio-proxy"
Nov 02 15:29:22.991956 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:29:22.991903    5304 cpu_manager.go:414] "containerMap: removing container" podUID="3567913b-db5f-4a11-9911-9d0fb8dfe57a" containerName="payload-pre-processing"
Nov 02 15:29:22.991956 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:29:22.991910    5304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3567913b-db5f-4a11-9911-9d0fb8dfe57a" containerName="payload-pre-processing"
Nov 02 15:29:22.992072 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:29:22.992016    5304 memory_manager.go:365] "RemoveStaleState removing state" podUID="864c95a0-3f74-41bc-8d00-2dd60f1d5cdc" containerName="payload-processing"
Nov 02 15:29:22.992072 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:29:22.992026    5304 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b33eaba-53ea-47b6-bdfe-3d3d9ffd8a36" containerName="istio-proxy"
Nov 02 15:29:22.992072 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:29:22.992033    5304 memory_manager.go:365] "RemoveStaleState removing state" podUID="c41d5c79-8a9b-4ad5-995c-818fd3faf115" containerName="istio-proxy"
Nov 02 15:29:22.992072 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:29:22.992047    5304 memory_manager.go:365] "RemoveStaleState removing state" podUID="3567913b-db5f-4a11-9911-9d0fb8dfe57a" containerName="payload-pre-processing"
Nov 02 15:29:22.995834 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:29:22.995816    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2e4ebc-6d869c5c4b-7zfcq"
Nov 02 15:29:23.014295 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:29:23.014265    5304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-2e4ebc-serving-cert\""
Nov 02 15:29:23.019719 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:29:23.019687    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-2e4ebc-6d869c5c4b-7zfcq"]
Nov 02 15:29:23.070552 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:29:23.070519    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gfjg7\" (UniqueName: \"kubernetes.io/projected/a48f863f-1237-46a0-9949-0ddecd8b350c-kube-api-access-gfjg7\") pod \"maas-api-e2e-inf-b-2e4ebc-6d869c5c4b-7zfcq\" (UID: \"a48f863f-1237-46a0-9949-0ddecd8b350c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2e4ebc-6d869c5c4b-7zfcq"
Nov 02 15:29:23.070730 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:29:23.070573    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a48f863f-1237-46a0-9949-0ddecd8b350c-openshift-service-ca\") pod \"maas-api-e2e-inf-b-2e4ebc-6d869c5c4b-7zfcq\" (UID: \"a48f863f-1237-46a0-9949-0ddecd8b350c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2e4ebc-6d869c5c4b-7zfcq"
Nov 02 15:29:23.070730 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:29:23.070643    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a48f863f-1237-46a0-9949-0ddecd8b350c-metrics-certs\") pod \"maas-api-e2e-inf-b-2e4ebc-6d869c5c4b-7zfcq\" (UID: \"a48f863f-1237-46a0-9949-0ddecd8b350c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2e4ebc-6d869c5c4b-7zfcq"
Nov 02 15:29:23.070895 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:29:23.070881    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a48f863f-1237-46a0-9949-0ddecd8b350c-maas-api-tls\") pod \"maas-api-e2e-inf-b-2e4ebc-6d869c5c4b-7zfcq\" (UID: \"a48f863f-1237-46a0-9949-0ddecd8b350c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2e4ebc-6d869c5c4b-7zfcq"
Nov 02 15:29:23.181271 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:29:23.179203    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gfjg7\" (UniqueName: \"kubernetes.io/projected/a48f863f-1237-46a0-9949-0ddecd8b350c-kube-api-access-gfjg7\") pod \"maas-api-e2e-inf-b-2e4ebc-6d869c5c4b-7zfcq\" (UID: \"a48f863f-1237-46a0-9949-0ddecd8b350c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2e4ebc-6d869c5c4b-7zfcq"
Nov 02 15:29:23.181271 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:29:23.179296    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a48f863f-1237-46a0-9949-0ddecd8b350c-openshift-service-ca\") pod \"maas-api-e2e-inf-b-2e4ebc-6d869c5c4b-7zfcq\" (UID: \"a48f863f-1237-46a0-9949-0ddecd8b350c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2e4ebc-6d869c5c4b-7zfcq"
Nov 02 15:29:23.181271 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:29:23.179328    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a48f863f-1237-46a0-9949-0ddecd8b350c-metrics-certs\") pod \"maas-api-e2e-inf-b-2e4ebc-6d869c5c4b-7zfcq\" (UID: \"a48f863f-1237-46a0-9949-0ddecd8b350c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2e4ebc-6d869c5c4b-7zfcq"
Nov 02 15:29:23.181271 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:29:23.179671    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a48f863f-1237-46a0-9949-0ddecd8b350c-maas-api-tls\") pod \"maas-api-e2e-inf-b-2e4ebc-6d869c5c4b-7zfcq\" (UID: \"a48f863f-1237-46a0-9949-0ddecd8b350c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2e4ebc-6d869c5c4b-7zfcq"
Nov 02 15:29:23.181271 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:29:23.179940    5304 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-2e4ebc-serving-cert: secret "maas-api-e2e-inf-b-2e4ebc-serving-cert" not found
Nov 02 15:29:23.181271 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:29:23.180008    5304 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a48f863f-1237-46a0-9949-0ddecd8b350c-maas-api-tls podName:a48f863f-1237-46a0-9949-0ddecd8b350c nodeName:}" failed. No retries permitted until 2026-11-02 15:29:23.679987948 +0000 UTC m=+1963.673812358 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a48f863f-1237-46a0-9949-0ddecd8b350c-maas-api-tls") pod "maas-api-e2e-inf-b-2e4ebc-6d869c5c4b-7zfcq" (UID: "a48f863f-1237-46a0-9949-0ddecd8b350c") : secret "maas-api-e2e-inf-b-2e4ebc-serving-cert" not found
Nov 02 15:29:23.183874 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:29:23.183534    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a48f863f-1237-46a0-9949-0ddecd8b350c-openshift-service-ca\") pod \"maas-api-e2e-inf-b-2e4ebc-6d869c5c4b-7zfcq\" (UID: \"a48f863f-1237-46a0-9949-0ddecd8b350c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2e4ebc-6d869c5c4b-7zfcq"
Nov 02 15:29:23.192084 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:29:23.192031    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a48f863f-1237-46a0-9949-0ddecd8b350c-metrics-certs\") pod \"maas-api-e2e-inf-b-2e4ebc-6d869c5c4b-7zfcq\" (UID: \"a48f863f-1237-46a0-9949-0ddecd8b350c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2e4ebc-6d869c5c4b-7zfcq"
Nov 02 15:29:23.209723 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:29:23.209689    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gfjg7\" (UniqueName: \"kubernetes.io/projected/a48f863f-1237-46a0-9949-0ddecd8b350c-kube-api-access-gfjg7\") pod \"maas-api-e2e-inf-b-2e4ebc-6d869c5c4b-7zfcq\" (UID: \"a48f863f-1237-46a0-9949-0ddecd8b350c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2e4ebc-6d869c5c4b-7zfcq"
Nov 02 15:29:23.687346 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:29:23.687314    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a48f863f-1237-46a0-9949-0ddecd8b350c-maas-api-tls\") pod \"maas-api-e2e-inf-b-2e4ebc-6d869c5c4b-7zfcq\" (UID: \"a48f863f-1237-46a0-9949-0ddecd8b350c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2e4ebc-6d869c5c4b-7zfcq"
Nov 02 15:29:23.689774 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:29:23.689738    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a48f863f-1237-46a0-9949-0ddecd8b350c-maas-api-tls\") pod \"maas-api-e2e-inf-b-2e4ebc-6d869c5c4b-7zfcq\" (UID: \"a48f863f-1237-46a0-9949-0ddecd8b350c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2e4ebc-6d869c5c4b-7zfcq"
Nov 02 15:29:23.909885 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:29:23.909853    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2e4ebc-6d869c5c4b-7zfcq"
Nov 02 15:29:24.239732 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:29:24.239707    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-2e4ebc-6d869c5c4b-7zfcq"]
Nov 02 15:29:24.242526 ip-10-0-135-138 kubenswrapper[5304]: W1102 15:29:24.242493    5304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda48f863f_1237_46a0_9949_0ddecd8b350c.slice/crio-3ae1e80f99060bc6eee04d86151fd8a3b0a5d758bd50c22ba76da68f79fd7fef WatchSource:0}: Error finding container 3ae1e80f99060bc6eee04d86151fd8a3b0a5d758bd50c22ba76da68f79fd7fef: Status 404 returned error can't find the container with id 3ae1e80f99060bc6eee04d86151fd8a3b0a5d758bd50c22ba76da68f79fd7fef
Nov 02 15:29:24.244171 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:29:24.244100    5304 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 02 15:29:24.516036 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:29:24.515996    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2e4ebc-6d869c5c4b-7zfcq" event={"ID":"a48f863f-1237-46a0-9949-0ddecd8b350c","Type":"ContainerStarted","Data":"3ae1e80f99060bc6eee04d86151fd8a3b0a5d758bd50c22ba76da68f79fd7fef"}
Nov 02 15:29:24.803082 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:29:24.803000    5304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k6fjk" containerName="extractor"
Nov 02 15:29:25.525980 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:29:25.525943    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2e4ebc-6d869c5c4b-7zfcq" event={"ID":"a48f863f-1237-46a0-9949-0ddecd8b350c","Type":"ContainerStarted","Data":"431bf7259acaecd2ae83969a0f007f36be1e2938e991dbaadfc12461845918df"}
Nov 02 15:29:25.526501 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:29:25.526019    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2e4ebc-6d869c5c4b-7zfcq"
Nov 02 15:29:25.569770 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:29:25.569714    5304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2e4ebc-6d869c5c4b-7zfcq" podStartSLOduration=3.313063895 podStartE2EDuration="3.569701327s" totalImagesPullingTime="256.637432ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:29:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:29:25.56944456 +0000 UTC m=+1965.563268969" watchObservedRunningTime="2026-11-02 15:29:25.569701327 +0000 UTC m=+1965.563525735"
Nov 02 15:29:31.549022 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:29:31.548992    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2e4ebc-6d869c5c4b-7zfcq"
Nov 02 15:29:39.171979 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:29:39.171941    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-2bbcb5-openshift-default-788c5c5b65-bxbrl"]
Nov 02 15:29:39.172613 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:29:39.172232    5304 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-2bbcb5-openshift-default-788c5c5b65-bxbrl" podUID="28cababb-e85d-48e8-b8a3-69e0914862aa" containerName="istio-proxy" containerID="cri-o://fa82afe42a7012cde8fbce10600f1c051b4e2ad5c1fafd8c1f5ba7f39d9503c9" gracePeriod=30
Nov 02 15:29:40.517575 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:29:40.517536    5304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w6-b955c5-openshift-default-9fd99b497-tpwvk"]
Nov 02 15:29:40.519024 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:29:40.518993    5304 cpu_manager.go:414] "containerMap: removing container" podUID="9b33eaba-53ea-47b6-bdfe-3d3d9ffd8a36" containerName="istio-proxy"
Nov 02 15:29:40.519024 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:29:40.519021    5304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b33eaba-53ea-47b6-bdfe-3d3d9ffd8a36" containerName="istio-proxy"
Nov 02 15:29:40.519240 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:29:40.519185    5304 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b33eaba-53ea-47b6-bdfe-3d3d9ffd8a36" containerName="istio-proxy"
Nov 02 15:29:40.522995 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:29:40.522973    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-b955c5-openshift-default-9fd99b497-tpwvk"
Nov 02 15:29:40.548421 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:29:40.548385    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-b955c5-openshift-default-9fd99b497-tpwvk"]
Nov 02 15:29:40.551347 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:29:40.551308    5304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w6-b955c5-openshift-default-dockercfg-fnnwj\""
Nov 02 15:29:40.627215 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:29:40.627175    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/783e5ed2-e421-4d98-87e2-6b57a5662b66-workload-certs\") pod \"e2e-shared-a-w6-b955c5-openshift-default-9fd99b497-tpwvk\" (UID: \"783e5ed2-e421-4d98-87e2-6b57a5662b66\") " pod="openshift-ingress/e2e-shared-a-w6-b955c5-openshift-default-9fd99b497-tpwvk"
Nov 02 15:29:40.627215 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:29:40.627219    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/783e5ed2-e421-4d98-87e2-6b57a5662b66-credential-socket\") pod \"e2e-shared-a-w6-b955c5-openshift-default-9fd99b497-tpwvk\" (UID: \"783e5ed2-e421-4d98-87e2-6b57a5662b66\") " pod="openshift-ingress/e2e-shared-a-w6-b955c5-openshift-default-9fd99b497-tpwvk"
Nov 02 15:29:40.627440 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:29:40.627298    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/783e5ed2-e421-4d98-87e2-6b57a5662b66-istio-token\") pod \"e2e-shared-a-w6-b955c5-openshift-default-9fd99b497-tpwvk\" (UID: \"783e5ed2-e421-4d98-87e2-6b57a5662b66\") " pod="openshift-ingress/e2e-shared-a-w6-b955c5-openshift-default-9fd99b497-tpwvk"
Nov 02 15:29:40.627480 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:29:40.627441    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/783e5ed2-e421-4d98-87e2-6b57a5662b66-istio-envoy\") pod \"e2e-shared-a-w6-b955c5-openshift-default-9fd99b497-tpwvk\" (UID: \"783e5ed2-e421-4d98-87e2-6b57a5662b66\") " pod="openshift-ingress/e2e-shared-a-w6-b955c5-openshift-default-9fd99b497-tpwvk"
Nov 02 15:29:40.627480 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:29:40.627462    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/783e5ed2-e421-4d98-87e2-6b57a5662b66-istio-data\") pod \"e2e-shared-a-w6-b955c5-openshift-default-9fd99b497-tpwvk\" (UID: \"783e5ed2-e421-4d98-87e2-6b57a5662b66\") " pod="openshift-ingress/e2e-shared-a-w6-b955c5-openshift-default-9fd99b497-tpwvk"
Nov 02 15:29:40.627612 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:29:40.627599    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5q6wz\" (UniqueName: \"kubernetes.io/projected/783e5ed2-e421-4d98-87e2-6b57a5662b66-kube-api-access-5q6wz\") pod \"e2e-shared-a-w6-b955c5-openshift-default-9fd99b497-tpwvk\" (UID: \"783e5ed2-e421-4d98-87e2-6b57a5662b66\") " pod="openshift-ingress/e2e-shared-a-w6-b955c5-openshift-default-9fd99b497-tpwvk"
Nov 02 15:29:40.627645 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:29:40.627621    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/783e5ed2-e421-4d98-87e2-6b57a5662b66-istiod-ca-cert\") pod \"e2e-shared-a-w6-b955c5-openshift-default-9fd99b497-tpwvk\" (UID: \"783e5ed2-e421-4d98-87e2-6b57a5662b66\") " pod="openshift-ingress/e2e-shared-a-w6-b955c5-openshift-default-9fd99b497-tpwvk"
Nov 02 15:29:40.627679 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:29:40.627659    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/783e5ed2-e421-4d98-87e2-6b57a5662b66-istio-podinfo\") pod \"e2e-shared-a-w6-b955c5-openshift-default-9fd99b497-tpwvk\" (UID: \"783e5ed2-e421-4d98-87e2-6b57a5662b66\") " pod="openshift-ingress/e2e-shared-a-w6-b955c5-openshift-default-9fd99b497-tpwvk"
Nov 02 15:29:40.627745 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:29:40.627732    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/783e5ed2-e421-4d98-87e2-6b57a5662b66-workload-socket\") pod \"e2e-shared-a-w6-b955c5-openshift-default-9fd99b497-tpwvk\" (UID: \"783e5ed2-e421-4d98-87e2-6b57a5662b66\") " pod="openshift-ingress/e2e-shared-a-w6-b955c5-openshift-default-9fd99b497-tpwvk"
Nov 02 15:29:40.734708 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:29:40.734666    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/783e5ed2-e421-4d98-87e2-6b57a5662b66-workload-certs\") pod \"e2e-shared-a-w6-b955c5-openshift-default-9fd99b497-tpwvk\" (UID: \"783e5ed2-e421-4d98-87e2-6b57a5662b66\") " pod="openshift-ingress/e2e-shared-a-w6-b955c5-openshift-default-9fd99b497-tpwvk"
Nov 02 15:29:40.734708 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:29:40.734718    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/783e5ed2-e421-4d98-87e2-6b57a5662b66-credential-socket\") pod \"e2e-shared-a-w6-b955c5-openshift-default-9fd99b497-tpwvk\" (UID: \"783e5ed2-e421-4d98-87e2-6b57a5662b66\") " pod="openshift-ingress/e2e-shared-a-w6-b955c5-openshift-default-9fd99b497-tpwvk"
Nov 02 15:29:40.735001 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:29:40.734769    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/783e5ed2-e421-4d98-87e2-6b57a5662b66-istio-token\") pod \"e2e-shared-a-w6-b955c5-openshift-default-9fd99b497-tpwvk\" (UID: \"783e5ed2-e421-4d98-87e2-6b57a5662b66\") " pod="openshift-ingress/e2e-shared-a-w6-b955c5-openshift-default-9fd99b497-tpwvk"
Nov 02 15:29:40.735001 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:29:40.734909    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/783e5ed2-e421-4d98-87e2-6b57a5662b66-istio-envoy\") pod \"e2e-shared-a-w6-b955c5-openshift-default-9fd99b497-tpwvk\" (UID: \"783e5ed2-e421-4d98-87e2-6b57a5662b66\") " pod="openshift-ingress/e2e-shared-a-w6-b955c5-openshift-default-9fd99b497-tpwvk"
Nov 02 15:29:40.735001 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:29:40.734959    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/783e5ed2-e421-4d98-87e2-6b57a5662b66-istio-data\") pod \"e2e-shared-a-w6-b955c5-openshift-default-9fd99b497-tpwvk\" (UID: \"783e5ed2-e421-4d98-87e2-6b57a5662b66\") " pod="openshift-ingress/e2e-shared-a-w6-b955c5-openshift-default-9fd99b497-tpwvk"
Nov 02 15:29:40.735223 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:29:40.735183    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5q6wz\" (UniqueName: \"kubernetes.io/projected/783e5ed2-e421-4d98-87e2-6b57a5662b66-kube-api-access-5q6wz\") pod \"e2e-shared-a-w6-b955c5-openshift-default-9fd99b497-tpwvk\" (UID: \"783e5ed2-e421-4d98-87e2-6b57a5662b66\") " pod="openshift-ingress/e2e-shared-a-w6-b955c5-openshift-default-9fd99b497-tpwvk"
Nov 02 15:29:40.735286 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:29:40.735228    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/783e5ed2-e421-4d98-87e2-6b57a5662b66-istiod-ca-cert\") pod \"e2e-shared-a-w6-b955c5-openshift-default-9fd99b497-tpwvk\" (UID: \"783e5ed2-e421-4d98-87e2-6b57a5662b66\") " pod="openshift-ingress/e2e-shared-a-w6-b955c5-openshift-default-9fd99b497-tpwvk"
Nov 02 15:29:40.735286 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:29:40.735268    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/783e5ed2-e421-4d98-87e2-6b57a5662b66-credential-socket\") pod \"e2e-shared-a-w6-b955c5-openshift-default-9fd99b497-tpwvk\" (UID: \"783e5ed2-e421-4d98-87e2-6b57a5662b66\") " pod="openshift-ingress/e2e-shared-a-w6-b955c5-openshift-default-9fd99b497-tpwvk"
Nov 02 15:29:40.735404 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:29:40.735307    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/783e5ed2-e421-4d98-87e2-6b57a5662b66-istio-podinfo\") pod \"e2e-shared-a-w6-b955c5-openshift-default-9fd99b497-tpwvk\" (UID: \"783e5ed2-e421-4d98-87e2-6b57a5662b66\") " pod="openshift-ingress/e2e-shared-a-w6-b955c5-openshift-default-9fd99b497-tpwvk"
Nov 02 15:29:40.735404 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:29:40.735338    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/783e5ed2-e421-4d98-87e2-6b57a5662b66-workload-socket\") pod \"e2e-shared-a-w6-b955c5-openshift-default-9fd99b497-tpwvk\" (UID: \"783e5ed2-e421-4d98-87e2-6b57a5662b66\") " pod="openshift-ingress/e2e-shared-a-w6-b955c5-openshift-default-9fd99b497-tpwvk"
Nov 02 15:29:40.749203 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:29:40.735687    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/783e5ed2-e421-4d98-87e2-6b57a5662b66-istio-data\") pod \"e2e-shared-a-w6-b955c5-openshift-default-9fd99b497-tpwvk\" (UID: \"783e5ed2-e421-4d98-87e2-6b57a5662b66\") " pod="openshift-ingress/e2e-shared-a-w6-b955c5-openshift-default-9fd99b497-tpwvk"
Nov 02 15:29:40.749495 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:29:40.735861    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/783e5ed2-e421-4d98-87e2-6b57a5662b66-workload-socket\") pod \"e2e-shared-a-w6-b955c5-openshift-default-9fd99b497-tpwvk\" (UID: \"783e5ed2-e421-4d98-87e2-6b57a5662b66\") " pod="openshift-ingress/e2e-shared-a-w6-b955c5-openshift-default-9fd99b497-tpwvk"
Nov 02 15:29:40.749658 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:29:40.736755    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/783e5ed2-e421-4d98-87e2-6b57a5662b66-istiod-ca-cert\") pod \"e2e-shared-a-w6-b955c5-openshift-default-9fd99b497-tpwvk\" (UID: \"783e5ed2-e421-4d98-87e2-6b57a5662b66\") " pod="openshift-ingress/e2e-shared-a-w6-b955c5-openshift-default-9fd99b497-tpwvk"
Nov 02 15:29:40.749658 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:29:40.740473    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/783e5ed2-e421-4d98-87e2-6b57a5662b66-istio-envoy\") pod \"e2e-shared-a-w6-b955c5-openshift-default-9fd99b497-tpwvk\" (UID: \"783e5ed2-e421-4d98-87e2-6b57a5662b66\") " pod="openshift-ingress/e2e-shared-a-w6-b955c5-openshift-default-9fd99b497-tpwvk"
Nov 02 15:29:40.749658 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:29:40.745613    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/783e5ed2-e421-4d98-87e2-6b57a5662b66-istio-podinfo\") pod \"e2e-shared-a-w6-b955c5-openshift-default-9fd99b497-tpwvk\" (UID: \"783e5ed2-e421-4d98-87e2-6b57a5662b66\") " pod="openshift-ingress/e2e-shared-a-w6-b955c5-openshift-default-9fd99b497-tpwvk"
Nov 02 15:29:40.749658 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:29:40.747630    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/783e5ed2-e421-4d98-87e2-6b57a5662b66-workload-certs\") pod \"e2e-shared-a-w6-b955c5-openshift-default-9fd99b497-tpwvk\" (UID: \"783e5ed2-e421-4d98-87e2-6b57a5662b66\") " pod="openshift-ingress/e2e-shared-a-w6-b955c5-openshift-default-9fd99b497-tpwvk"
Nov 02 15:29:40.789817 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:29:40.789740    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/783e5ed2-e421-4d98-87e2-6b57a5662b66-istio-token\") pod \"e2e-shared-a-w6-b955c5-openshift-default-9fd99b497-tpwvk\" (UID: \"783e5ed2-e421-4d98-87e2-6b57a5662b66\") " pod="openshift-ingress/e2e-shared-a-w6-b955c5-openshift-default-9fd99b497-tpwvk"
Nov 02 15:29:40.805082 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:29:40.805047    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5q6wz\" (UniqueName: \"kubernetes.io/projected/783e5ed2-e421-4d98-87e2-6b57a5662b66-kube-api-access-5q6wz\") pod \"e2e-shared-a-w6-b955c5-openshift-default-9fd99b497-tpwvk\" (UID: \"783e5ed2-e421-4d98-87e2-6b57a5662b66\") " pod="openshift-ingress/e2e-shared-a-w6-b955c5-openshift-default-9fd99b497-tpwvk"
Nov 02 15:29:40.837461 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:29:40.837428    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-b955c5-openshift-default-9fd99b497-tpwvk"
Nov 02 15:29:41.255022 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:29:41.254998    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-b955c5-openshift-default-9fd99b497-tpwvk"]
Nov 02 15:29:41.256963 ip-10-0-135-138 kubenswrapper[5304]: W1102 15:29:41.256927    5304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod783e5ed2_e421_4d98_87e2_6b57a5662b66.slice/crio-74b9156014112823728effc572cb4aa2048b4f72e8412f3b76f76e88e4e46648 WatchSource:0}: Error finding container 74b9156014112823728effc572cb4aa2048b4f72e8412f3b76f76e88e4e46648: Status 404 returned error can't find the container with id 74b9156014112823728effc572cb4aa2048b4f72e8412f3b76f76e88e4e46648
Nov 02 15:29:41.259872 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:29:41.259835    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:29:41.259959 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:29:41.259905    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:29:41.259959 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:29:41.259934    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:29:41.663816 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:29:41.663777    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-b955c5-openshift-default-9fd99b497-tpwvk" event={"ID":"783e5ed2-e421-4d98-87e2-6b57a5662b66","Type":"ContainerStarted","Data":"b76a218e8ab6e917ec5fcb6945b04f34c456fa3c5c5a1ed0b17b71a32a20b8c8"}
Nov 02 15:29:41.664263 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:29:41.663841    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-b955c5-openshift-default-9fd99b497-tpwvk" event={"ID":"783e5ed2-e421-4d98-87e2-6b57a5662b66","Type":"ContainerStarted","Data":"74b9156014112823728effc572cb4aa2048b4f72e8412f3b76f76e88e4e46648"}
Nov 02 15:29:41.742900 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:29:41.742834    5304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w6-b955c5-openshift-default-9fd99b497-tpwvk" podStartSLOduration=1.742815365 podStartE2EDuration="1.742815365s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:29:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:29:41.727092403 +0000 UTC m=+1981.720916813" watchObservedRunningTime="2026-11-02 15:29:41.742815365 +0000 UTC m=+1981.736639777"
Nov 02 15:29:41.842149 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:29:41.841318    5304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w6-b955c5-openshift-default-9fd99b497-tpwvk"
Nov 02 15:29:42.845869 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:29:42.845838    5304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w6-b955c5-openshift-default-9fd99b497-tpwvk"
Nov 02 15:29:43.685838 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:29:43.685807    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w6-b955c5-openshift-default-9fd99b497-tpwvk"
Nov 02 15:29:43.686967 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:29:43.686935    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w6-b955c5-openshift-default-9fd99b497-tpwvk"
Nov 02 15:29:44.620826 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:29:44.620792    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-2bbcb5-openshift-default-788c5c5b65-bxbrl"
Nov 02 15:29:44.695860 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:29:44.695827    5304 generic.go:356] "Generic (PLEG): container finished" podID="28cababb-e85d-48e8-b8a3-69e0914862aa" containerID="fa82afe42a7012cde8fbce10600f1c051b4e2ad5c1fafd8c1f5ba7f39d9503c9" exitCode=0
Nov 02 15:29:44.695860 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:29:44.695869    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-2bbcb5-openshift-default-788c5c5b65-bxbrl"
Nov 02 15:29:44.696171 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:29:44.695910    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-2bbcb5-openshift-default-788c5c5b65-bxbrl" event={"ID":"28cababb-e85d-48e8-b8a3-69e0914862aa","Type":"ContainerDied","Data":"fa82afe42a7012cde8fbce10600f1c051b4e2ad5c1fafd8c1f5ba7f39d9503c9"}
Nov 02 15:29:44.696171 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:29:44.695965    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-2bbcb5-openshift-default-788c5c5b65-bxbrl" event={"ID":"28cababb-e85d-48e8-b8a3-69e0914862aa","Type":"ContainerDied","Data":"e5588a52b346ffffefdaf9314a2762ebebb6d7e491be2f1507b7a34061737c69"}
Nov 02 15:29:44.696171 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:29:44.695996    5304 scope.go:122] "RemoveContainer" containerID="fa82afe42a7012cde8fbce10600f1c051b4e2ad5c1fafd8c1f5ba7f39d9503c9"
Nov 02 15:29:44.708020 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:29:44.707999    5304 scope.go:122] "RemoveContainer" containerID="fa82afe42a7012cde8fbce10600f1c051b4e2ad5c1fafd8c1f5ba7f39d9503c9"
Nov 02 15:29:44.708387 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:29:44.708359    5304 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa82afe42a7012cde8fbce10600f1c051b4e2ad5c1fafd8c1f5ba7f39d9503c9\": container with ID starting with fa82afe42a7012cde8fbce10600f1c051b4e2ad5c1fafd8c1f5ba7f39d9503c9 not found: ID does not exist" containerID="fa82afe42a7012cde8fbce10600f1c051b4e2ad5c1fafd8c1f5ba7f39d9503c9"
Nov 02 15:29:44.708472 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:29:44.708401    5304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa82afe42a7012cde8fbce10600f1c051b4e2ad5c1fafd8c1f5ba7f39d9503c9"} err="failed to get container status \"fa82afe42a7012cde8fbce10600f1c051b4e2ad5c1fafd8c1f5ba7f39d9503c9\": rpc error: code = NotFound desc = could not find container \"fa82afe42a7012cde8fbce10600f1c051b4e2ad5c1fafd8c1f5ba7f39d9503c9\": container with ID starting with fa82afe42a7012cde8fbce10600f1c051b4e2ad5c1fafd8c1f5ba7f39d9503c9 not found: ID does not exist"
Nov 02 15:29:44.748363 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:29:44.748282    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28cababb-e85d-48e8-b8a3-69e0914862aa-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/28cababb-e85d-48e8-b8a3-69e0914862aa-workload-certs\") pod \"28cababb-e85d-48e8-b8a3-69e0914862aa\" (UID: \"28cababb-e85d-48e8-b8a3-69e0914862aa\") "
Nov 02 15:29:44.748544 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:29:44.748387    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28cababb-e85d-48e8-b8a3-69e0914862aa-istio-token\" (UniqueName: \"kubernetes.io/projected/28cababb-e85d-48e8-b8a3-69e0914862aa-istio-token\") pod \"28cababb-e85d-48e8-b8a3-69e0914862aa\" (UID: \"28cababb-e85d-48e8-b8a3-69e0914862aa\") "
Nov 02 15:29:44.748544 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:29:44.748417    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/28cababb-e85d-48e8-b8a3-69e0914862aa-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/28cababb-e85d-48e8-b8a3-69e0914862aa-istiod-ca-cert\") pod \"28cababb-e85d-48e8-b8a3-69e0914862aa\" (UID: \"28cababb-e85d-48e8-b8a3-69e0914862aa\") "
Nov 02 15:29:44.748544 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:29:44.748487    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/28cababb-e85d-48e8-b8a3-69e0914862aa-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/28cababb-e85d-48e8-b8a3-69e0914862aa-istio-podinfo\") pod \"28cababb-e85d-48e8-b8a3-69e0914862aa\" (UID: \"28cababb-e85d-48e8-b8a3-69e0914862aa\") "
Nov 02 15:29:44.748712 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:29:44.748580    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28cababb-e85d-48e8-b8a3-69e0914862aa-workload-certs" pod "28cababb-e85d-48e8-b8a3-69e0914862aa" (UID: "28cababb-e85d-48e8-b8a3-69e0914862aa"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:29:44.748712 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:29:44.748605    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28cababb-e85d-48e8-b8a3-69e0914862aa-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/28cababb-e85d-48e8-b8a3-69e0914862aa-istio-data\") pod \"28cababb-e85d-48e8-b8a3-69e0914862aa\" (UID: \"28cababb-e85d-48e8-b8a3-69e0914862aa\") "
Nov 02 15:29:44.748827 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:29:44.748728    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28cababb-e85d-48e8-b8a3-69e0914862aa-kube-api-access-8zckp\" (UniqueName: \"kubernetes.io/projected/28cababb-e85d-48e8-b8a3-69e0914862aa-kube-api-access-8zckp\") pod \"28cababb-e85d-48e8-b8a3-69e0914862aa\" (UID: \"28cababb-e85d-48e8-b8a3-69e0914862aa\") "
Nov 02 15:29:44.748827 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:29:44.748801    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28cababb-e85d-48e8-b8a3-69e0914862aa-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/28cababb-e85d-48e8-b8a3-69e0914862aa-workload-socket\") pod \"28cababb-e85d-48e8-b8a3-69e0914862aa\" (UID: \"28cababb-e85d-48e8-b8a3-69e0914862aa\") "
Nov 02 15:29:44.748933 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:29:44.748823    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28cababb-e85d-48e8-b8a3-69e0914862aa-istio-data" pod "28cababb-e85d-48e8-b8a3-69e0914862aa" (UID: "28cababb-e85d-48e8-b8a3-69e0914862aa"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:29:44.748933 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:29:44.748837    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28cababb-e85d-48e8-b8a3-69e0914862aa-istiod-ca-cert" pod "28cababb-e85d-48e8-b8a3-69e0914862aa" (UID: "28cababb-e85d-48e8-b8a3-69e0914862aa"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:29:44.748933 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:29:44.748858    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28cababb-e85d-48e8-b8a3-69e0914862aa-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/28cababb-e85d-48e8-b8a3-69e0914862aa-istio-envoy\") pod \"28cababb-e85d-48e8-b8a3-69e0914862aa\" (UID: \"28cababb-e85d-48e8-b8a3-69e0914862aa\") "
Nov 02 15:29:44.748933 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:29:44.748919    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28cababb-e85d-48e8-b8a3-69e0914862aa-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/28cababb-e85d-48e8-b8a3-69e0914862aa-credential-socket\") pod \"28cababb-e85d-48e8-b8a3-69e0914862aa\" (UID: \"28cababb-e85d-48e8-b8a3-69e0914862aa\") "
Nov 02 15:29:44.749181 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:29:44.749019    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28cababb-e85d-48e8-b8a3-69e0914862aa-workload-socket" pod "28cababb-e85d-48e8-b8a3-69e0914862aa" (UID: "28cababb-e85d-48e8-b8a3-69e0914862aa"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:29:44.749244 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:29:44.749216    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28cababb-e85d-48e8-b8a3-69e0914862aa-credential-socket" pod "28cababb-e85d-48e8-b8a3-69e0914862aa" (UID: "28cababb-e85d-48e8-b8a3-69e0914862aa"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:29:44.749942 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:29:44.749913    5304 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/28cababb-e85d-48e8-b8a3-69e0914862aa-istio-data\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:29:44.749942 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:29:44.749943    5304 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/28cababb-e85d-48e8-b8a3-69e0914862aa-workload-socket\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:29:44.750193 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:29:44.749956    5304 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/28cababb-e85d-48e8-b8a3-69e0914862aa-credential-socket\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:29:44.750193 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:29:44.749965    5304 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/28cababb-e85d-48e8-b8a3-69e0914862aa-workload-certs\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:29:44.750193 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:29:44.749974    5304 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/28cababb-e85d-48e8-b8a3-69e0914862aa-istiod-ca-cert\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:29:44.750861 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:29:44.750836    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28cababb-e85d-48e8-b8a3-69e0914862aa-kube-api-access-8zckp" pod "28cababb-e85d-48e8-b8a3-69e0914862aa" (UID: "28cababb-e85d-48e8-b8a3-69e0914862aa"). InnerVolumeSpecName "kube-api-access-8zckp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:29:44.751110 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:29:44.751092    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/28cababb-e85d-48e8-b8a3-69e0914862aa-istio-podinfo" pod "28cababb-e85d-48e8-b8a3-69e0914862aa" (UID: "28cababb-e85d-48e8-b8a3-69e0914862aa"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 15:29:44.751238 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:29:44.751116    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28cababb-e85d-48e8-b8a3-69e0914862aa-istio-token" pod "28cababb-e85d-48e8-b8a3-69e0914862aa" (UID: "28cababb-e85d-48e8-b8a3-69e0914862aa"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:29:44.751322 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:29:44.751305    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28cababb-e85d-48e8-b8a3-69e0914862aa-istio-envoy" pod "28cababb-e85d-48e8-b8a3-69e0914862aa" (UID: "28cababb-e85d-48e8-b8a3-69e0914862aa"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:29:44.851485 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:29:44.851448    5304 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/28cababb-e85d-48e8-b8a3-69e0914862aa-istio-token\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:29:44.851485 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:29:44.851481    5304 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/28cababb-e85d-48e8-b8a3-69e0914862aa-istio-podinfo\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:29:44.851485 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:29:44.851490    5304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8zckp\" (UniqueName: \"kubernetes.io/projected/28cababb-e85d-48e8-b8a3-69e0914862aa-kube-api-access-8zckp\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:29:44.851742 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:29:44.851501    5304 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/28cababb-e85d-48e8-b8a3-69e0914862aa-istio-envoy\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:29:45.054611 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:29:45.054573    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-2bbcb5-openshift-default-788c5c5b65-bxbrl"]
Nov 02 15:29:45.065955 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:29:45.065924    5304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-2bbcb5-openshift-default-788c5c5b65-bxbrl"]
Nov 02 15:29:46.090566 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:29:46.090534    5304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-b955c5-578fdd955-qxkzl"]
Nov 02 15:29:46.091260 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:29:46.091245    5304 cpu_manager.go:414] "containerMap: removing container" podUID="28cababb-e85d-48e8-b8a3-69e0914862aa" containerName="istio-proxy"
Nov 02 15:29:46.091315 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:29:46.091261    5304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28cababb-e85d-48e8-b8a3-69e0914862aa" containerName="istio-proxy"
Nov 02 15:29:46.091434 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:29:46.091424    5304 memory_manager.go:365] "RemoveStaleState removing state" podUID="28cababb-e85d-48e8-b8a3-69e0914862aa" containerName="istio-proxy"
Nov 02 15:29:46.096345 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:29:46.096325    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-b955c5-578fdd955-qxkzl"
Nov 02 15:29:46.112387 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:29:46.112358    5304 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w6-b955c5\""
Nov 02 15:29:46.124444 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:29:46.124407    5304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="28cababb-e85d-48e8-b8a3-69e0914862aa" path="/var/lib/kubelet/pods/28cababb-e85d-48e8-b8a3-69e0914862aa/volumes"
Nov 02 15:29:46.124920 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:29:46.124899    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-b955c5-578fdd955-qxkzl"]
Nov 02 15:29:46.172298 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:29:46.172255    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6752f\" (UniqueName: \"kubernetes.io/projected/52064ae5-d62f-42f6-a50e-5030d6382aa3-kube-api-access-6752f\") pod \"payload-pre-processing-e2e-shared-a-w6-b955c5-578fdd955-qxkzl\" (UID: \"52064ae5-d62f-42f6-a50e-5030d6382aa3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-b955c5-578fdd955-qxkzl"
Nov 02 15:29:46.172578 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:29:46.172551    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52064ae5-d62f-42f6-a50e-5030d6382aa3-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w6-b955c5-578fdd955-qxkzl\" (UID: \"52064ae5-d62f-42f6-a50e-5030d6382aa3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-b955c5-578fdd955-qxkzl"
Nov 02 15:29:46.199734 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:29:46.199622    5304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w6-b955c5-dockercfg-7c5xw\""
Nov 02 15:29:46.373794 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:29:46.373588    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52064ae5-d62f-42f6-a50e-5030d6382aa3-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w6-b955c5-578fdd955-qxkzl\" (UID: \"52064ae5-d62f-42f6-a50e-5030d6382aa3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-b955c5-578fdd955-qxkzl"
Nov 02 15:29:46.374378 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:29:46.374348    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6752f\" (UniqueName: \"kubernetes.io/projected/52064ae5-d62f-42f6-a50e-5030d6382aa3-kube-api-access-6752f\") pod \"payload-pre-processing-e2e-shared-a-w6-b955c5-578fdd955-qxkzl\" (UID: \"52064ae5-d62f-42f6-a50e-5030d6382aa3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-b955c5-578fdd955-qxkzl"
Nov 02 15:29:46.381749 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:29:46.376023    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52064ae5-d62f-42f6-a50e-5030d6382aa3-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w6-b955c5-578fdd955-qxkzl\" (UID: \"52064ae5-d62f-42f6-a50e-5030d6382aa3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-b955c5-578fdd955-qxkzl"
Nov 02 15:29:46.429476 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:29:46.429437    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6752f\" (UniqueName: \"kubernetes.io/projected/52064ae5-d62f-42f6-a50e-5030d6382aa3-kube-api-access-6752f\") pod \"payload-pre-processing-e2e-shared-a-w6-b955c5-578fdd955-qxkzl\" (UID: \"52064ae5-d62f-42f6-a50e-5030d6382aa3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-b955c5-578fdd955-qxkzl"
Nov 02 15:29:46.710182 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:29:46.710054    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-b955c5-578fdd955-qxkzl"
Nov 02 15:29:47.020898 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:29:47.020875    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-b955c5-578fdd955-qxkzl"]
Nov 02 15:29:47.023252 ip-10-0-135-138 kubenswrapper[5304]: W1102 15:29:47.023222    5304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52064ae5_d62f_42f6_a50e_5030d6382aa3.slice/crio-6fe52feaff20589dc9aae37856f47506176bc0e944c1ac704c4dc2b8f7b56bdd WatchSource:0}: Error finding container 6fe52feaff20589dc9aae37856f47506176bc0e944c1ac704c4dc2b8f7b56bdd: Status 404 returned error can't find the container with id 6fe52feaff20589dc9aae37856f47506176bc0e944c1ac704c4dc2b8f7b56bdd
Nov 02 15:29:47.721559 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:29:47.721518    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-b955c5-578fdd955-qxkzl" event={"ID":"52064ae5-d62f-42f6-a50e-5030d6382aa3","Type":"ContainerStarted","Data":"80666ffdeeb5ff716fcebe7423c23bcbdf7346f148a5e923df725361c0c89214"}
Nov 02 15:29:47.721992 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:29:47.721608    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-b955c5-578fdd955-qxkzl" event={"ID":"52064ae5-d62f-42f6-a50e-5030d6382aa3","Type":"ContainerStarted","Data":"6fe52feaff20589dc9aae37856f47506176bc0e944c1ac704c4dc2b8f7b56bdd"}
Nov 02 15:29:47.721992 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:29:47.721645    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-b955c5-578fdd955-qxkzl"
Nov 02 15:29:47.769674 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:29:47.769608    5304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-b955c5-578fdd955-qxkzl" podStartSLOduration=1.769589729 podStartE2EDuration="1.769589729s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:29:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:29:47.764945748 +0000 UTC m=+1987.758770160" watchObservedRunningTime="2026-11-02 15:29:47.769589729 +0000 UTC m=+1987.763414139"
Nov 02 15:29:54.800637 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:29:54.800557    5304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k6fjk" containerName="extractor"
Nov 02 15:29:58.732147 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:29:58.732100    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-b955c5-578fdd955-qxkzl"
Nov 02 15:30:00.205988 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:00.205957    5304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-501738-29893890-59r7h"]
Nov 02 15:30:00.211931 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:00.211907    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-501738-29893890-59r7h"
Nov 02 15:30:00.233661 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:00.233626    5304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-fda646-29893890-g8wfc"]
Nov 02 15:30:00.244376 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:00.244348    5304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-246137-29893890-h98qk"]
Nov 02 15:30:00.244630 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:00.244463    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-fda646-29893890-g8wfc"
Nov 02 15:30:00.250463 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:00.250112    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-501738-29893890-59r7h"]
Nov 02 15:30:00.250463 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:00.250258    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-246137-29893890-h98qk"
Nov 02 15:30:00.258311 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:00.258285    5304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8a3fb0-29893890-t64lm"]
Nov 02 15:30:00.263689 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:00.263662    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8a3fb0-29893890-t64lm"
Nov 02 15:30:00.270235 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:00.270214    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-246137-29893890-h98qk"]
Nov 02 15:30:00.277244 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:00.277216    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-fda646-29893890-g8wfc"]
Nov 02 15:30:00.285112 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:00.285086    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8a3fb0-29893890-t64lm"]
Nov 02 15:30:00.302200 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:00.302172    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-24fdv\" (UniqueName: \"kubernetes.io/projected/875ff34a-5486-4647-bbc6-bbd0df2633e6-kube-api-access-24fdv\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-8a3fb0-29893890-t64lm\" (UID: \"875ff34a-5486-4647-bbc6-bbd0df2633e6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8a3fb0-29893890-t64lm"
Nov 02 15:30:00.302356 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:00.302212    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cpflk\" (UniqueName: \"kubernetes.io/projected/3b856b33-22ae-40c4-b3a5-8d90c81a16de-kube-api-access-cpflk\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-fda646-29893890-g8wfc\" (UID: \"3b856b33-22ae-40c4-b3a5-8d90c81a16de\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-fda646-29893890-g8wfc"
Nov 02 15:30:00.302356 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:00.302247    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b61992b-ffa1-4606-bf49-9efccefbc460-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-501738-29893890-59r7h\" (UID: \"8b61992b-ffa1-4606-bf49-9efccefbc460\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-501738-29893890-59r7h"
Nov 02 15:30:00.302356 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:00.302315    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/875ff34a-5486-4647-bbc6-bbd0df2633e6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-8a3fb0-29893890-t64lm\" (UID: \"875ff34a-5486-4647-bbc6-bbd0df2633e6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8a3fb0-29893890-t64lm"
Nov 02 15:30:00.302356 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:00.302352    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6wqjm\" (UniqueName: \"kubernetes.io/projected/8f70c555-72cb-453b-8d47-8bdbe39222e7-kube-api-access-6wqjm\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-246137-29893890-h98qk\" (UID: \"8f70c555-72cb-453b-8d47-8bdbe39222e7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-246137-29893890-h98qk"
Nov 02 15:30:00.302591 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:00.302390    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b856b33-22ae-40c4-b3a5-8d90c81a16de-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-fda646-29893890-g8wfc\" (UID: \"3b856b33-22ae-40c4-b3a5-8d90c81a16de\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-fda646-29893890-g8wfc"
Nov 02 15:30:00.302591 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:00.302410    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jzvsn\" (UniqueName: \"kubernetes.io/projected/8b61992b-ffa1-4606-bf49-9efccefbc460-kube-api-access-jzvsn\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-501738-29893890-59r7h\" (UID: \"8b61992b-ffa1-4606-bf49-9efccefbc460\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-501738-29893890-59r7h"
Nov 02 15:30:00.302591 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:00.302451    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f70c555-72cb-453b-8d47-8bdbe39222e7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-246137-29893890-h98qk\" (UID: \"8f70c555-72cb-453b-8d47-8bdbe39222e7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-246137-29893890-h98qk"
Nov 02 15:30:00.311275 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:00.311243    5304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-m5926\""
Nov 02 15:30:00.313072 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:00.313042    5304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-6af50f-29893890-95trm"]
Nov 02 15:30:00.318146 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:00.318107    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-6af50f-29893890-95trm"
Nov 02 15:30:00.336000 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:00.335971    5304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5dd8e9-29893890-wgcbb"]
Nov 02 15:30:00.341092 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:00.341065    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5dd8e9-29893890-wgcbb"
Nov 02 15:30:00.358522 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:00.358372    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-6af50f-29893890-95trm"]
Nov 02 15:30:00.368606 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:00.368577    5304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-624254-29893890-hxj28"]
Nov 02 15:30:00.374267 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:00.374241    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-624254-29893890-hxj28"
Nov 02 15:30:00.374408 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:00.374356    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5dd8e9-29893890-wgcbb"]
Nov 02 15:30:00.394945 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:00.394918    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-624254-29893890-hxj28"]
Nov 02 15:30:00.399660 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:00.399448    5304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-28f621-29893890-8pdnl"]
Nov 02 15:30:00.403706 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:00.403682    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/875ff34a-5486-4647-bbc6-bbd0df2633e6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-8a3fb0-29893890-t64lm\" (UID: \"875ff34a-5486-4647-bbc6-bbd0df2633e6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8a3fb0-29893890-t64lm"
Nov 02 15:30:00.403889 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:00.403749    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6wqjm\" (UniqueName: \"kubernetes.io/projected/8f70c555-72cb-453b-8d47-8bdbe39222e7-kube-api-access-6wqjm\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-246137-29893890-h98qk\" (UID: \"8f70c555-72cb-453b-8d47-8bdbe39222e7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-246137-29893890-h98qk"
Nov 02 15:30:00.403889 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:00.403799    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2b513e5-c608-45e2-8dc6-d798bf902d83-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-624254-29893890-hxj28\" (UID: \"b2b513e5-c608-45e2-8dc6-d798bf902d83\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-624254-29893890-hxj28"
Nov 02 15:30:00.403889 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:00.403839    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b856b33-22ae-40c4-b3a5-8d90c81a16de-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-fda646-29893890-g8wfc\" (UID: \"3b856b33-22ae-40c4-b3a5-8d90c81a16de\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-fda646-29893890-g8wfc"
Nov 02 15:30:00.403889 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:00.403868    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jzvsn\" (UniqueName: \"kubernetes.io/projected/8b61992b-ffa1-4606-bf49-9efccefbc460-kube-api-access-jzvsn\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-501738-29893890-59r7h\" (UID: \"8b61992b-ffa1-4606-bf49-9efccefbc460\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-501738-29893890-59r7h"
Nov 02 15:30:00.404324 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:00.403903    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jg4bc\" (UniqueName: \"kubernetes.io/projected/c0293ead-8a17-4860-8691-12a4bf2d0ed6-kube-api-access-jg4bc\") pod \"maas-api-key-cleanup-e2e-worker-w0-6af50f-29893890-95trm\" (UID: \"c0293ead-8a17-4860-8691-12a4bf2d0ed6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-6af50f-29893890-95trm"
Nov 02 15:30:00.404324 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:00.403967    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f70c555-72cb-453b-8d47-8bdbe39222e7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-246137-29893890-h98qk\" (UID: \"8f70c555-72cb-453b-8d47-8bdbe39222e7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-246137-29893890-h98qk"
Nov 02 15:30:00.404324 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:00.404046    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fecd4670-5e5f-477b-9f3c-689d6bdb79e4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-5dd8e9-29893890-wgcbb\" (UID: \"fecd4670-5e5f-477b-9f3c-689d6bdb79e4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5dd8e9-29893890-wgcbb"
Nov 02 15:30:00.404324 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:00.404166    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9vtr9\" (UniqueName: \"kubernetes.io/projected/fecd4670-5e5f-477b-9f3c-689d6bdb79e4-kube-api-access-9vtr9\") pod \"maas-api-key-cleanup-e2e-worker-w1-5dd8e9-29893890-wgcbb\" (UID: \"fecd4670-5e5f-477b-9f3c-689d6bdb79e4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5dd8e9-29893890-wgcbb"
Nov 02 15:30:00.404324 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:00.404201    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-24fdv\" (UniqueName: \"kubernetes.io/projected/875ff34a-5486-4647-bbc6-bbd0df2633e6-kube-api-access-24fdv\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-8a3fb0-29893890-t64lm\" (UID: \"875ff34a-5486-4647-bbc6-bbd0df2633e6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8a3fb0-29893890-t64lm"
Nov 02 15:30:00.404324 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:00.404237    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cpflk\" (UniqueName: \"kubernetes.io/projected/3b856b33-22ae-40c4-b3a5-8d90c81a16de-kube-api-access-cpflk\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-fda646-29893890-g8wfc\" (UID: \"3b856b33-22ae-40c4-b3a5-8d90c81a16de\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-fda646-29893890-g8wfc"
Nov 02 15:30:00.404324 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:00.404289    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b61992b-ffa1-4606-bf49-9efccefbc460-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-501738-29893890-59r7h\" (UID: \"8b61992b-ffa1-4606-bf49-9efccefbc460\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-501738-29893890-59r7h"
Nov 02 15:30:00.404713 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:00.404351    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-44dq2\" (UniqueName: \"kubernetes.io/projected/b2b513e5-c608-45e2-8dc6-d798bf902d83-kube-api-access-44dq2\") pod \"maas-api-key-cleanup-e2e-worker-w3-624254-29893890-hxj28\" (UID: \"b2b513e5-c608-45e2-8dc6-d798bf902d83\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-624254-29893890-hxj28"
Nov 02 15:30:00.404713 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:00.404404    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0293ead-8a17-4860-8691-12a4bf2d0ed6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-6af50f-29893890-95trm\" (UID: \"c0293ead-8a17-4860-8691-12a4bf2d0ed6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-6af50f-29893890-95trm"
Nov 02 15:30:00.404713 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:00.404491    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/875ff34a-5486-4647-bbc6-bbd0df2633e6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-8a3fb0-29893890-t64lm\" (UID: \"875ff34a-5486-4647-bbc6-bbd0df2633e6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8a3fb0-29893890-t64lm"
Nov 02 15:30:00.405488 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:00.405465    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b856b33-22ae-40c4-b3a5-8d90c81a16de-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-fda646-29893890-g8wfc\" (UID: \"3b856b33-22ae-40c4-b3a5-8d90c81a16de\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-fda646-29893890-g8wfc"
Nov 02 15:30:00.406608 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:00.406583    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f70c555-72cb-453b-8d47-8bdbe39222e7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-246137-29893890-h98qk\" (UID: \"8f70c555-72cb-453b-8d47-8bdbe39222e7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-246137-29893890-h98qk"
Nov 02 15:30:00.406779 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:00.406618    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b61992b-ffa1-4606-bf49-9efccefbc460-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-501738-29893890-59r7h\" (UID: \"8b61992b-ffa1-4606-bf49-9efccefbc460\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-501738-29893890-59r7h"
Nov 02 15:30:00.407359 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:00.407335    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-28f621-29893890-8pdnl"
Nov 02 15:30:00.414797 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:00.414750    5304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2e4ebc-29893890-qbrcz"]
Nov 02 15:30:00.435720 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:00.435686    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2e4ebc-29893890-qbrcz"
Nov 02 15:30:00.451283 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:00.450872    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jzvsn\" (UniqueName: \"kubernetes.io/projected/8b61992b-ffa1-4606-bf49-9efccefbc460-kube-api-access-jzvsn\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-501738-29893890-59r7h\" (UID: \"8b61992b-ffa1-4606-bf49-9efccefbc460\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-501738-29893890-59r7h"
Nov 02 15:30:00.452822 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:00.452773    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6wqjm\" (UniqueName: \"kubernetes.io/projected/8f70c555-72cb-453b-8d47-8bdbe39222e7-kube-api-access-6wqjm\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-246137-29893890-h98qk\" (UID: \"8f70c555-72cb-453b-8d47-8bdbe39222e7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-246137-29893890-h98qk"
Nov 02 15:30:00.453055 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:00.452787    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cpflk\" (UniqueName: \"kubernetes.io/projected/3b856b33-22ae-40c4-b3a5-8d90c81a16de-kube-api-access-cpflk\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-fda646-29893890-g8wfc\" (UID: \"3b856b33-22ae-40c4-b3a5-8d90c81a16de\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-fda646-29893890-g8wfc"
Nov 02 15:30:00.453055 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:00.452955    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-28f621-29893890-8pdnl"]
Nov 02 15:30:00.454096 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:00.454071    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-24fdv\" (UniqueName: \"kubernetes.io/projected/875ff34a-5486-4647-bbc6-bbd0df2633e6-kube-api-access-24fdv\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-8a3fb0-29893890-t64lm\" (UID: \"875ff34a-5486-4647-bbc6-bbd0df2633e6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8a3fb0-29893890-t64lm"
Nov 02 15:30:00.466254 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:00.466167    5304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7ecfe1-29893890-bxmzk"]
Nov 02 15:30:00.474982 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:00.474956    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2e4ebc-29893890-qbrcz"]
Nov 02 15:30:00.475198 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:00.475081    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7ecfe1-29893890-bxmzk"
Nov 02 15:30:00.475333 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:00.475316    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7ecfe1-29893890-bxmzk"]
Nov 02 15:30:00.505456 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:00.505424    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xbvrq\" (UniqueName: \"kubernetes.io/projected/5341d918-4b2b-4c8d-8898-0e764f23c230-kube-api-access-xbvrq\") pod \"maas-api-key-cleanup-e2e-inf-b-2e4ebc-29893890-qbrcz\" (UID: \"5341d918-4b2b-4c8d-8898-0e764f23c230\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2e4ebc-29893890-qbrcz"
Nov 02 15:30:00.505589 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:00.505572    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2b513e5-c608-45e2-8dc6-d798bf902d83-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-624254-29893890-hxj28\" (UID: \"b2b513e5-c608-45e2-8dc6-d798bf902d83\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-624254-29893890-hxj28"
Nov 02 15:30:00.505688 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:00.505644    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jg4bc\" (UniqueName: \"kubernetes.io/projected/c0293ead-8a17-4860-8691-12a4bf2d0ed6-kube-api-access-jg4bc\") pod \"maas-api-key-cleanup-e2e-worker-w0-6af50f-29893890-95trm\" (UID: \"c0293ead-8a17-4860-8691-12a4bf2d0ed6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-6af50f-29893890-95trm"
Nov 02 15:30:00.505751 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:00.505719    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5341d918-4b2b-4c8d-8898-0e764f23c230-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-2e4ebc-29893890-qbrcz\" (UID: \"5341d918-4b2b-4c8d-8898-0e764f23c230\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2e4ebc-29893890-qbrcz"
Nov 02 15:30:00.505817 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:00.505754    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/33921120-0f00-4584-9e1d-d486c475bd10-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-7ecfe1-29893890-bxmzk\" (UID: \"33921120-0f00-4584-9e1d-d486c475bd10\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7ecfe1-29893890-bxmzk"
Nov 02 15:30:00.505817 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:00.505808    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zzwx9\" (UniqueName: \"kubernetes.io/projected/33921120-0f00-4584-9e1d-d486c475bd10-kube-api-access-zzwx9\") pod \"maas-api-key-cleanup-e2e-worker-w4-7ecfe1-29893890-bxmzk\" (UID: \"33921120-0f00-4584-9e1d-d486c475bd10\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7ecfe1-29893890-bxmzk"
Nov 02 15:30:00.505903 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:00.505847    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fecd4670-5e5f-477b-9f3c-689d6bdb79e4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-5dd8e9-29893890-wgcbb\" (UID: \"fecd4670-5e5f-477b-9f3c-689d6bdb79e4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5dd8e9-29893890-wgcbb"
Nov 02 15:30:00.506054 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:00.506030    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g6kl4\" (UniqueName: \"kubernetes.io/projected/d6984b63-9bb9-4081-a3ba-51be6260e1ea-kube-api-access-g6kl4\") pod \"maas-api-key-cleanup-e2e-inf-a-28f621-29893890-8pdnl\" (UID: \"d6984b63-9bb9-4081-a3ba-51be6260e1ea\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-28f621-29893890-8pdnl"
Nov 02 15:30:00.506177 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:00.506157    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d6984b63-9bb9-4081-a3ba-51be6260e1ea-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-28f621-29893890-8pdnl\" (UID: \"d6984b63-9bb9-4081-a3ba-51be6260e1ea\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-28f621-29893890-8pdnl"
Nov 02 15:30:00.506278 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:00.506258    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9vtr9\" (UniqueName: \"kubernetes.io/projected/fecd4670-5e5f-477b-9f3c-689d6bdb79e4-kube-api-access-9vtr9\") pod \"maas-api-key-cleanup-e2e-worker-w1-5dd8e9-29893890-wgcbb\" (UID: \"fecd4670-5e5f-477b-9f3c-689d6bdb79e4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5dd8e9-29893890-wgcbb"
Nov 02 15:30:00.506345 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:00.506296    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2b513e5-c608-45e2-8dc6-d798bf902d83-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-624254-29893890-hxj28\" (UID: \"b2b513e5-c608-45e2-8dc6-d798bf902d83\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-624254-29893890-hxj28"
Nov 02 15:30:00.506398 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:00.506377    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-44dq2\" (UniqueName: \"kubernetes.io/projected/b2b513e5-c608-45e2-8dc6-d798bf902d83-kube-api-access-44dq2\") pod \"maas-api-key-cleanup-e2e-worker-w3-624254-29893890-hxj28\" (UID: \"b2b513e5-c608-45e2-8dc6-d798bf902d83\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-624254-29893890-hxj28"
Nov 02 15:30:00.506460 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:00.506428    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0293ead-8a17-4860-8691-12a4bf2d0ed6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-6af50f-29893890-95trm\" (UID: \"c0293ead-8a17-4860-8691-12a4bf2d0ed6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-6af50f-29893890-95trm"
Nov 02 15:30:00.506576 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:00.506467    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fecd4670-5e5f-477b-9f3c-689d6bdb79e4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-5dd8e9-29893890-wgcbb\" (UID: \"fecd4670-5e5f-477b-9f3c-689d6bdb79e4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5dd8e9-29893890-wgcbb"
Nov 02 15:30:00.507211 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:00.507189    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0293ead-8a17-4860-8691-12a4bf2d0ed6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-6af50f-29893890-95trm\" (UID: \"c0293ead-8a17-4860-8691-12a4bf2d0ed6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-6af50f-29893890-95trm"
Nov 02 15:30:00.529344 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:00.529319    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9vtr9\" (UniqueName: \"kubernetes.io/projected/fecd4670-5e5f-477b-9f3c-689d6bdb79e4-kube-api-access-9vtr9\") pod \"maas-api-key-cleanup-e2e-worker-w1-5dd8e9-29893890-wgcbb\" (UID: \"fecd4670-5e5f-477b-9f3c-689d6bdb79e4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5dd8e9-29893890-wgcbb"
Nov 02 15:30:00.529440 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:00.529376    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-44dq2\" (UniqueName: \"kubernetes.io/projected/b2b513e5-c608-45e2-8dc6-d798bf902d83-kube-api-access-44dq2\") pod \"maas-api-key-cleanup-e2e-worker-w3-624254-29893890-hxj28\" (UID: \"b2b513e5-c608-45e2-8dc6-d798bf902d83\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-624254-29893890-hxj28"
Nov 02 15:30:00.529501 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:00.529481    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jg4bc\" (UniqueName: \"kubernetes.io/projected/c0293ead-8a17-4860-8691-12a4bf2d0ed6-kube-api-access-jg4bc\") pod \"maas-api-key-cleanup-e2e-worker-w0-6af50f-29893890-95trm\" (UID: \"c0293ead-8a17-4860-8691-12a4bf2d0ed6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-6af50f-29893890-95trm"
Nov 02 15:30:00.531118 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:00.531101    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-501738-29893890-59r7h"
Nov 02 15:30:00.571815 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:00.571782    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-fda646-29893890-g8wfc"
Nov 02 15:30:00.591593 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:00.591559    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-246137-29893890-h98qk"
Nov 02 15:30:00.629900 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:00.629866    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xbvrq\" (UniqueName: \"kubernetes.io/projected/5341d918-4b2b-4c8d-8898-0e764f23c230-kube-api-access-xbvrq\") pod \"maas-api-key-cleanup-e2e-inf-b-2e4ebc-29893890-qbrcz\" (UID: \"5341d918-4b2b-4c8d-8898-0e764f23c230\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2e4ebc-29893890-qbrcz"
Nov 02 15:30:00.630105 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:00.630034    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5341d918-4b2b-4c8d-8898-0e764f23c230-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-2e4ebc-29893890-qbrcz\" (UID: \"5341d918-4b2b-4c8d-8898-0e764f23c230\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2e4ebc-29893890-qbrcz"
Nov 02 15:30:00.630105 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:00.630068    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/33921120-0f00-4584-9e1d-d486c475bd10-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-7ecfe1-29893890-bxmzk\" (UID: \"33921120-0f00-4584-9e1d-d486c475bd10\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7ecfe1-29893890-bxmzk"
Nov 02 15:30:00.630323 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:00.630119    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zzwx9\" (UniqueName: \"kubernetes.io/projected/33921120-0f00-4584-9e1d-d486c475bd10-kube-api-access-zzwx9\") pod \"maas-api-key-cleanup-e2e-worker-w4-7ecfe1-29893890-bxmzk\" (UID: \"33921120-0f00-4584-9e1d-d486c475bd10\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7ecfe1-29893890-bxmzk"
Nov 02 15:30:00.630323 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:00.630187    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g6kl4\" (UniqueName: \"kubernetes.io/projected/d6984b63-9bb9-4081-a3ba-51be6260e1ea-kube-api-access-g6kl4\") pod \"maas-api-key-cleanup-e2e-inf-a-28f621-29893890-8pdnl\" (UID: \"d6984b63-9bb9-4081-a3ba-51be6260e1ea\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-28f621-29893890-8pdnl"
Nov 02 15:30:00.630323 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:00.630231    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d6984b63-9bb9-4081-a3ba-51be6260e1ea-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-28f621-29893890-8pdnl\" (UID: \"d6984b63-9bb9-4081-a3ba-51be6260e1ea\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-28f621-29893890-8pdnl"
Nov 02 15:30:00.631024 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:00.630986    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d6984b63-9bb9-4081-a3ba-51be6260e1ea-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-28f621-29893890-8pdnl\" (UID: \"d6984b63-9bb9-4081-a3ba-51be6260e1ea\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-28f621-29893890-8pdnl"
Nov 02 15:30:00.639047 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:00.639022    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/33921120-0f00-4584-9e1d-d486c475bd10-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-7ecfe1-29893890-bxmzk\" (UID: \"33921120-0f00-4584-9e1d-d486c475bd10\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7ecfe1-29893890-bxmzk"
Nov 02 15:30:00.639733 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:00.639703    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5341d918-4b2b-4c8d-8898-0e764f23c230-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-2e4ebc-29893890-qbrcz\" (UID: \"5341d918-4b2b-4c8d-8898-0e764f23c230\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2e4ebc-29893890-qbrcz"
Nov 02 15:30:00.639893 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:00.639873    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8a3fb0-29893890-t64lm"
Nov 02 15:30:00.640586 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:00.640564    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-6af50f-29893890-95trm"
Nov 02 15:30:00.669311 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:00.669238    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xbvrq\" (UniqueName: \"kubernetes.io/projected/5341d918-4b2b-4c8d-8898-0e764f23c230-kube-api-access-xbvrq\") pod \"maas-api-key-cleanup-e2e-inf-b-2e4ebc-29893890-qbrcz\" (UID: \"5341d918-4b2b-4c8d-8898-0e764f23c230\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2e4ebc-29893890-qbrcz"
Nov 02 15:30:00.671678 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:00.671649    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g6kl4\" (UniqueName: \"kubernetes.io/projected/d6984b63-9bb9-4081-a3ba-51be6260e1ea-kube-api-access-g6kl4\") pod \"maas-api-key-cleanup-e2e-inf-a-28f621-29893890-8pdnl\" (UID: \"d6984b63-9bb9-4081-a3ba-51be6260e1ea\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-28f621-29893890-8pdnl"
Nov 02 15:30:00.672629 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:00.672576    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5dd8e9-29893890-wgcbb"
Nov 02 15:30:00.673599 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:00.673570    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zzwx9\" (UniqueName: \"kubernetes.io/projected/33921120-0f00-4584-9e1d-d486c475bd10-kube-api-access-zzwx9\") pod \"maas-api-key-cleanup-e2e-worker-w4-7ecfe1-29893890-bxmzk\" (UID: \"33921120-0f00-4584-9e1d-d486c475bd10\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7ecfe1-29893890-bxmzk"
Nov 02 15:30:00.709224 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:00.708957    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-624254-29893890-hxj28"
Nov 02 15:30:00.745165 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:00.745115    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-28f621-29893890-8pdnl"
Nov 02 15:30:00.807254 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:00.806982    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2e4ebc-29893890-qbrcz"
Nov 02 15:30:00.816887 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:00.812800    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7ecfe1-29893890-bxmzk"
Nov 02 15:30:00.867057 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:00.866070    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-501738-29893890-59r7h"]
Nov 02 15:30:01.053345 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:01.053247    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8a3fb0-29893890-t64lm"]
Nov 02 15:30:01.102725 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:01.101996    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-624254-29893890-hxj28"]
Nov 02 15:30:01.116027 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:01.115999    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-fda646-29893890-g8wfc"]
Nov 02 15:30:01.127092 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:01.127063    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-246137-29893890-h98qk"]
Nov 02 15:30:01.135659 ip-10-0-135-138 kubenswrapper[5304]: W1102 15:30:01.135618    5304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2b513e5_c608_45e2_8dc6_d798bf902d83.slice/crio-8fb9bc944da30c0f4846e86c7094c0836b71eae476d78e0255cb33b584462be3 WatchSource:0}: Error finding container 8fb9bc944da30c0f4846e86c7094c0836b71eae476d78e0255cb33b584462be3: Status 404 returned error can't find the container with id 8fb9bc944da30c0f4846e86c7094c0836b71eae476d78e0255cb33b584462be3
Nov 02 15:30:01.178581 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:01.178557    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2e4ebc-29893890-qbrcz"]
Nov 02 15:30:01.188503 ip-10-0-135-138 kubenswrapper[5304]: W1102 15:30:01.188471    5304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5341d918_4b2b_4c8d_8898_0e764f23c230.slice/crio-2366ffe03ece6e22a102c5b8d89c98c486a207a41aa0ffe107084003a0ae966f WatchSource:0}: Error finding container 2366ffe03ece6e22a102c5b8d89c98c486a207a41aa0ffe107084003a0ae966f: Status 404 returned error can't find the container with id 2366ffe03ece6e22a102c5b8d89c98c486a207a41aa0ffe107084003a0ae966f
Nov 02 15:30:01.202699 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:01.202670    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7ecfe1-29893890-bxmzk"]
Nov 02 15:30:01.249733 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:01.249666    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5dd8e9-29893890-wgcbb"]
Nov 02 15:30:01.250382 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:01.249771    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-6af50f-29893890-95trm"]
Nov 02 15:30:01.339987 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:01.339965    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-28f621-29893890-8pdnl"]
Nov 02 15:30:01.443452 ip-10-0-135-138 kubenswrapper[5304]: W1102 15:30:01.443418    5304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0293ead_8a17_4860_8691_12a4bf2d0ed6.slice/crio-185b87d12bc386c042873afd2ecc4d9ca28b20374ee982b5974e4a6c248e7dd7 WatchSource:0}: Error finding container 185b87d12bc386c042873afd2ecc4d9ca28b20374ee982b5974e4a6c248e7dd7: Status 404 returned error can't find the container with id 185b87d12bc386c042873afd2ecc4d9ca28b20374ee982b5974e4a6c248e7dd7
Nov 02 15:30:01.443682 ip-10-0-135-138 kubenswrapper[5304]: W1102 15:30:01.443652    5304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfecd4670_5e5f_477b_9f3c_689d6bdb79e4.slice/crio-bb10da287d596077ee1dd72cabc77d2db26d476321adab94f7ba7ea585c98dfa WatchSource:0}: Error finding container bb10da287d596077ee1dd72cabc77d2db26d476321adab94f7ba7ea585c98dfa: Status 404 returned error can't find the container with id bb10da287d596077ee1dd72cabc77d2db26d476321adab94f7ba7ea585c98dfa
Nov 02 15:30:01.443909 ip-10-0-135-138 kubenswrapper[5304]: W1102 15:30:01.443891    5304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6984b63_9bb9_4081_a3ba_51be6260e1ea.slice/crio-a4ec0704d04ec3dd45fdb77fd9e9a32ba9eebd18ef54d41f5bb23d205c8d9430 WatchSource:0}: Error finding container a4ec0704d04ec3dd45fdb77fd9e9a32ba9eebd18ef54d41f5bb23d205c8d9430: Status 404 returned error can't find the container with id a4ec0704d04ec3dd45fdb77fd9e9a32ba9eebd18ef54d41f5bb23d205c8d9430
Nov 02 15:30:01.843017 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:01.842983    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5dd8e9-29893890-wgcbb" event={"ID":"fecd4670-5e5f-477b-9f3c-689d6bdb79e4","Type":"ContainerStarted","Data":"bb10da287d596077ee1dd72cabc77d2db26d476321adab94f7ba7ea585c98dfa"}
Nov 02 15:30:01.844885 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:01.844850    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-501738-29893890-59r7h" event={"ID":"8b61992b-ffa1-4606-bf49-9efccefbc460","Type":"ContainerStarted","Data":"98a27277bb73a8af3bf152fe2a5039e17d18526aa082669a2faaf5f5b7699f66"}
Nov 02 15:30:01.845006 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:01.844917    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-501738-29893890-59r7h" event={"ID":"8b61992b-ffa1-4606-bf49-9efccefbc460","Type":"ContainerStarted","Data":"006c11978ccdfa84819ff2e80c18b280b1fb46d65253d64c15ffed1c787f8c83"}
Nov 02 15:30:01.846350 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:01.846327    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-28f621-29893890-8pdnl" event={"ID":"d6984b63-9bb9-4081-a3ba-51be6260e1ea","Type":"ContainerStarted","Data":"a4ec0704d04ec3dd45fdb77fd9e9a32ba9eebd18ef54d41f5bb23d205c8d9430"}
Nov 02 15:30:01.848052 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:01.848029    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-fda646-29893890-g8wfc" event={"ID":"3b856b33-22ae-40c4-b3a5-8d90c81a16de","Type":"ContainerStarted","Data":"448a8ccdb72ed1663b419d86ef3af9275955131e9381e253760e2d46b9a62d7d"}
Nov 02 15:30:01.848186 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:01.848086    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-fda646-29893890-g8wfc" event={"ID":"3b856b33-22ae-40c4-b3a5-8d90c81a16de","Type":"ContainerStarted","Data":"07c7e43068fc2afde5be03ed191e031a86ee1fa35f780ba25011b3b4081e53a4"}
Nov 02 15:30:01.849826 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:01.849804    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2e4ebc-29893890-qbrcz" event={"ID":"5341d918-4b2b-4c8d-8898-0e764f23c230","Type":"ContainerStarted","Data":"2366ffe03ece6e22a102c5b8d89c98c486a207a41aa0ffe107084003a0ae966f"}
Nov 02 15:30:01.851405 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:01.851379    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-246137-29893890-h98qk" event={"ID":"8f70c555-72cb-453b-8d47-8bdbe39222e7","Type":"ContainerStarted","Data":"de9c81f6f5e76fcd30e8744ae802065f924b6bf850d3cf52a7feb1b11cb6bfc2"}
Nov 02 15:30:01.851524 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:01.851426    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-246137-29893890-h98qk" event={"ID":"8f70c555-72cb-453b-8d47-8bdbe39222e7","Type":"ContainerStarted","Data":"85f6aa8e7f0fb8e59f02e0f080248ce85acc00298fabb465812b291936082c2e"}
Nov 02 15:30:01.852722 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:01.852699    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-6af50f-29893890-95trm" event={"ID":"c0293ead-8a17-4860-8691-12a4bf2d0ed6","Type":"ContainerStarted","Data":"185b87d12bc386c042873afd2ecc4d9ca28b20374ee982b5974e4a6c248e7dd7"}
Nov 02 15:30:01.853897 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:01.853874    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7ecfe1-29893890-bxmzk" event={"ID":"33921120-0f00-4584-9e1d-d486c475bd10","Type":"ContainerStarted","Data":"7aeb2f44969147ab10a33d9d2e2b64409efab2bd8ca131c8cd065ebb540b2760"}
Nov 02 15:30:01.855375 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:01.855353    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8a3fb0-29893890-t64lm" event={"ID":"875ff34a-5486-4647-bbc6-bbd0df2633e6","Type":"ContainerStarted","Data":"c082366e5bbe09ce98979dae1fc675349e448a224f2d9b0693a8a5573b11666b"}
Nov 02 15:30:01.855457 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:01.855384    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8a3fb0-29893890-t64lm" event={"ID":"875ff34a-5486-4647-bbc6-bbd0df2633e6","Type":"ContainerStarted","Data":"7ba67597d21930d04d8ccb85da2a31bffe867cf305a0f4db977461c0e7746202"}
Nov 02 15:30:01.856660 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:01.856640    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-624254-29893890-hxj28" event={"ID":"b2b513e5-c608-45e2-8dc6-d798bf902d83","Type":"ContainerStarted","Data":"8fb9bc944da30c0f4846e86c7094c0836b71eae476d78e0255cb33b584462be3"}
Nov 02 15:30:01.887654 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:01.887605    5304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-501738-29893890-59r7h" podStartSLOduration=1.8875901000000002 podStartE2EDuration="1.8875901s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:30:01.884823027 +0000 UTC m=+2001.878647437" watchObservedRunningTime="2026-11-02 15:30:01.8875901 +0000 UTC m=+2001.881414508"
Nov 02 15:30:01.937590 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:01.937490    5304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-624254-29893890-hxj28" podStartSLOduration=1.937471485 podStartE2EDuration="1.937471485s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:30:01.930119471 +0000 UTC m=+2001.923943881" watchObservedRunningTime="2026-11-02 15:30:01.937471485 +0000 UTC m=+2001.931295896"
Nov 02 15:30:01.972596 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:01.972538    5304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8a3fb0-29893890-t64lm" podStartSLOduration=1.97252424 podStartE2EDuration="1.97252424s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:30:01.972324366 +0000 UTC m=+2001.966148776" watchObservedRunningTime="2026-11-02 15:30:01.97252424 +0000 UTC m=+2001.966348646"
Nov 02 15:30:02.024804 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:02.024746    5304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-fda646-29893890-g8wfc" podStartSLOduration=2.024731079 podStartE2EDuration="2.024731079s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:30:02.022759726 +0000 UTC m=+2002.016584128" watchObservedRunningTime="2026-11-02 15:30:02.024731079 +0000 UTC m=+2002.018555487"
Nov 02 15:30:02.067548 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:02.067499    5304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-246137-29893890-h98qk" podStartSLOduration=2.067475311 podStartE2EDuration="2.067475311s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:30:02.064669462 +0000 UTC m=+2002.058493871" watchObservedRunningTime="2026-11-02 15:30:02.067475311 +0000 UTC m=+2002.061299719"
Nov 02 15:30:02.866078 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:02.866047    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5dd8e9-29893890-wgcbb" event={"ID":"fecd4670-5e5f-477b-9f3c-689d6bdb79e4","Type":"ContainerStarted","Data":"ebaf16056a962b2326c126b8daae6b6edac00801c9e24cba1c116b308b9adaf9"}
Nov 02 15:30:02.867692 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:02.867667    5304 generic.go:356] "Generic (PLEG): container finished" podID="8b61992b-ffa1-4606-bf49-9efccefbc460" containerID="98a27277bb73a8af3bf152fe2a5039e17d18526aa082669a2faaf5f5b7699f66" exitCode=60
Nov 02 15:30:02.867806 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:02.867722    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-501738-29893890-59r7h" event={"ID":"8b61992b-ffa1-4606-bf49-9efccefbc460","Type":"ContainerDied","Data":"98a27277bb73a8af3bf152fe2a5039e17d18526aa082669a2faaf5f5b7699f66"}
Nov 02 15:30:02.867992 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:02.867977    5304 scope.go:122] "RemoveContainer" containerID="98a27277bb73a8af3bf152fe2a5039e17d18526aa082669a2faaf5f5b7699f66"
Nov 02 15:30:02.869800 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:02.869773    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-28f621-29893890-8pdnl" event={"ID":"d6984b63-9bb9-4081-a3ba-51be6260e1ea","Type":"ContainerStarted","Data":"30e9c7b2b5ab21c077a3c52fc250dda2d40046160b843d601d15aa43da8e51fc"}
Nov 02 15:30:02.898712 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:02.898666    5304 generic.go:356] "Generic (PLEG): container finished" podID="3b856b33-22ae-40c4-b3a5-8d90c81a16de" containerID="448a8ccdb72ed1663b419d86ef3af9275955131e9381e253760e2d46b9a62d7d" exitCode=60
Nov 02 15:30:02.898876 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:02.898798    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-fda646-29893890-g8wfc" event={"ID":"3b856b33-22ae-40c4-b3a5-8d90c81a16de","Type":"ContainerDied","Data":"448a8ccdb72ed1663b419d86ef3af9275955131e9381e253760e2d46b9a62d7d"}
Nov 02 15:30:02.899393 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:02.899367    5304 scope.go:122] "RemoveContainer" containerID="448a8ccdb72ed1663b419d86ef3af9275955131e9381e253760e2d46b9a62d7d"
Nov 02 15:30:02.903602 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:02.903577    5304 generic.go:356] "Generic (PLEG): container finished" podID="5341d918-4b2b-4c8d-8898-0e764f23c230" containerID="1d58cdd21b03cc3029e22f77eec012b4fe5cd0e01885d890731453f39efacb0e" exitCode=60
Nov 02 15:30:02.903708 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:02.903692    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2e4ebc-29893890-qbrcz" event={"ID":"5341d918-4b2b-4c8d-8898-0e764f23c230","Type":"ContainerDied","Data":"1d58cdd21b03cc3029e22f77eec012b4fe5cd0e01885d890731453f39efacb0e"}
Nov 02 15:30:02.904121 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:02.904046    5304 scope.go:122] "RemoveContainer" containerID="1d58cdd21b03cc3029e22f77eec012b4fe5cd0e01885d890731453f39efacb0e"
Nov 02 15:30:02.905777 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:02.905753    5304 generic.go:356] "Generic (PLEG): container finished" podID="8f70c555-72cb-453b-8d47-8bdbe39222e7" containerID="de9c81f6f5e76fcd30e8744ae802065f924b6bf850d3cf52a7feb1b11cb6bfc2" exitCode=60
Nov 02 15:30:02.905894 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:02.905838    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-246137-29893890-h98qk" event={"ID":"8f70c555-72cb-453b-8d47-8bdbe39222e7","Type":"ContainerDied","Data":"de9c81f6f5e76fcd30e8744ae802065f924b6bf850d3cf52a7feb1b11cb6bfc2"}
Nov 02 15:30:02.906100 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:02.906068    5304 scope.go:122] "RemoveContainer" containerID="de9c81f6f5e76fcd30e8744ae802065f924b6bf850d3cf52a7feb1b11cb6bfc2"
Nov 02 15:30:02.907917 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:02.907894    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-6af50f-29893890-95trm" event={"ID":"c0293ead-8a17-4860-8691-12a4bf2d0ed6","Type":"ContainerStarted","Data":"3da256661562d28ebd34660355e0dee64fa3ec3455d793b3e0cfbfa7d7aef4f7"}
Nov 02 15:30:02.909798 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:02.909772    5304 generic.go:356] "Generic (PLEG): container finished" podID="33921120-0f00-4584-9e1d-d486c475bd10" containerID="11a1b1c57a6476bb74452901687ad78d493089e5a8e607564d885e7e6c138d1a" exitCode=60
Nov 02 15:30:02.909949 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:02.909921    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7ecfe1-29893890-bxmzk" event={"ID":"33921120-0f00-4584-9e1d-d486c475bd10","Type":"ContainerDied","Data":"11a1b1c57a6476bb74452901687ad78d493089e5a8e607564d885e7e6c138d1a"}
Nov 02 15:30:02.910083 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:02.910066    5304 scope.go:122] "RemoveContainer" containerID="11a1b1c57a6476bb74452901687ad78d493089e5a8e607564d885e7e6c138d1a"
Nov 02 15:30:02.911849 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:02.911825    5304 generic.go:356] "Generic (PLEG): container finished" podID="875ff34a-5486-4647-bbc6-bbd0df2633e6" containerID="c082366e5bbe09ce98979dae1fc675349e448a224f2d9b0693a8a5573b11666b" exitCode=60
Nov 02 15:30:02.911944 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:02.911908    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8a3fb0-29893890-t64lm" event={"ID":"875ff34a-5486-4647-bbc6-bbd0df2633e6","Type":"ContainerDied","Data":"c082366e5bbe09ce98979dae1fc675349e448a224f2d9b0693a8a5573b11666b"}
Nov 02 15:30:02.912121 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:02.912102    5304 scope.go:122] "RemoveContainer" containerID="c082366e5bbe09ce98979dae1fc675349e448a224f2d9b0693a8a5573b11666b"
Nov 02 15:30:02.918523 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:02.918496    5304 generic.go:356] "Generic (PLEG): container finished" podID="b2b513e5-c608-45e2-8dc6-d798bf902d83" containerID="de588a78dbd37f0fa911984c00047968a327ac88e5b14df06f0d93e8a620e5ec" exitCode=60
Nov 02 15:30:02.918625 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:02.918604    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-624254-29893890-hxj28" event={"ID":"b2b513e5-c608-45e2-8dc6-d798bf902d83","Type":"ContainerDied","Data":"de588a78dbd37f0fa911984c00047968a327ac88e5b14df06f0d93e8a620e5ec"}
Nov 02 15:30:02.918850 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:02.918834    5304 scope.go:122] "RemoveContainer" containerID="de588a78dbd37f0fa911984c00047968a327ac88e5b14df06f0d93e8a620e5ec"
Nov 02 15:30:02.982078 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:02.982030    5304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5dd8e9-29893890-wgcbb" podStartSLOduration=2.9820166930000003 podStartE2EDuration="2.982016693s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:30:02.934863856 +0000 UTC m=+2002.928688267" watchObservedRunningTime="2026-11-02 15:30:02.982016693 +0000 UTC m=+2002.975841102"
Nov 02 15:30:03.168101 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:03.168048    5304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-28f621-29893890-8pdnl" podStartSLOduration=3.168031737 podStartE2EDuration="3.168031737s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:30:03.165583602 +0000 UTC m=+2003.159408015" watchObservedRunningTime="2026-11-02 15:30:03.168031737 +0000 UTC m=+2003.161856165"
Nov 02 15:30:03.168862 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:03.168829    5304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-6af50f-29893890-95trm" podStartSLOduration=3.168821854 podStartE2EDuration="3.168821854s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:30:03.1176807 +0000 UTC m=+2003.111505138" watchObservedRunningTime="2026-11-02 15:30:03.168821854 +0000 UTC m=+2003.162646309"
Nov 02 15:30:03.927173 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:03.927105    5304 generic.go:356] "Generic (PLEG): container finished" podID="c0293ead-8a17-4860-8691-12a4bf2d0ed6" containerID="3da256661562d28ebd34660355e0dee64fa3ec3455d793b3e0cfbfa7d7aef4f7" exitCode=60
Nov 02 15:30:03.927625 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:03.927193    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-6af50f-29893890-95trm" event={"ID":"c0293ead-8a17-4860-8691-12a4bf2d0ed6","Type":"ContainerDied","Data":"3da256661562d28ebd34660355e0dee64fa3ec3455d793b3e0cfbfa7d7aef4f7"}
Nov 02 15:30:03.927625 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:03.927481    5304 scope.go:122] "RemoveContainer" containerID="3da256661562d28ebd34660355e0dee64fa3ec3455d793b3e0cfbfa7d7aef4f7"
Nov 02 15:30:03.929341 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:03.929263    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7ecfe1-29893890-bxmzk" event={"ID":"33921120-0f00-4584-9e1d-d486c475bd10","Type":"ContainerStarted","Data":"c530719fe0e701c1acb9514d2225cd11a3d88f7b39a266b01d3fc5770727a6ef"}
Nov 02 15:30:03.935038 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:03.935006    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8a3fb0-29893890-t64lm" event={"ID":"875ff34a-5486-4647-bbc6-bbd0df2633e6","Type":"ContainerStarted","Data":"7c88b8e712361eea0ce0dc4f9481a80825db2334206ca483f2e3f041aadc8067"}
Nov 02 15:30:03.938011 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:03.937986    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-624254-29893890-hxj28" event={"ID":"b2b513e5-c608-45e2-8dc6-d798bf902d83","Type":"ContainerStarted","Data":"e9648fe3c8c705c9ddc66c2ed17eaea97a6a2cae2772483efecb49cf053da18a"}
Nov 02 15:30:03.939491 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:03.939468    5304 generic.go:356] "Generic (PLEG): container finished" podID="fecd4670-5e5f-477b-9f3c-689d6bdb79e4" containerID="ebaf16056a962b2326c126b8daae6b6edac00801c9e24cba1c116b308b9adaf9" exitCode=60
Nov 02 15:30:03.939600 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:03.939538    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5dd8e9-29893890-wgcbb" event={"ID":"fecd4670-5e5f-477b-9f3c-689d6bdb79e4","Type":"ContainerDied","Data":"ebaf16056a962b2326c126b8daae6b6edac00801c9e24cba1c116b308b9adaf9"}
Nov 02 15:30:03.939846 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:03.939823    5304 scope.go:122] "RemoveContainer" containerID="ebaf16056a962b2326c126b8daae6b6edac00801c9e24cba1c116b308b9adaf9"
Nov 02 15:30:03.941679 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:03.941648    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-501738-29893890-59r7h" event={"ID":"8b61992b-ffa1-4606-bf49-9efccefbc460","Type":"ContainerStarted","Data":"46a9bb80820904b38a92ef599ccba74b30d65eefbc4fd30075cf657d313dcc5a"}
Nov 02 15:30:03.943415 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:03.943394    5304 generic.go:356] "Generic (PLEG): container finished" podID="d6984b63-9bb9-4081-a3ba-51be6260e1ea" containerID="30e9c7b2b5ab21c077a3c52fc250dda2d40046160b843d601d15aa43da8e51fc" exitCode=60
Nov 02 15:30:03.943498 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:03.943476    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-28f621-29893890-8pdnl" event={"ID":"d6984b63-9bb9-4081-a3ba-51be6260e1ea","Type":"ContainerDied","Data":"30e9c7b2b5ab21c077a3c52fc250dda2d40046160b843d601d15aa43da8e51fc"}
Nov 02 15:30:03.943728 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:03.943704    5304 scope.go:122] "RemoveContainer" containerID="30e9c7b2b5ab21c077a3c52fc250dda2d40046160b843d601d15aa43da8e51fc"
Nov 02 15:30:03.945492 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:03.945473    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-fda646-29893890-g8wfc" event={"ID":"3b856b33-22ae-40c4-b3a5-8d90c81a16de","Type":"ContainerStarted","Data":"6d0b3b9408da320e1ca6727c1fc6eca9ae0b5b62ab16f1b4bc3fa0877b4f849e"}
Nov 02 15:30:03.947503 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:03.947481    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2e4ebc-29893890-qbrcz" event={"ID":"5341d918-4b2b-4c8d-8898-0e764f23c230","Type":"ContainerStarted","Data":"fc3d88a1e25b5bbfd483a2b6a9bb6885ba8cca5c46c3a5fc09b741c09d06d2ec"}
Nov 02 15:30:03.949362 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:03.949342    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-246137-29893890-h98qk" event={"ID":"8f70c555-72cb-453b-8d47-8bdbe39222e7","Type":"ContainerStarted","Data":"dff8ba2f4e5d6af9da8a3c2e8861d5c5d546189ee190717805ba5d5ff478e381"}
Nov 02 15:30:04.099707 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:04.099637    5304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2e4ebc-29893890-qbrcz" podStartSLOduration=4.099617101 podStartE2EDuration="4.099617101s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:30:04.093189725 +0000 UTC m=+2004.087014135" watchObservedRunningTime="2026-11-02 15:30:04.099617101 +0000 UTC m=+2004.093441512"
Nov 02 15:30:04.221283 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:04.221221    5304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7ecfe1-29893890-bxmzk" podStartSLOduration=4.221202847 podStartE2EDuration="4.221202847s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:30:04.218844159 +0000 UTC m=+2004.212668581" watchObservedRunningTime="2026-11-02 15:30:04.221202847 +0000 UTC m=+2004.215027258"
Nov 02 15:30:04.958619 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:04.958590    5304 generic.go:356] "Generic (PLEG): container finished" podID="8f70c555-72cb-453b-8d47-8bdbe39222e7" containerID="dff8ba2f4e5d6af9da8a3c2e8861d5c5d546189ee190717805ba5d5ff478e381" exitCode=60
Nov 02 15:30:04.959007 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:04.958666    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-246137-29893890-h98qk" event={"ID":"8f70c555-72cb-453b-8d47-8bdbe39222e7","Type":"ContainerDied","Data":"dff8ba2f4e5d6af9da8a3c2e8861d5c5d546189ee190717805ba5d5ff478e381"}
Nov 02 15:30:04.959007 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:04.958742    5304 scope.go:122] "RemoveContainer" containerID="de9c81f6f5e76fcd30e8744ae802065f924b6bf850d3cf52a7feb1b11cb6bfc2"
Nov 02 15:30:04.959007 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:04.958962    5304 scope.go:122] "RemoveContainer" containerID="dff8ba2f4e5d6af9da8a3c2e8861d5c5d546189ee190717805ba5d5ff478e381"
Nov 02 15:30:04.959406 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:30:04.959380    5304 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w5-246137-29893890-h98qk_odh-ai-gateway-infra(8f70c555-72cb-453b-8d47-8bdbe39222e7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-246137-29893890-h98qk" podUID="8f70c555-72cb-453b-8d47-8bdbe39222e7"
Nov 02 15:30:04.960858 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:04.960837    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-6af50f-29893890-95trm" event={"ID":"c0293ead-8a17-4860-8691-12a4bf2d0ed6","Type":"ContainerStarted","Data":"dda9efc71ff6cece3b7ac3c956fc135e3c675c91b204b64dacc3e83e8bb559ba"}
Nov 02 15:30:04.962762 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:04.962740    5304 generic.go:356] "Generic (PLEG): container finished" podID="33921120-0f00-4584-9e1d-d486c475bd10" containerID="c530719fe0e701c1acb9514d2225cd11a3d88f7b39a266b01d3fc5770727a6ef" exitCode=60
Nov 02 15:30:04.962866 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:04.962832    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7ecfe1-29893890-bxmzk" event={"ID":"33921120-0f00-4584-9e1d-d486c475bd10","Type":"ContainerDied","Data":"c530719fe0e701c1acb9514d2225cd11a3d88f7b39a266b01d3fc5770727a6ef"}
Nov 02 15:30:04.963077 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:04.963063    5304 scope.go:122] "RemoveContainer" containerID="c530719fe0e701c1acb9514d2225cd11a3d88f7b39a266b01d3fc5770727a6ef"
Nov 02 15:30:04.963393 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:30:04.963373    5304 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-7ecfe1-29893890-bxmzk_odh-ai-gateway-infra(33921120-0f00-4584-9e1d-d486c475bd10)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7ecfe1-29893890-bxmzk" podUID="33921120-0f00-4584-9e1d-d486c475bd10"
Nov 02 15:30:04.964693 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:04.964669    5304 generic.go:356] "Generic (PLEG): container finished" podID="875ff34a-5486-4647-bbc6-bbd0df2633e6" containerID="7c88b8e712361eea0ce0dc4f9481a80825db2334206ca483f2e3f041aadc8067" exitCode=60
Nov 02 15:30:04.964862 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:04.964834    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8a3fb0-29893890-t64lm" event={"ID":"875ff34a-5486-4647-bbc6-bbd0df2633e6","Type":"ContainerDied","Data":"7c88b8e712361eea0ce0dc4f9481a80825db2334206ca483f2e3f041aadc8067"}
Nov 02 15:30:04.965075 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:04.965061    5304 scope.go:122] "RemoveContainer" containerID="7c88b8e712361eea0ce0dc4f9481a80825db2334206ca483f2e3f041aadc8067"
Nov 02 15:30:04.965390 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:30:04.965367    5304 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-8a3fb0-29893890-t64lm_odh-ai-gateway-infra(875ff34a-5486-4647-bbc6-bbd0df2633e6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8a3fb0-29893890-t64lm" podUID="875ff34a-5486-4647-bbc6-bbd0df2633e6"
Nov 02 15:30:04.966742 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:04.966722    5304 generic.go:356] "Generic (PLEG): container finished" podID="b2b513e5-c608-45e2-8dc6-d798bf902d83" containerID="e9648fe3c8c705c9ddc66c2ed17eaea97a6a2cae2772483efecb49cf053da18a" exitCode=60
Nov 02 15:30:04.966827 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:04.966764    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-624254-29893890-hxj28" event={"ID":"b2b513e5-c608-45e2-8dc6-d798bf902d83","Type":"ContainerDied","Data":"e9648fe3c8c705c9ddc66c2ed17eaea97a6a2cae2772483efecb49cf053da18a"}
Nov 02 15:30:04.967024 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:04.967006    5304 scope.go:122] "RemoveContainer" containerID="e9648fe3c8c705c9ddc66c2ed17eaea97a6a2cae2772483efecb49cf053da18a"
Nov 02 15:30:04.967292 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:30:04.967270    5304 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-624254-29893890-hxj28_odh-ai-gateway-infra(b2b513e5-c608-45e2-8dc6-d798bf902d83)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-624254-29893890-hxj28" podUID="b2b513e5-c608-45e2-8dc6-d798bf902d83"
Nov 02 15:30:04.968299 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:04.968281    5304 generic.go:356] "Generic (PLEG): container finished" podID="fecd4670-5e5f-477b-9f3c-689d6bdb79e4" containerID="4b3d77921ae79c47c7f8e9cf12563cfa531684342f1ae90753a974682205ac89" exitCode=60
Nov 02 15:30:04.968381 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:04.968337    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5dd8e9-29893890-wgcbb" event={"ID":"fecd4670-5e5f-477b-9f3c-689d6bdb79e4","Type":"ContainerDied","Data":"4b3d77921ae79c47c7f8e9cf12563cfa531684342f1ae90753a974682205ac89"}
Nov 02 15:30:04.968578 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:04.968561    5304 scope.go:122] "RemoveContainer" containerID="4b3d77921ae79c47c7f8e9cf12563cfa531684342f1ae90753a974682205ac89"
Nov 02 15:30:04.968809 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:30:04.968790    5304 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-5dd8e9-29893890-wgcbb_odh-ai-gateway-infra(fecd4670-5e5f-477b-9f3c-689d6bdb79e4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5dd8e9-29893890-wgcbb" podUID="fecd4670-5e5f-477b-9f3c-689d6bdb79e4"
Nov 02 15:30:04.970248 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:04.970226    5304 generic.go:356] "Generic (PLEG): container finished" podID="8b61992b-ffa1-4606-bf49-9efccefbc460" containerID="46a9bb80820904b38a92ef599ccba74b30d65eefbc4fd30075cf657d313dcc5a" exitCode=60
Nov 02 15:30:04.970402 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:04.970252    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-501738-29893890-59r7h" event={"ID":"8b61992b-ffa1-4606-bf49-9efccefbc460","Type":"ContainerDied","Data":"46a9bb80820904b38a92ef599ccba74b30d65eefbc4fd30075cf657d313dcc5a"}
Nov 02 15:30:04.970535 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:04.970518    5304 scope.go:122] "RemoveContainer" containerID="46a9bb80820904b38a92ef599ccba74b30d65eefbc4fd30075cf657d313dcc5a"
Nov 02 15:30:04.970773 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:30:04.970755    5304 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-501738-29893890-59r7h_odh-ai-gateway-infra(8b61992b-ffa1-4606-bf49-9efccefbc460)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-501738-29893890-59r7h" podUID="8b61992b-ffa1-4606-bf49-9efccefbc460"
Nov 02 15:30:04.972373 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:04.972338    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-28f621-29893890-8pdnl" event={"ID":"d6984b63-9bb9-4081-a3ba-51be6260e1ea","Type":"ContainerStarted","Data":"a42f17989f46323b71e7cd3bf8ffb8a598132644fa7984087c3cdb1155468209"}
Nov 02 15:30:04.972919 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:04.972902    5304 scope.go:122] "RemoveContainer" containerID="11a1b1c57a6476bb74452901687ad78d493089e5a8e607564d885e7e6c138d1a"
Nov 02 15:30:04.974446 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:04.974425    5304 generic.go:356] "Generic (PLEG): container finished" podID="3b856b33-22ae-40c4-b3a5-8d90c81a16de" containerID="6d0b3b9408da320e1ca6727c1fc6eca9ae0b5b62ab16f1b4bc3fa0877b4f849e" exitCode=60
Nov 02 15:30:04.974542 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:04.974443    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-fda646-29893890-g8wfc" event={"ID":"3b856b33-22ae-40c4-b3a5-8d90c81a16de","Type":"ContainerDied","Data":"6d0b3b9408da320e1ca6727c1fc6eca9ae0b5b62ab16f1b4bc3fa0877b4f849e"}
Nov 02 15:30:04.974692 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:04.974670    5304 scope.go:122] "RemoveContainer" containerID="6d0b3b9408da320e1ca6727c1fc6eca9ae0b5b62ab16f1b4bc3fa0877b4f849e"
Nov 02 15:30:04.974943 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:30:04.974922    5304 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-fda646-29893890-g8wfc_odh-ai-gateway-infra(3b856b33-22ae-40c4-b3a5-8d90c81a16de)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-fda646-29893890-g8wfc" podUID="3b856b33-22ae-40c4-b3a5-8d90c81a16de"
Nov 02 15:30:04.976377 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:04.976359    5304 generic.go:356] "Generic (PLEG): container finished" podID="5341d918-4b2b-4c8d-8898-0e764f23c230" containerID="fc3d88a1e25b5bbfd483a2b6a9bb6885ba8cca5c46c3a5fc09b741c09d06d2ec" exitCode=60
Nov 02 15:30:04.976449 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:04.976401    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2e4ebc-29893890-qbrcz" event={"ID":"5341d918-4b2b-4c8d-8898-0e764f23c230","Type":"ContainerDied","Data":"fc3d88a1e25b5bbfd483a2b6a9bb6885ba8cca5c46c3a5fc09b741c09d06d2ec"}
Nov 02 15:30:04.976624 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:04.976610    5304 scope.go:122] "RemoveContainer" containerID="fc3d88a1e25b5bbfd483a2b6a9bb6885ba8cca5c46c3a5fc09b741c09d06d2ec"
Nov 02 15:30:04.976831 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:30:04.976808    5304 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-2e4ebc-29893890-qbrcz_odh-ai-gateway-infra(5341d918-4b2b-4c8d-8898-0e764f23c230)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2e4ebc-29893890-qbrcz" podUID="5341d918-4b2b-4c8d-8898-0e764f23c230"
Nov 02 15:30:04.987463 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:04.987441    5304 scope.go:122] "RemoveContainer" containerID="c082366e5bbe09ce98979dae1fc675349e448a224f2d9b0693a8a5573b11666b"
Nov 02 15:30:05.001081 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:05.001059    5304 scope.go:122] "RemoveContainer" containerID="de588a78dbd37f0fa911984c00047968a327ac88e5b14df06f0d93e8a620e5ec"
Nov 02 15:30:05.015536 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:05.015516    5304 scope.go:122] "RemoveContainer" containerID="ebaf16056a962b2326c126b8daae6b6edac00801c9e24cba1c116b308b9adaf9"
Nov 02 15:30:05.028926 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:05.028893    5304 scope.go:122] "RemoveContainer" containerID="98a27277bb73a8af3bf152fe2a5039e17d18526aa082669a2faaf5f5b7699f66"
Nov 02 15:30:05.078562 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:05.078532    5304 scope.go:122] "RemoveContainer" containerID="448a8ccdb72ed1663b419d86ef3af9275955131e9381e253760e2d46b9a62d7d"
Nov 02 15:30:05.093245 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:05.093221    5304 scope.go:122] "RemoveContainer" containerID="1d58cdd21b03cc3029e22f77eec012b4fe5cd0e01885d890731453f39efacb0e"
Nov 02 15:30:05.989940 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:05.989913    5304 scope.go:122] "RemoveContainer" containerID="fc3d88a1e25b5bbfd483a2b6a9bb6885ba8cca5c46c3a5fc09b741c09d06d2ec"
Nov 02 15:30:05.990462 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:30:05.990331    5304 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-2e4ebc-29893890-qbrcz_odh-ai-gateway-infra(5341d918-4b2b-4c8d-8898-0e764f23c230)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2e4ebc-29893890-qbrcz" podUID="5341d918-4b2b-4c8d-8898-0e764f23c230"
Nov 02 15:30:05.991657 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:05.991634    5304 scope.go:122] "RemoveContainer" containerID="dff8ba2f4e5d6af9da8a3c2e8861d5c5d546189ee190717805ba5d5ff478e381"
Nov 02 15:30:05.992037 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:30:05.992013    5304 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w5-246137-29893890-h98qk_odh-ai-gateway-infra(8f70c555-72cb-453b-8d47-8bdbe39222e7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-246137-29893890-h98qk" podUID="8f70c555-72cb-453b-8d47-8bdbe39222e7"
Nov 02 15:30:05.993363 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:05.993340    5304 generic.go:356] "Generic (PLEG): container finished" podID="c0293ead-8a17-4860-8691-12a4bf2d0ed6" containerID="dda9efc71ff6cece3b7ac3c956fc135e3c675c91b204b64dacc3e83e8bb559ba" exitCode=60
Nov 02 15:30:05.993474 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:05.993407    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-6af50f-29893890-95trm" event={"ID":"c0293ead-8a17-4860-8691-12a4bf2d0ed6","Type":"ContainerDied","Data":"dda9efc71ff6cece3b7ac3c956fc135e3c675c91b204b64dacc3e83e8bb559ba"}
Nov 02 15:30:05.993474 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:05.993468    5304 scope.go:122] "RemoveContainer" containerID="3da256661562d28ebd34660355e0dee64fa3ec3455d793b3e0cfbfa7d7aef4f7"
Nov 02 15:30:05.993670 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:05.993653    5304 scope.go:122] "RemoveContainer" containerID="dda9efc71ff6cece3b7ac3c956fc135e3c675c91b204b64dacc3e83e8bb559ba"
Nov 02 15:30:05.993931 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:30:05.993908    5304 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-6af50f-29893890-95trm_odh-ai-gateway-infra(c0293ead-8a17-4860-8691-12a4bf2d0ed6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-6af50f-29893890-95trm" podUID="c0293ead-8a17-4860-8691-12a4bf2d0ed6"
Nov 02 15:30:05.995281 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:05.995265    5304 scope.go:122] "RemoveContainer" containerID="c530719fe0e701c1acb9514d2225cd11a3d88f7b39a266b01d3fc5770727a6ef"
Nov 02 15:30:05.995560 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:30:05.995534    5304 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-7ecfe1-29893890-bxmzk_odh-ai-gateway-infra(33921120-0f00-4584-9e1d-d486c475bd10)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7ecfe1-29893890-bxmzk" podUID="33921120-0f00-4584-9e1d-d486c475bd10"
Nov 02 15:30:05.997107 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:05.997090    5304 scope.go:122] "RemoveContainer" containerID="7c88b8e712361eea0ce0dc4f9481a80825db2334206ca483f2e3f041aadc8067"
Nov 02 15:30:05.997433 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:30:05.997399    5304 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-8a3fb0-29893890-t64lm_odh-ai-gateway-infra(875ff34a-5486-4647-bbc6-bbd0df2633e6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8a3fb0-29893890-t64lm" podUID="875ff34a-5486-4647-bbc6-bbd0df2633e6"
Nov 02 15:30:05.999032 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:05.999008    5304 scope.go:122] "RemoveContainer" containerID="e9648fe3c8c705c9ddc66c2ed17eaea97a6a2cae2772483efecb49cf053da18a"
Nov 02 15:30:05.999385 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:30:05.999361    5304 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-624254-29893890-hxj28_odh-ai-gateway-infra(b2b513e5-c608-45e2-8dc6-d798bf902d83)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-624254-29893890-hxj28" podUID="b2b513e5-c608-45e2-8dc6-d798bf902d83"
Nov 02 15:30:06.000537 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:06.000517    5304 scope.go:122] "RemoveContainer" containerID="4b3d77921ae79c47c7f8e9cf12563cfa531684342f1ae90753a974682205ac89"
Nov 02 15:30:06.000744 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:30:06.000714    5304 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-5dd8e9-29893890-wgcbb_odh-ai-gateway-infra(fecd4670-5e5f-477b-9f3c-689d6bdb79e4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5dd8e9-29893890-wgcbb" podUID="fecd4670-5e5f-477b-9f3c-689d6bdb79e4"
Nov 02 15:30:06.002252 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:06.002232    5304 scope.go:122] "RemoveContainer" containerID="46a9bb80820904b38a92ef599ccba74b30d65eefbc4fd30075cf657d313dcc5a"
Nov 02 15:30:06.002485 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:30:06.002457    5304 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-501738-29893890-59r7h_odh-ai-gateway-infra(8b61992b-ffa1-4606-bf49-9efccefbc460)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-501738-29893890-59r7h" podUID="8b61992b-ffa1-4606-bf49-9efccefbc460"
Nov 02 15:30:06.004039 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:06.004014    5304 generic.go:356] "Generic (PLEG): container finished" podID="d6984b63-9bb9-4081-a3ba-51be6260e1ea" containerID="a42f17989f46323b71e7cd3bf8ffb8a598132644fa7984087c3cdb1155468209" exitCode=60
Nov 02 15:30:06.004143 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:06.004097    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-28f621-29893890-8pdnl" event={"ID":"d6984b63-9bb9-4081-a3ba-51be6260e1ea","Type":"ContainerDied","Data":"a42f17989f46323b71e7cd3bf8ffb8a598132644fa7984087c3cdb1155468209"}
Nov 02 15:30:06.004320 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:06.004302    5304 scope.go:122] "RemoveContainer" containerID="a42f17989f46323b71e7cd3bf8ffb8a598132644fa7984087c3cdb1155468209"
Nov 02 15:30:06.004566 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:30:06.004544    5304 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-28f621-29893890-8pdnl_odh-ai-gateway-infra(d6984b63-9bb9-4081-a3ba-51be6260e1ea)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-28f621-29893890-8pdnl" podUID="d6984b63-9bb9-4081-a3ba-51be6260e1ea"
Nov 02 15:30:06.006005 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:06.005986    5304 scope.go:122] "RemoveContainer" containerID="6d0b3b9408da320e1ca6727c1fc6eca9ae0b5b62ab16f1b4bc3fa0877b4f849e"
Nov 02 15:30:06.006284 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:30:06.006263    5304 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-fda646-29893890-g8wfc_odh-ai-gateway-infra(3b856b33-22ae-40c4-b3a5-8d90c81a16de)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-fda646-29893890-g8wfc" podUID="3b856b33-22ae-40c4-b3a5-8d90c81a16de"
Nov 02 15:30:06.008253 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:06.008237    5304 scope.go:122] "RemoveContainer" containerID="30e9c7b2b5ab21c077a3c52fc250dda2d40046160b843d601d15aa43da8e51fc"
Nov 02 15:30:07.015180 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:07.015154    5304 scope.go:122] "RemoveContainer" containerID="a42f17989f46323b71e7cd3bf8ffb8a598132644fa7984087c3cdb1155468209"
Nov 02 15:30:07.015686 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:30:07.015541    5304 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-28f621-29893890-8pdnl_odh-ai-gateway-infra(d6984b63-9bb9-4081-a3ba-51be6260e1ea)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-28f621-29893890-8pdnl" podUID="d6984b63-9bb9-4081-a3ba-51be6260e1ea"
Nov 02 15:30:07.016810 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:07.016792    5304 scope.go:122] "RemoveContainer" containerID="dda9efc71ff6cece3b7ac3c956fc135e3c675c91b204b64dacc3e83e8bb559ba"
Nov 02 15:30:07.017199 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:30:07.017171    5304 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-6af50f-29893890-95trm_odh-ai-gateway-infra(c0293ead-8a17-4860-8691-12a4bf2d0ed6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-6af50f-29893890-95trm" podUID="c0293ead-8a17-4860-8691-12a4bf2d0ed6"
Nov 02 15:30:15.112173 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:15.112142    5304 scope.go:122] "RemoveContainer" containerID="6d0b3b9408da320e1ca6727c1fc6eca9ae0b5b62ab16f1b4bc3fa0877b4f849e"
Nov 02 15:30:16.087387 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:16.087354    5304 generic.go:356] "Generic (PLEG): container finished" podID="3b856b33-22ae-40c4-b3a5-8d90c81a16de" containerID="aac59c12169a9f78a1c2821f04eae229e58061f72a42b3b36dfbe30b18d95599" exitCode=60
Nov 02 15:30:16.087581 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:16.087429    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-fda646-29893890-g8wfc" event={"ID":"3b856b33-22ae-40c4-b3a5-8d90c81a16de","Type":"ContainerDied","Data":"aac59c12169a9f78a1c2821f04eae229e58061f72a42b3b36dfbe30b18d95599"}
Nov 02 15:30:16.087581 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:16.087504    5304 scope.go:122] "RemoveContainer" containerID="6d0b3b9408da320e1ca6727c1fc6eca9ae0b5b62ab16f1b4bc3fa0877b4f849e"
Nov 02 15:30:16.088005 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:16.087985    5304 scope.go:122] "RemoveContainer" containerID="aac59c12169a9f78a1c2821f04eae229e58061f72a42b3b36dfbe30b18d95599"
Nov 02 15:30:16.088398 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:30:16.088377    5304 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-fda646-29893890-g8wfc_odh-ai-gateway-infra(3b856b33-22ae-40c4-b3a5-8d90c81a16de)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-fda646-29893890-g8wfc" podUID="3b856b33-22ae-40c4-b3a5-8d90c81a16de"
Nov 02 15:30:16.127107 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:16.122167    5304 scope.go:122] "RemoveContainer" containerID="dda9efc71ff6cece3b7ac3c956fc135e3c675c91b204b64dacc3e83e8bb559ba"
Nov 02 15:30:16.127107 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:16.123155    5304 scope.go:122] "RemoveContainer" containerID="7c88b8e712361eea0ce0dc4f9481a80825db2334206ca483f2e3f041aadc8067"
Nov 02 15:30:17.099801 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:17.099766    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-6af50f-29893890-95trm" event={"ID":"c0293ead-8a17-4860-8691-12a4bf2d0ed6","Type":"ContainerStarted","Data":"d355b5d4519bc73652b6e82b73931ce31e439c97e70b5f8517f81f8d6bb0dfcd"}
Nov 02 15:30:17.101684 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:17.101658    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8a3fb0-29893890-t64lm" event={"ID":"875ff34a-5486-4647-bbc6-bbd0df2633e6","Type":"ContainerStarted","Data":"d250d04ace4c59f92dfff0c8552c7b7efaf03178963758267b491a63570c2007"}
Nov 02 15:30:17.111486 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:17.111456    5304 scope.go:122] "RemoveContainer" containerID="fc3d88a1e25b5bbfd483a2b6a9bb6885ba8cca5c46c3a5fc09b741c09d06d2ec"
Nov 02 15:30:17.111621 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:17.111592    5304 scope.go:122] "RemoveContainer" containerID="a42f17989f46323b71e7cd3bf8ffb8a598132644fa7984087c3cdb1155468209"
Nov 02 15:30:17.111783 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:17.111765    5304 scope.go:122] "RemoveContainer" containerID="dff8ba2f4e5d6af9da8a3c2e8861d5c5d546189ee190717805ba5d5ff478e381"
Nov 02 15:30:17.181956 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:17.181919    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-fda646-29893890-g8wfc"]
Nov 02 15:30:17.496451 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:17.496428    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-fda646-29893890-g8wfc"
Nov 02 15:30:17.564692 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:17.564660    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3b856b33-22ae-40c4-b3a5-8d90c81a16de-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b856b33-22ae-40c4-b3a5-8d90c81a16de-maas-api-service-ca\") pod \"3b856b33-22ae-40c4-b3a5-8d90c81a16de\" (UID: \"3b856b33-22ae-40c4-b3a5-8d90c81a16de\") "
Nov 02 15:30:17.564692 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:17.564809    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b856b33-22ae-40c4-b3a5-8d90c81a16de-kube-api-access-cpflk\" (UniqueName: \"kubernetes.io/projected/3b856b33-22ae-40c4-b3a5-8d90c81a16de-kube-api-access-cpflk\") pod \"3b856b33-22ae-40c4-b3a5-8d90c81a16de\" (UID: \"3b856b33-22ae-40c4-b3a5-8d90c81a16de\") "
Nov 02 15:30:17.565074 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:17.565054    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b856b33-22ae-40c4-b3a5-8d90c81a16de-maas-api-service-ca" pod "3b856b33-22ae-40c4-b3a5-8d90c81a16de" (UID: "3b856b33-22ae-40c4-b3a5-8d90c81a16de"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:30:17.565866 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:17.565842    5304 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b856b33-22ae-40c4-b3a5-8d90c81a16de-maas-api-service-ca\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:30:17.567327 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:17.567290    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b856b33-22ae-40c4-b3a5-8d90c81a16de-kube-api-access-cpflk" pod "3b856b33-22ae-40c4-b3a5-8d90c81a16de" (UID: "3b856b33-22ae-40c4-b3a5-8d90c81a16de"). InnerVolumeSpecName "kube-api-access-cpflk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:30:17.667857 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:17.667825    5304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cpflk\" (UniqueName: \"kubernetes.io/projected/3b856b33-22ae-40c4-b3a5-8d90c81a16de-kube-api-access-cpflk\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:30:18.111409 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:18.111381    5304 scope.go:122] "RemoveContainer" containerID="46a9bb80820904b38a92ef599ccba74b30d65eefbc4fd30075cf657d313dcc5a"
Nov 02 15:30:18.111686 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:18.111655    5304 generic.go:356] "Generic (PLEG): container finished" podID="c0293ead-8a17-4860-8691-12a4bf2d0ed6" containerID="d355b5d4519bc73652b6e82b73931ce31e439c97e70b5f8517f81f8d6bb0dfcd" exitCode=60
Nov 02 15:30:18.111818 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:18.111792    5304 scope.go:122] "RemoveContainer" containerID="e9648fe3c8c705c9ddc66c2ed17eaea97a6a2cae2772483efecb49cf053da18a"
Nov 02 15:30:18.112116 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:18.112056    5304 scope.go:122] "RemoveContainer" containerID="c530719fe0e701c1acb9514d2225cd11a3d88f7b39a266b01d3fc5770727a6ef"
Nov 02 15:30:18.115714 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:18.115693    5304 generic.go:356] "Generic (PLEG): container finished" podID="875ff34a-5486-4647-bbc6-bbd0df2633e6" containerID="d250d04ace4c59f92dfff0c8552c7b7efaf03178963758267b491a63570c2007" exitCode=60
Nov 02 15:30:18.119291 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:18.119271    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-fda646-29893890-g8wfc"
Nov 02 15:30:18.124099 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:18.124073    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-6af50f-29893890-95trm" event={"ID":"c0293ead-8a17-4860-8691-12a4bf2d0ed6","Type":"ContainerDied","Data":"d355b5d4519bc73652b6e82b73931ce31e439c97e70b5f8517f81f8d6bb0dfcd"}
Nov 02 15:30:18.124216 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:18.124173    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8a3fb0-29893890-t64lm" event={"ID":"875ff34a-5486-4647-bbc6-bbd0df2633e6","Type":"ContainerDied","Data":"d250d04ace4c59f92dfff0c8552c7b7efaf03178963758267b491a63570c2007"}
Nov 02 15:30:18.124216 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:18.124207    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-28f621-29893890-8pdnl" event={"ID":"d6984b63-9bb9-4081-a3ba-51be6260e1ea","Type":"ContainerStarted","Data":"7855a8d76ab8e33b470cd3c19d64cd9b681362e7a2dad65ddcab6bb18bd1c990"}
Nov 02 15:30:18.124341 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:18.124215    5304 scope.go:122] "RemoveContainer" containerID="dda9efc71ff6cece3b7ac3c956fc135e3c675c91b204b64dacc3e83e8bb559ba"
Nov 02 15:30:18.124341 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:18.124231    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-fda646-29893890-g8wfc" event={"ID":"3b856b33-22ae-40c4-b3a5-8d90c81a16de","Type":"ContainerDied","Data":"07c7e43068fc2afde5be03ed191e031a86ee1fa35f780ba25011b3b4081e53a4"}
Nov 02 15:30:18.124341 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:18.124257    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2e4ebc-29893890-qbrcz" event={"ID":"5341d918-4b2b-4c8d-8898-0e764f23c230","Type":"ContainerStarted","Data":"a543cf53371d40f3ce4e60e5e582f3bfa6f54ef798907a4b4050f778e831d201"}
Nov 02 15:30:18.124341 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:18.124281    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-246137-29893890-h98qk" event={"ID":"8f70c555-72cb-453b-8d47-8bdbe39222e7","Type":"ContainerStarted","Data":"927a848e251ed6b5c545b0e56afc1a60892d9cdc12488c0dfc15ea2564fad6b8"}
Nov 02 15:30:18.125229 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:18.125156    5304 scope.go:122] "RemoveContainer" containerID="d250d04ace4c59f92dfff0c8552c7b7efaf03178963758267b491a63570c2007"
Nov 02 15:30:18.125412 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:18.125395    5304 scope.go:122] "RemoveContainer" containerID="d355b5d4519bc73652b6e82b73931ce31e439c97e70b5f8517f81f8d6bb0dfcd"
Nov 02 15:30:18.125504 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:30:18.125432    5304 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-8a3fb0-29893890-t64lm_odh-ai-gateway-infra(875ff34a-5486-4647-bbc6-bbd0df2633e6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8a3fb0-29893890-t64lm" podUID="875ff34a-5486-4647-bbc6-bbd0df2633e6"
Nov 02 15:30:18.125664 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:30:18.125642    5304 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-6af50f-29893890-95trm_odh-ai-gateway-infra(c0293ead-8a17-4860-8691-12a4bf2d0ed6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-6af50f-29893890-95trm" podUID="c0293ead-8a17-4860-8691-12a4bf2d0ed6"
Nov 02 15:30:18.154992 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:18.154966    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-6af50f-29893890-95trm"]
Nov 02 15:30:18.209056 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:18.209024    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8a3fb0-29893890-t64lm"]
Nov 02 15:30:18.282576 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:18.282544    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-fda646-29893890-g8wfc"]
Nov 02 15:30:18.292371 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:18.292344    5304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-fda646-29893890-g8wfc"]
Nov 02 15:30:18.413730 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:18.413702    5304 scope.go:122] "RemoveContainer" containerID="7c88b8e712361eea0ce0dc4f9481a80825db2334206ca483f2e3f041aadc8067"
Nov 02 15:30:18.430324 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:18.430301    5304 scope.go:122] "RemoveContainer" containerID="aac59c12169a9f78a1c2821f04eae229e58061f72a42b3b36dfbe30b18d95599"
Nov 02 15:30:19.111783 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:19.111750    5304 scope.go:122] "RemoveContainer" containerID="4b3d77921ae79c47c7f8e9cf12563cfa531684342f1ae90753a974682205ac89"
Nov 02 15:30:19.136821 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:19.136789    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7ecfe1-29893890-bxmzk" event={"ID":"33921120-0f00-4584-9e1d-d486c475bd10","Type":"ContainerStarted","Data":"ec5e15e902c3edc6e8568d542adf25b68f0750c089141856c87532be5c8322ec"}
Nov 02 15:30:19.140747 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:19.140719    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-624254-29893890-hxj28" event={"ID":"b2b513e5-c608-45e2-8dc6-d798bf902d83","Type":"ContainerStarted","Data":"7c3ff546d7b92e9f9ff19c0a731b2b42bd63ae0a5111654250c385048237cbfa"}
Nov 02 15:30:19.144010 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:19.143195    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-501738-29893890-59r7h" event={"ID":"8b61992b-ffa1-4606-bf49-9efccefbc460","Type":"ContainerStarted","Data":"300eaaf3199ebd4afdef968eae8885ca8b41a30e69aea460352cd2e3618e7d50"}
Nov 02 15:30:19.146136 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:19.146098    5304 generic.go:356] "Generic (PLEG): container finished" podID="d6984b63-9bb9-4081-a3ba-51be6260e1ea" containerID="7855a8d76ab8e33b470cd3c19d64cd9b681362e7a2dad65ddcab6bb18bd1c990" exitCode=60
Nov 02 15:30:19.146250 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:19.146173    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-28f621-29893890-8pdnl" event={"ID":"d6984b63-9bb9-4081-a3ba-51be6260e1ea","Type":"ContainerDied","Data":"7855a8d76ab8e33b470cd3c19d64cd9b681362e7a2dad65ddcab6bb18bd1c990"}
Nov 02 15:30:19.146314 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:19.146246    5304 scope.go:122] "RemoveContainer" containerID="a42f17989f46323b71e7cd3bf8ffb8a598132644fa7984087c3cdb1155468209"
Nov 02 15:30:19.147287 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:19.147265    5304 scope.go:122] "RemoveContainer" containerID="7855a8d76ab8e33b470cd3c19d64cd9b681362e7a2dad65ddcab6bb18bd1c990"
Nov 02 15:30:19.148761 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:30:19.147641    5304 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-28f621-29893890-8pdnl_odh-ai-gateway-infra(d6984b63-9bb9-4081-a3ba-51be6260e1ea)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-28f621-29893890-8pdnl" podUID="d6984b63-9bb9-4081-a3ba-51be6260e1ea"
Nov 02 15:30:19.150516 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:19.150492    5304 generic.go:356] "Generic (PLEG): container finished" podID="5341d918-4b2b-4c8d-8898-0e764f23c230" containerID="a543cf53371d40f3ce4e60e5e582f3bfa6f54ef798907a4b4050f778e831d201" exitCode=60
Nov 02 15:30:19.150615 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:19.150564    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2e4ebc-29893890-qbrcz" event={"ID":"5341d918-4b2b-4c8d-8898-0e764f23c230","Type":"ContainerDied","Data":"a543cf53371d40f3ce4e60e5e582f3bfa6f54ef798907a4b4050f778e831d201"}
Nov 02 15:30:19.150913 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:19.150892    5304 scope.go:122] "RemoveContainer" containerID="a543cf53371d40f3ce4e60e5e582f3bfa6f54ef798907a4b4050f778e831d201"
Nov 02 15:30:19.151198 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:30:19.151173    5304 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-2e4ebc-29893890-qbrcz_odh-ai-gateway-infra(5341d918-4b2b-4c8d-8898-0e764f23c230)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2e4ebc-29893890-qbrcz" podUID="5341d918-4b2b-4c8d-8898-0e764f23c230"
Nov 02 15:30:19.152884 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:19.152860    5304 generic.go:356] "Generic (PLEG): container finished" podID="8f70c555-72cb-453b-8d47-8bdbe39222e7" containerID="927a848e251ed6b5c545b0e56afc1a60892d9cdc12488c0dfc15ea2564fad6b8" exitCode=60
Nov 02 15:30:19.152988 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:19.152915    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-246137-29893890-h98qk" event={"ID":"8f70c555-72cb-453b-8d47-8bdbe39222e7","Type":"ContainerDied","Data":"927a848e251ed6b5c545b0e56afc1a60892d9cdc12488c0dfc15ea2564fad6b8"}
Nov 02 15:30:19.153308 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:19.153288    5304 scope.go:122] "RemoveContainer" containerID="927a848e251ed6b5c545b0e56afc1a60892d9cdc12488c0dfc15ea2564fad6b8"
Nov 02 15:30:19.153625 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:30:19.153603    5304 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w5-246137-29893890-h98qk_odh-ai-gateway-infra(8f70c555-72cb-453b-8d47-8bdbe39222e7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-246137-29893890-h98qk" podUID="8f70c555-72cb-453b-8d47-8bdbe39222e7"
Nov 02 15:30:19.167023 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:19.166995    5304 scope.go:122] "RemoveContainer" containerID="fc3d88a1e25b5bbfd483a2b6a9bb6885ba8cca5c46c3a5fc09b741c09d06d2ec"
Nov 02 15:30:19.188183 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:19.188162    5304 scope.go:122] "RemoveContainer" containerID="dff8ba2f4e5d6af9da8a3c2e8861d5c5d546189ee190717805ba5d5ff478e381"
Nov 02 15:30:19.244897 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:19.244863    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2e4ebc-29893890-qbrcz"]
Nov 02 15:30:19.388031 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:19.388000    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-28f621-29893890-8pdnl"]
Nov 02 15:30:19.399499 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:19.399467    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8a3fb0-29893890-t64lm"
Nov 02 15:30:19.402488 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:19.402466    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-6af50f-29893890-95trm"
Nov 02 15:30:19.438434 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:19.438404    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-246137-29893890-h98qk"]
Nov 02 15:30:19.497878 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:19.497839    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c0293ead-8a17-4860-8691-12a4bf2d0ed6-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0293ead-8a17-4860-8691-12a4bf2d0ed6-maas-api-service-ca\") pod \"c0293ead-8a17-4860-8691-12a4bf2d0ed6\" (UID: \"c0293ead-8a17-4860-8691-12a4bf2d0ed6\") "
Nov 02 15:30:19.498046 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:19.497896    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0293ead-8a17-4860-8691-12a4bf2d0ed6-kube-api-access-jg4bc\" (UniqueName: \"kubernetes.io/projected/c0293ead-8a17-4860-8691-12a4bf2d0ed6-kube-api-access-jg4bc\") pod \"c0293ead-8a17-4860-8691-12a4bf2d0ed6\" (UID: \"c0293ead-8a17-4860-8691-12a4bf2d0ed6\") "
Nov 02 15:30:19.498046 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:19.497972    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/875ff34a-5486-4647-bbc6-bbd0df2633e6-kube-api-access-24fdv\" (UniqueName: \"kubernetes.io/projected/875ff34a-5486-4647-bbc6-bbd0df2633e6-kube-api-access-24fdv\") pod \"875ff34a-5486-4647-bbc6-bbd0df2633e6\" (UID: \"875ff34a-5486-4647-bbc6-bbd0df2633e6\") "
Nov 02 15:30:19.498046 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:19.497996    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/875ff34a-5486-4647-bbc6-bbd0df2633e6-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/875ff34a-5486-4647-bbc6-bbd0df2633e6-maas-api-service-ca\") pod \"875ff34a-5486-4647-bbc6-bbd0df2633e6\" (UID: \"875ff34a-5486-4647-bbc6-bbd0df2633e6\") "
Nov 02 15:30:19.498367 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:19.498340    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0293ead-8a17-4860-8691-12a4bf2d0ed6-maas-api-service-ca" pod "c0293ead-8a17-4860-8691-12a4bf2d0ed6" (UID: "c0293ead-8a17-4860-8691-12a4bf2d0ed6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:30:19.498477 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:19.498452    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/875ff34a-5486-4647-bbc6-bbd0df2633e6-maas-api-service-ca" pod "875ff34a-5486-4647-bbc6-bbd0df2633e6" (UID: "875ff34a-5486-4647-bbc6-bbd0df2633e6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:30:19.498883 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:19.498861    5304 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0293ead-8a17-4860-8691-12a4bf2d0ed6-maas-api-service-ca\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:30:19.498981 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:19.498886    5304 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/875ff34a-5486-4647-bbc6-bbd0df2633e6-maas-api-service-ca\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:30:19.531662 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:19.530680    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/875ff34a-5486-4647-bbc6-bbd0df2633e6-kube-api-access-24fdv" pod "875ff34a-5486-4647-bbc6-bbd0df2633e6" (UID: "875ff34a-5486-4647-bbc6-bbd0df2633e6"). InnerVolumeSpecName "kube-api-access-24fdv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:30:19.531662 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:19.530945    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0293ead-8a17-4860-8691-12a4bf2d0ed6-kube-api-access-jg4bc" pod "c0293ead-8a17-4860-8691-12a4bf2d0ed6" (UID: "c0293ead-8a17-4860-8691-12a4bf2d0ed6"). InnerVolumeSpecName "kube-api-access-jg4bc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:30:19.602487 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:19.602452    5304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jg4bc\" (UniqueName: \"kubernetes.io/projected/c0293ead-8a17-4860-8691-12a4bf2d0ed6-kube-api-access-jg4bc\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:30:19.602487 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:19.602488    5304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-24fdv\" (UniqueName: \"kubernetes.io/projected/875ff34a-5486-4647-bbc6-bbd0df2633e6-kube-api-access-24fdv\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:30:20.123210 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:20.123164    5304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3b856b33-22ae-40c4-b3a5-8d90c81a16de" path="/var/lib/kubelet/pods/3b856b33-22ae-40c4-b3a5-8d90c81a16de/volumes"
Nov 02 15:30:20.162893 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:20.162863    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8a3fb0-29893890-t64lm"
Nov 02 15:30:20.162893 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:20.162876    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8a3fb0-29893890-t64lm" event={"ID":"875ff34a-5486-4647-bbc6-bbd0df2633e6","Type":"ContainerDied","Data":"7ba67597d21930d04d8ccb85da2a31bffe867cf305a0f4db977461c0e7746202"}
Nov 02 15:30:20.163146 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:20.162947    5304 scope.go:122] "RemoveContainer" containerID="d250d04ace4c59f92dfff0c8552c7b7efaf03178963758267b491a63570c2007"
Nov 02 15:30:20.165204 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:20.165176    5304 generic.go:356] "Generic (PLEG): container finished" podID="b2b513e5-c608-45e2-8dc6-d798bf902d83" containerID="7c3ff546d7b92e9f9ff19c0a731b2b42bd63ae0a5111654250c385048237cbfa" exitCode=60
Nov 02 15:30:20.165322 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:20.165254    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-624254-29893890-hxj28" event={"ID":"b2b513e5-c608-45e2-8dc6-d798bf902d83","Type":"ContainerDied","Data":"7c3ff546d7b92e9f9ff19c0a731b2b42bd63ae0a5111654250c385048237cbfa"}
Nov 02 15:30:20.165711 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:20.165687    5304 scope.go:122] "RemoveContainer" containerID="7c3ff546d7b92e9f9ff19c0a731b2b42bd63ae0a5111654250c385048237cbfa"
Nov 02 15:30:20.166040 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:30:20.166020    5304 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-624254-29893890-hxj28_odh-ai-gateway-infra(b2b513e5-c608-45e2-8dc6-d798bf902d83)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-624254-29893890-hxj28" podUID="b2b513e5-c608-45e2-8dc6-d798bf902d83"
Nov 02 15:30:20.167471 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:20.167452    5304 generic.go:356] "Generic (PLEG): container finished" podID="fecd4670-5e5f-477b-9f3c-689d6bdb79e4" containerID="75dd8b13868488f5d6f0f26fe2eacd923cdfe0e2a2f0c17abf66f08ad7a37eea" exitCode=60
Nov 02 15:30:20.167560 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:20.167522    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5dd8e9-29893890-wgcbb" event={"ID":"fecd4670-5e5f-477b-9f3c-689d6bdb79e4","Type":"ContainerDied","Data":"75dd8b13868488f5d6f0f26fe2eacd923cdfe0e2a2f0c17abf66f08ad7a37eea"}
Nov 02 15:30:20.168002 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:20.167983    5304 scope.go:122] "RemoveContainer" containerID="75dd8b13868488f5d6f0f26fe2eacd923cdfe0e2a2f0c17abf66f08ad7a37eea"
Nov 02 15:30:20.168327 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:30:20.168306    5304 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-5dd8e9-29893890-wgcbb_odh-ai-gateway-infra(fecd4670-5e5f-477b-9f3c-689d6bdb79e4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5dd8e9-29893890-wgcbb" podUID="fecd4670-5e5f-477b-9f3c-689d6bdb79e4"
Nov 02 15:30:20.169471 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:20.169453    5304 generic.go:356] "Generic (PLEG): container finished" podID="8b61992b-ffa1-4606-bf49-9efccefbc460" containerID="300eaaf3199ebd4afdef968eae8885ca8b41a30e69aea460352cd2e3618e7d50" exitCode=60
Nov 02 15:30:20.169574 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:20.169526    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-501738-29893890-59r7h" event={"ID":"8b61992b-ffa1-4606-bf49-9efccefbc460","Type":"ContainerDied","Data":"300eaaf3199ebd4afdef968eae8885ca8b41a30e69aea460352cd2e3618e7d50"}
Nov 02 15:30:20.169875 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:20.169854    5304 scope.go:122] "RemoveContainer" containerID="300eaaf3199ebd4afdef968eae8885ca8b41a30e69aea460352cd2e3618e7d50"
Nov 02 15:30:20.170138 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:30:20.170106    5304 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-501738-29893890-59r7h_odh-ai-gateway-infra(8b61992b-ffa1-4606-bf49-9efccefbc460)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-501738-29893890-59r7h" podUID="8b61992b-ffa1-4606-bf49-9efccefbc460"
Nov 02 15:30:20.178019 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:20.177995    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-6af50f-29893890-95trm" event={"ID":"c0293ead-8a17-4860-8691-12a4bf2d0ed6","Type":"ContainerDied","Data":"185b87d12bc386c042873afd2ecc4d9ca28b20374ee982b5974e4a6c248e7dd7"}
Nov 02 15:30:20.178019 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:20.178009    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-6af50f-29893890-95trm"
Nov 02 15:30:20.180439 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:20.180417    5304 generic.go:356] "Generic (PLEG): container finished" podID="33921120-0f00-4584-9e1d-d486c475bd10" containerID="ec5e15e902c3edc6e8568d542adf25b68f0750c089141856c87532be5c8322ec" exitCode=60
Nov 02 15:30:20.180559 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:20.180473    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7ecfe1-29893890-bxmzk" event={"ID":"33921120-0f00-4584-9e1d-d486c475bd10","Type":"ContainerDied","Data":"ec5e15e902c3edc6e8568d542adf25b68f0750c089141856c87532be5c8322ec"}
Nov 02 15:30:20.180873 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:20.180853    5304 scope.go:122] "RemoveContainer" containerID="ec5e15e902c3edc6e8568d542adf25b68f0750c089141856c87532be5c8322ec"
Nov 02 15:30:20.181222 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:30:20.181198    5304 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-7ecfe1-29893890-bxmzk_odh-ai-gateway-infra(33921120-0f00-4584-9e1d-d486c475bd10)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7ecfe1-29893890-bxmzk" podUID="33921120-0f00-4584-9e1d-d486c475bd10"
Nov 02 15:30:20.206902 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:20.206716    5304 scope.go:122] "RemoveContainer" containerID="e9648fe3c8c705c9ddc66c2ed17eaea97a6a2cae2772483efecb49cf053da18a"
Nov 02 15:30:20.227273 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:20.227246    5304 scope.go:122] "RemoveContainer" containerID="4b3d77921ae79c47c7f8e9cf12563cfa531684342f1ae90753a974682205ac89"
Nov 02 15:30:20.239828 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:20.239780    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-501738-29893890-59r7h"]
Nov 02 15:30:20.250364 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:20.250341    5304 scope.go:122] "RemoveContainer" containerID="46a9bb80820904b38a92ef599ccba74b30d65eefbc4fd30075cf657d313dcc5a"
Nov 02 15:30:20.271857 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:20.270743    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-624254-29893890-hxj28"]
Nov 02 15:30:20.275778 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:20.275758    5304 scope.go:122] "RemoveContainer" containerID="d355b5d4519bc73652b6e82b73931ce31e439c97e70b5f8517f81f8d6bb0dfcd"
Nov 02 15:30:20.296669 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:20.294833    5304 scope.go:122] "RemoveContainer" containerID="c530719fe0e701c1acb9514d2225cd11a3d88f7b39a266b01d3fc5770727a6ef"
Nov 02 15:30:20.417968 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:20.417936    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-28f621-29893890-8pdnl"
Nov 02 15:30:20.474263 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:20.474235    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2e4ebc-29893890-qbrcz"
Nov 02 15:30:20.478049 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:20.478025    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-246137-29893890-h98qk"
Nov 02 15:30:20.482557 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:20.482532    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7ecfe1-29893890-bxmzk"]
Nov 02 15:30:20.521200 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:20.521169    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f70c555-72cb-453b-8d47-8bdbe39222e7-kube-api-access-6wqjm\" (UniqueName: \"kubernetes.io/projected/8f70c555-72cb-453b-8d47-8bdbe39222e7-kube-api-access-6wqjm\") pod \"8f70c555-72cb-453b-8d47-8bdbe39222e7\" (UID: \"8f70c555-72cb-453b-8d47-8bdbe39222e7\") "
Nov 02 15:30:20.521392 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:20.521236    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5341d918-4b2b-4c8d-8898-0e764f23c230-kube-api-access-xbvrq\" (UniqueName: \"kubernetes.io/projected/5341d918-4b2b-4c8d-8898-0e764f23c230-kube-api-access-xbvrq\") pod \"5341d918-4b2b-4c8d-8898-0e764f23c230\" (UID: \"5341d918-4b2b-4c8d-8898-0e764f23c230\") "
Nov 02 15:30:20.521392 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:20.521302    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5341d918-4b2b-4c8d-8898-0e764f23c230-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5341d918-4b2b-4c8d-8898-0e764f23c230-maas-api-service-ca\") pod \"5341d918-4b2b-4c8d-8898-0e764f23c230\" (UID: \"5341d918-4b2b-4c8d-8898-0e764f23c230\") "
Nov 02 15:30:20.521506 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:20.521456    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6984b63-9bb9-4081-a3ba-51be6260e1ea-kube-api-access-g6kl4\" (UniqueName: \"kubernetes.io/projected/d6984b63-9bb9-4081-a3ba-51be6260e1ea-kube-api-access-g6kl4\") pod \"d6984b63-9bb9-4081-a3ba-51be6260e1ea\" (UID: \"d6984b63-9bb9-4081-a3ba-51be6260e1ea\") "
Nov 02 15:30:20.521559 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:20.521514    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d6984b63-9bb9-4081-a3ba-51be6260e1ea-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d6984b63-9bb9-4081-a3ba-51be6260e1ea-maas-api-service-ca\") pod \"d6984b63-9bb9-4081-a3ba-51be6260e1ea\" (UID: \"d6984b63-9bb9-4081-a3ba-51be6260e1ea\") "
Nov 02 15:30:20.521613 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:20.521592    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8f70c555-72cb-453b-8d47-8bdbe39222e7-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f70c555-72cb-453b-8d47-8bdbe39222e7-maas-api-service-ca\") pod \"8f70c555-72cb-453b-8d47-8bdbe39222e7\" (UID: \"8f70c555-72cb-453b-8d47-8bdbe39222e7\") "
Nov 02 15:30:20.521668 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:20.521648    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5341d918-4b2b-4c8d-8898-0e764f23c230-maas-api-service-ca" pod "5341d918-4b2b-4c8d-8898-0e764f23c230" (UID: "5341d918-4b2b-4c8d-8898-0e764f23c230"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:30:20.521964 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:20.521891    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6984b63-9bb9-4081-a3ba-51be6260e1ea-maas-api-service-ca" pod "d6984b63-9bb9-4081-a3ba-51be6260e1ea" (UID: "d6984b63-9bb9-4081-a3ba-51be6260e1ea"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:30:20.522196 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:20.522167    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f70c555-72cb-453b-8d47-8bdbe39222e7-maas-api-service-ca" pod "8f70c555-72cb-453b-8d47-8bdbe39222e7" (UID: "8f70c555-72cb-453b-8d47-8bdbe39222e7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:30:20.522410 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:20.522394    5304 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5341d918-4b2b-4c8d-8898-0e764f23c230-maas-api-service-ca\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:30:20.522508 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:20.522414    5304 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d6984b63-9bb9-4081-a3ba-51be6260e1ea-maas-api-service-ca\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:30:20.522508 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:20.522429    5304 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f70c555-72cb-453b-8d47-8bdbe39222e7-maas-api-service-ca\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:30:20.523622 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:20.523593    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5341d918-4b2b-4c8d-8898-0e764f23c230-kube-api-access-xbvrq" pod "5341d918-4b2b-4c8d-8898-0e764f23c230" (UID: "5341d918-4b2b-4c8d-8898-0e764f23c230"). InnerVolumeSpecName "kube-api-access-xbvrq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:30:20.523834 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:20.523814    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f70c555-72cb-453b-8d47-8bdbe39222e7-kube-api-access-6wqjm" pod "8f70c555-72cb-453b-8d47-8bdbe39222e7" (UID: "8f70c555-72cb-453b-8d47-8bdbe39222e7"). InnerVolumeSpecName "kube-api-access-6wqjm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:30:20.523957 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:20.523939    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6984b63-9bb9-4081-a3ba-51be6260e1ea-kube-api-access-g6kl4" pod "d6984b63-9bb9-4081-a3ba-51be6260e1ea" (UID: "d6984b63-9bb9-4081-a3ba-51be6260e1ea"). InnerVolumeSpecName "kube-api-access-g6kl4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:30:20.557657 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:20.557620    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-6af50f-29893890-95trm"]
Nov 02 15:30:20.565330 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:20.565296    5304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-6af50f-29893890-95trm"]
Nov 02 15:30:20.623527 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:20.623492    5304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xbvrq\" (UniqueName: \"kubernetes.io/projected/5341d918-4b2b-4c8d-8898-0e764f23c230-kube-api-access-xbvrq\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:30:20.623527 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:20.623524    5304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g6kl4\" (UniqueName: \"kubernetes.io/projected/d6984b63-9bb9-4081-a3ba-51be6260e1ea-kube-api-access-g6kl4\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:30:20.623527 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:20.623536    5304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6wqjm\" (UniqueName: \"kubernetes.io/projected/8f70c555-72cb-453b-8d47-8bdbe39222e7-kube-api-access-6wqjm\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:30:20.727751 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:20.727667    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8a3fb0-29893890-t64lm"]
Nov 02 15:30:20.735560 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:20.735528    5304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8a3fb0-29893890-t64lm"]
Nov 02 15:30:21.195849 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:21.195827    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-28f621-29893890-8pdnl"
Nov 02 15:30:21.195981 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:21.195859    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-28f621-29893890-8pdnl" event={"ID":"d6984b63-9bb9-4081-a3ba-51be6260e1ea","Type":"ContainerDied","Data":"a4ec0704d04ec3dd45fdb77fd9e9a32ba9eebd18ef54d41f5bb23d205c8d9430"}
Nov 02 15:30:21.195981 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:21.195922    5304 scope.go:122] "RemoveContainer" containerID="7855a8d76ab8e33b470cd3c19d64cd9b681362e7a2dad65ddcab6bb18bd1c990"
Nov 02 15:30:21.197516 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:21.197495    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2e4ebc-29893890-qbrcz"
Nov 02 15:30:21.197619 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:21.197516    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2e4ebc-29893890-qbrcz" event={"ID":"5341d918-4b2b-4c8d-8898-0e764f23c230","Type":"ContainerDied","Data":"2366ffe03ece6e22a102c5b8d89c98c486a207a41aa0ffe107084003a0ae966f"}
Nov 02 15:30:21.199253 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:21.199219    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-246137-29893890-h98qk"
Nov 02 15:30:21.199377 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:21.199313    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-246137-29893890-h98qk" event={"ID":"8f70c555-72cb-453b-8d47-8bdbe39222e7","Type":"ContainerDied","Data":"85f6aa8e7f0fb8e59f02e0f080248ce85acc00298fabb465812b291936082c2e"}
Nov 02 15:30:21.210831 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:21.210805    5304 scope.go:122] "RemoveContainer" containerID="a543cf53371d40f3ce4e60e5e582f3bfa6f54ef798907a4b4050f778e831d201"
Nov 02 15:30:21.227473 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:21.227447    5304 scope.go:122] "RemoveContainer" containerID="927a848e251ed6b5c545b0e56afc1a60892d9cdc12488c0dfc15ea2564fad6b8"
Nov 02 15:30:21.259167 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:21.257998    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5dd8e9-29893890-wgcbb"]
Nov 02 15:30:21.295934 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:21.295898    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2e4ebc-29893890-qbrcz"]
Nov 02 15:30:21.302179 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:21.302143    5304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2e4ebc-29893890-qbrcz"]
Nov 02 15:30:21.343761 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:21.337484    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-28f621-29893890-8pdnl"]
Nov 02 15:30:21.343761 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:21.343458    5304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-28f621-29893890-8pdnl"]
Nov 02 15:30:21.429774 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:21.429257    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-246137-29893890-h98qk"]
Nov 02 15:30:21.429774 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:21.429708    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-624254-29893890-hxj28"
Nov 02 15:30:21.434916 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:21.434884    5304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-246137-29893890-h98qk"]
Nov 02 15:30:21.512142 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:21.512103    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-501738-29893890-59r7h"
Nov 02 15:30:21.528982 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:21.527301    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7ecfe1-29893890-bxmzk"
Nov 02 15:30:21.546866 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:21.538747    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b2b513e5-c608-45e2-8dc6-d798bf902d83-kube-api-access-44dq2\" (UniqueName: \"kubernetes.io/projected/b2b513e5-c608-45e2-8dc6-d798bf902d83-kube-api-access-44dq2\") pod \"b2b513e5-c608-45e2-8dc6-d798bf902d83\" (UID: \"b2b513e5-c608-45e2-8dc6-d798bf902d83\") "
Nov 02 15:30:21.546866 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:21.538964    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b2b513e5-c608-45e2-8dc6-d798bf902d83-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2b513e5-c608-45e2-8dc6-d798bf902d83-maas-api-service-ca\") pod \"b2b513e5-c608-45e2-8dc6-d798bf902d83\" (UID: \"b2b513e5-c608-45e2-8dc6-d798bf902d83\") "
Nov 02 15:30:21.546866 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:21.546165    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2b513e5-c608-45e2-8dc6-d798bf902d83-maas-api-service-ca" pod "b2b513e5-c608-45e2-8dc6-d798bf902d83" (UID: "b2b513e5-c608-45e2-8dc6-d798bf902d83"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:30:21.556309 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:21.553249    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5dd8e9-29893890-wgcbb"
Nov 02 15:30:21.616522 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:21.616322    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2b513e5-c608-45e2-8dc6-d798bf902d83-kube-api-access-44dq2" pod "b2b513e5-c608-45e2-8dc6-d798bf902d83" (UID: "b2b513e5-c608-45e2-8dc6-d798bf902d83"). InnerVolumeSpecName "kube-api-access-44dq2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:30:21.646620 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:21.646586    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33921120-0f00-4584-9e1d-d486c475bd10-kube-api-access-zzwx9\" (UniqueName: \"kubernetes.io/projected/33921120-0f00-4584-9e1d-d486c475bd10-kube-api-access-zzwx9\") pod \"33921120-0f00-4584-9e1d-d486c475bd10\" (UID: \"33921120-0f00-4584-9e1d-d486c475bd10\") "
Nov 02 15:30:21.646822 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:21.646796    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8b61992b-ffa1-4606-bf49-9efccefbc460-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b61992b-ffa1-4606-bf49-9efccefbc460-maas-api-service-ca\") pod \"8b61992b-ffa1-4606-bf49-9efccefbc460\" (UID: \"8b61992b-ffa1-4606-bf49-9efccefbc460\") "
Nov 02 15:30:21.647009 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:21.646986    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b61992b-ffa1-4606-bf49-9efccefbc460-kube-api-access-jzvsn\" (UniqueName: \"kubernetes.io/projected/8b61992b-ffa1-4606-bf49-9efccefbc460-kube-api-access-jzvsn\") pod \"8b61992b-ffa1-4606-bf49-9efccefbc460\" (UID: \"8b61992b-ffa1-4606-bf49-9efccefbc460\") "
Nov 02 15:30:21.647437 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:21.647253    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b61992b-ffa1-4606-bf49-9efccefbc460-maas-api-service-ca" pod "8b61992b-ffa1-4606-bf49-9efccefbc460" (UID: "8b61992b-ffa1-4606-bf49-9efccefbc460"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:30:21.647533 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:21.647432    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fecd4670-5e5f-477b-9f3c-689d6bdb79e4-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fecd4670-5e5f-477b-9f3c-689d6bdb79e4-maas-api-service-ca\") pod \"fecd4670-5e5f-477b-9f3c-689d6bdb79e4\" (UID: \"fecd4670-5e5f-477b-9f3c-689d6bdb79e4\") "
Nov 02 15:30:21.647533 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:21.647494    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fecd4670-5e5f-477b-9f3c-689d6bdb79e4-kube-api-access-9vtr9\" (UniqueName: \"kubernetes.io/projected/fecd4670-5e5f-477b-9f3c-689d6bdb79e4-kube-api-access-9vtr9\") pod \"fecd4670-5e5f-477b-9f3c-689d6bdb79e4\" (UID: \"fecd4670-5e5f-477b-9f3c-689d6bdb79e4\") "
Nov 02 15:30:21.647640 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:21.647556    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/33921120-0f00-4584-9e1d-d486c475bd10-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/33921120-0f00-4584-9e1d-d486c475bd10-maas-api-service-ca\") pod \"33921120-0f00-4584-9e1d-d486c475bd10\" (UID: \"33921120-0f00-4584-9e1d-d486c475bd10\") "
Nov 02 15:30:21.647941 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:21.647917    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fecd4670-5e5f-477b-9f3c-689d6bdb79e4-maas-api-service-ca" pod "fecd4670-5e5f-477b-9f3c-689d6bdb79e4" (UID: "fecd4670-5e5f-477b-9f3c-689d6bdb79e4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:30:21.648057 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:21.648033    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33921120-0f00-4584-9e1d-d486c475bd10-maas-api-service-ca" pod "33921120-0f00-4584-9e1d-d486c475bd10" (UID: "33921120-0f00-4584-9e1d-d486c475bd10"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:30:21.648490 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:21.648465    5304 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fecd4670-5e5f-477b-9f3c-689d6bdb79e4-maas-api-service-ca\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:30:21.648598 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:21.648493    5304 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/33921120-0f00-4584-9e1d-d486c475bd10-maas-api-service-ca\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:30:21.648598 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:21.648517    5304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-44dq2\" (UniqueName: \"kubernetes.io/projected/b2b513e5-c608-45e2-8dc6-d798bf902d83-kube-api-access-44dq2\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:30:21.648598 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:21.648531    5304 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b61992b-ffa1-4606-bf49-9efccefbc460-maas-api-service-ca\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:30:21.648598 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:21.648544    5304 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2b513e5-c608-45e2-8dc6-d798bf902d83-maas-api-service-ca\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:30:21.649949 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:21.649926    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b61992b-ffa1-4606-bf49-9efccefbc460-kube-api-access-jzvsn" pod "8b61992b-ffa1-4606-bf49-9efccefbc460" (UID: "8b61992b-ffa1-4606-bf49-9efccefbc460"). InnerVolumeSpecName "kube-api-access-jzvsn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:30:21.650401 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:21.650374    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fecd4670-5e5f-477b-9f3c-689d6bdb79e4-kube-api-access-9vtr9" pod "fecd4670-5e5f-477b-9f3c-689d6bdb79e4" (UID: "fecd4670-5e5f-477b-9f3c-689d6bdb79e4"). InnerVolumeSpecName "kube-api-access-9vtr9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:30:21.650491 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:21.650406    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33921120-0f00-4584-9e1d-d486c475bd10-kube-api-access-zzwx9" pod "33921120-0f00-4584-9e1d-d486c475bd10" (UID: "33921120-0f00-4584-9e1d-d486c475bd10"). InnerVolumeSpecName "kube-api-access-zzwx9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:30:21.751014 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:21.750918    5304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9vtr9\" (UniqueName: \"kubernetes.io/projected/fecd4670-5e5f-477b-9f3c-689d6bdb79e4-kube-api-access-9vtr9\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:30:21.751014 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:21.750960    5304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zzwx9\" (UniqueName: \"kubernetes.io/projected/33921120-0f00-4584-9e1d-d486c475bd10-kube-api-access-zzwx9\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:30:21.751014 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:21.750974    5304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jzvsn\" (UniqueName: \"kubernetes.io/projected/8b61992b-ffa1-4606-bf49-9efccefbc460-kube-api-access-jzvsn\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:30:22.121282 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:22.121251    5304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5341d918-4b2b-4c8d-8898-0e764f23c230" path="/var/lib/kubelet/pods/5341d918-4b2b-4c8d-8898-0e764f23c230/volumes"
Nov 02 15:30:22.121751 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:22.121737    5304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="875ff34a-5486-4647-bbc6-bbd0df2633e6" path="/var/lib/kubelet/pods/875ff34a-5486-4647-bbc6-bbd0df2633e6/volumes"
Nov 02 15:30:22.122152 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:22.122137    5304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8f70c555-72cb-453b-8d47-8bdbe39222e7" path="/var/lib/kubelet/pods/8f70c555-72cb-453b-8d47-8bdbe39222e7/volumes"
Nov 02 15:30:22.122850 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:22.122828    5304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c0293ead-8a17-4860-8691-12a4bf2d0ed6" path="/var/lib/kubelet/pods/c0293ead-8a17-4860-8691-12a4bf2d0ed6/volumes"
Nov 02 15:30:22.123334 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:22.123319    5304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d6984b63-9bb9-4081-a3ba-51be6260e1ea" path="/var/lib/kubelet/pods/d6984b63-9bb9-4081-a3ba-51be6260e1ea/volumes"
Nov 02 15:30:22.223702 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:22.223672    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7ecfe1-29893890-bxmzk"
Nov 02 15:30:22.223892 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:22.223703    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7ecfe1-29893890-bxmzk" event={"ID":"33921120-0f00-4584-9e1d-d486c475bd10","Type":"ContainerDied","Data":"7aeb2f44969147ab10a33d9d2e2b64409efab2bd8ca131c8cd065ebb540b2760"}
Nov 02 15:30:22.223892 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:22.223768    5304 scope.go:122] "RemoveContainer" containerID="ec5e15e902c3edc6e8568d542adf25b68f0750c089141856c87532be5c8322ec"
Nov 02 15:30:22.226331 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:22.226291    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-624254-29893890-hxj28" event={"ID":"b2b513e5-c608-45e2-8dc6-d798bf902d83","Type":"ContainerDied","Data":"8fb9bc944da30c0f4846e86c7094c0836b71eae476d78e0255cb33b584462be3"}
Nov 02 15:30:22.226473 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:22.226460    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-624254-29893890-hxj28"
Nov 02 15:30:22.228027 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:22.227999    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5dd8e9-29893890-wgcbb" event={"ID":"fecd4670-5e5f-477b-9f3c-689d6bdb79e4","Type":"ContainerDied","Data":"bb10da287d596077ee1dd72cabc77d2db26d476321adab94f7ba7ea585c98dfa"}
Nov 02 15:30:22.228173 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:22.228148    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5dd8e9-29893890-wgcbb"
Nov 02 15:30:22.230070 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:22.230045    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-501738-29893890-59r7h" event={"ID":"8b61992b-ffa1-4606-bf49-9efccefbc460","Type":"ContainerDied","Data":"006c11978ccdfa84819ff2e80c18b280b1fb46d65253d64c15ffed1c787f8c83"}
Nov 02 15:30:22.230201 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:22.230191    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-501738-29893890-59r7h"
Nov 02 15:30:22.245486 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:22.245463    5304 scope.go:122] "RemoveContainer" containerID="7c3ff546d7b92e9f9ff19c0a731b2b42bd63ae0a5111654250c385048237cbfa"
Nov 02 15:30:22.255373 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:22.255354    5304 scope.go:122] "RemoveContainer" containerID="75dd8b13868488f5d6f0f26fe2eacd923cdfe0e2a2f0c17abf66f08ad7a37eea"
Nov 02 15:30:22.264523 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:22.264495    5304 scope.go:122] "RemoveContainer" containerID="300eaaf3199ebd4afdef968eae8885ca8b41a30e69aea460352cd2e3618e7d50"
Nov 02 15:30:22.322611 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:22.322579    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7ecfe1-29893890-bxmzk"]
Nov 02 15:30:22.327382 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:22.327352    5304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7ecfe1-29893890-bxmzk"]
Nov 02 15:30:22.367887 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:22.367856    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5dd8e9-29893890-wgcbb"]
Nov 02 15:30:22.372271 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:22.372215    5304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5dd8e9-29893890-wgcbb"]
Nov 02 15:30:22.406544 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:22.406518    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-501738-29893890-59r7h"]
Nov 02 15:30:22.410469 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:22.410446    5304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-501738-29893890-59r7h"]
Nov 02 15:30:22.442510 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:22.442482    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-624254-29893890-hxj28"]
Nov 02 15:30:22.448340 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:22.448312    5304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-624254-29893890-hxj28"]
Nov 02 15:30:24.120565 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:24.120528    5304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="33921120-0f00-4584-9e1d-d486c475bd10" path="/var/lib/kubelet/pods/33921120-0f00-4584-9e1d-d486c475bd10/volumes"
Nov 02 15:30:24.121018 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:24.121003    5304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8b61992b-ffa1-4606-bf49-9efccefbc460" path="/var/lib/kubelet/pods/8b61992b-ffa1-4606-bf49-9efccefbc460/volumes"
Nov 02 15:30:24.121444 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:24.121430    5304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b2b513e5-c608-45e2-8dc6-d798bf902d83" path="/var/lib/kubelet/pods/b2b513e5-c608-45e2-8dc6-d798bf902d83/volumes"
Nov 02 15:30:24.122153 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:24.122121    5304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fecd4670-5e5f-477b-9f3c-689d6bdb79e4" path="/var/lib/kubelet/pods/fecd4670-5e5f-477b-9f3c-689d6bdb79e4/volumes"
Nov 02 15:30:24.799856 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:30:24.799821    5304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k6fjk" containerName="extractor"
Nov 02 15:30:42.525975 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:42.525903    5304 generic.go:356] "Generic (PLEG): container finished" podID="e4bb2a0e-a2da-44e1-ac19-ee296cfff71c" containerID="7e4a3e30cfa510238243eb2e994d8e6d8236791608cbebd8fb72a14a3396aa63" exitCode=0
Nov 02 15:30:42.526546 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:42.526223    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-28f621-openshift-default-7dd8c5c789-jvmsw" event={"ID":"e4bb2a0e-a2da-44e1-ac19-ee296cfff71c","Type":"ContainerDied","Data":"7e4a3e30cfa510238243eb2e994d8e6d8236791608cbebd8fb72a14a3396aa63"}
Nov 02 15:30:42.526943 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:42.526908    5304 scope.go:122] "RemoveContainer" containerID="7e4a3e30cfa510238243eb2e994d8e6d8236791608cbebd8fb72a14a3396aa63"
Nov 02 15:30:42.530351 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:42.530311    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:30:42.530613 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:42.530590    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:30:42.530804 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:42.530785    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:30:42.743109 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:42.743043    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-cp5ms"]
Nov 02 15:30:42.743682 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:42.743642    5304 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-cp5ms" podUID="11a65ba1-0a55-4478-b043-637c0a9d6d64" containerName="istio-proxy" containerID="cri-o://f6c95516a630ebb1446e90f41aa47193036cdcdc56b49b12fa665ec5574f07af" gracePeriod=30
Nov 02 15:30:43.560083 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:43.560041    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-28f621-openshift-default-7dd8c5c789-jvmsw" event={"ID":"e4bb2a0e-a2da-44e1-ac19-ee296cfff71c","Type":"ContainerStarted","Data":"cfc0b88601501226138466e5118589e463cd9542b9b211e6989599faf715e907"}
Nov 02 15:30:44.260203 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:44.260167    5304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-28f621-openshift-default-7dd8c5c789-jvmsw"
Nov 02 15:30:44.261361 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:44.261334    5304 patch_prober.go:29] interesting pod/e2e-inf-a-28f621-openshift-default-7dd8c5c789-jvmsw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" start-of-body=
Nov 02 15:30:44.261505 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:44.261388    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-28f621-openshift-default-7dd8c5c789-jvmsw" podUID="e4bb2a0e-a2da-44e1-ac19-ee296cfff71c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused"
Nov 02 15:30:45.259652 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:45.259617    5304 patch_prober.go:29] interesting pod/e2e-inf-a-28f621-openshift-default-7dd8c5c789-jvmsw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" start-of-body=
Nov 02 15:30:45.260055 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:45.259673    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-28f621-openshift-default-7dd8c5c789-jvmsw" podUID="e4bb2a0e-a2da-44e1-ac19-ee296cfff71c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused"
Nov 02 15:30:46.150275 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:46.150238    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-28f621-openshift-default-7dd8c5c789-jvmsw"
Nov 02 15:30:46.259515 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:46.259476    5304 patch_prober.go:29] interesting pod/e2e-inf-a-28f621-openshift-default-7dd8c5c789-jvmsw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" start-of-body=
Nov 02 15:30:46.259728 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:46.259545    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-28f621-openshift-default-7dd8c5c789-jvmsw" podUID="e4bb2a0e-a2da-44e1-ac19-ee296cfff71c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused"
Nov 02 15:30:47.264156 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:47.261578    5304 patch_prober.go:29] interesting pod/e2e-inf-a-28f621-openshift-default-7dd8c5c789-jvmsw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" start-of-body=
Nov 02 15:30:47.264156 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:47.261640    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-28f621-openshift-default-7dd8c5c789-jvmsw" podUID="e4bb2a0e-a2da-44e1-ac19-ee296cfff71c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused"
Nov 02 15:30:48.073040 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:48.073012    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-cp5ms"
Nov 02 15:30:48.192296 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:48.192265    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11a65ba1-0a55-4478-b043-637c0a9d6d64-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11a65ba1-0a55-4478-b043-637c0a9d6d64-istio-envoy\") pod \"11a65ba1-0a55-4478-b043-637c0a9d6d64\" (UID: \"11a65ba1-0a55-4478-b043-637c0a9d6d64\") "
Nov 02 15:30:48.192469 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:48.192348    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/11a65ba1-0a55-4478-b043-637c0a9d6d64-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11a65ba1-0a55-4478-b043-637c0a9d6d64-istio-podinfo\") pod \"11a65ba1-0a55-4478-b043-637c0a9d6d64\" (UID: \"11a65ba1-0a55-4478-b043-637c0a9d6d64\") "
Nov 02 15:30:48.192469 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:48.192406    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/11a65ba1-0a55-4478-b043-637c0a9d6d64-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11a65ba1-0a55-4478-b043-637c0a9d6d64-istiod-ca-cert\") pod \"11a65ba1-0a55-4478-b043-637c0a9d6d64\" (UID: \"11a65ba1-0a55-4478-b043-637c0a9d6d64\") "
Nov 02 15:30:48.192633 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:48.192505    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11a65ba1-0a55-4478-b043-637c0a9d6d64-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11a65ba1-0a55-4478-b043-637c0a9d6d64-credential-socket\") pod \"11a65ba1-0a55-4478-b043-637c0a9d6d64\" (UID: \"11a65ba1-0a55-4478-b043-637c0a9d6d64\") "
Nov 02 15:30:48.192633 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:48.192544    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11a65ba1-0a55-4478-b043-637c0a9d6d64-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11a65ba1-0a55-4478-b043-637c0a9d6d64-workload-socket\") pod \"11a65ba1-0a55-4478-b043-637c0a9d6d64\" (UID: \"11a65ba1-0a55-4478-b043-637c0a9d6d64\") "
Nov 02 15:30:48.192633 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:48.192585    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11a65ba1-0a55-4478-b043-637c0a9d6d64-kube-api-access-wskwp\" (UniqueName: \"kubernetes.io/projected/11a65ba1-0a55-4478-b043-637c0a9d6d64-kube-api-access-wskwp\") pod \"11a65ba1-0a55-4478-b043-637c0a9d6d64\" (UID: \"11a65ba1-0a55-4478-b043-637c0a9d6d64\") "
Nov 02 15:30:48.192800 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:48.192652    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11a65ba1-0a55-4478-b043-637c0a9d6d64-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11a65ba1-0a55-4478-b043-637c0a9d6d64-workload-certs\") pod \"11a65ba1-0a55-4478-b043-637c0a9d6d64\" (UID: \"11a65ba1-0a55-4478-b043-637c0a9d6d64\") "
Nov 02 15:30:48.192866 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:48.192824    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11a65ba1-0a55-4478-b043-637c0a9d6d64-credential-socket" pod "11a65ba1-0a55-4478-b043-637c0a9d6d64" (UID: "11a65ba1-0a55-4478-b043-637c0a9d6d64"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:30:48.192866 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:48.192841    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11a65ba1-0a55-4478-b043-637c0a9d6d64-istio-token\" (UniqueName: \"kubernetes.io/projected/11a65ba1-0a55-4478-b043-637c0a9d6d64-istio-token\") pod \"11a65ba1-0a55-4478-b043-637c0a9d6d64\" (UID: \"11a65ba1-0a55-4478-b043-637c0a9d6d64\") "
Nov 02 15:30:48.192978 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:48.192950    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11a65ba1-0a55-4478-b043-637c0a9d6d64-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11a65ba1-0a55-4478-b043-637c0a9d6d64-istio-data\") pod \"11a65ba1-0a55-4478-b043-637c0a9d6d64\" (UID: \"11a65ba1-0a55-4478-b043-637c0a9d6d64\") "
Nov 02 15:30:48.193092 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:48.193065    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11a65ba1-0a55-4478-b043-637c0a9d6d64-istiod-ca-cert" pod "11a65ba1-0a55-4478-b043-637c0a9d6d64" (UID: "11a65ba1-0a55-4478-b043-637c0a9d6d64"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:30:48.193339 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:48.193318    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11a65ba1-0a55-4478-b043-637c0a9d6d64-workload-certs" pod "11a65ba1-0a55-4478-b043-637c0a9d6d64" (UID: "11a65ba1-0a55-4478-b043-637c0a9d6d64"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:30:48.193629 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:48.193613    5304 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11a65ba1-0a55-4478-b043-637c0a9d6d64-workload-certs\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:30:48.193629 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:48.193630    5304 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11a65ba1-0a55-4478-b043-637c0a9d6d64-istiod-ca-cert\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:30:48.193772 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:48.193640    5304 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11a65ba1-0a55-4478-b043-637c0a9d6d64-credential-socket\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:30:48.193832 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:48.193796    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11a65ba1-0a55-4478-b043-637c0a9d6d64-istio-data" pod "11a65ba1-0a55-4478-b043-637c0a9d6d64" (UID: "11a65ba1-0a55-4478-b043-637c0a9d6d64"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:30:48.194172 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:48.194146    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11a65ba1-0a55-4478-b043-637c0a9d6d64-workload-socket" pod "11a65ba1-0a55-4478-b043-637c0a9d6d64" (UID: "11a65ba1-0a55-4478-b043-637c0a9d6d64"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:30:48.194871 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:48.194848    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11a65ba1-0a55-4478-b043-637c0a9d6d64-istio-envoy" pod "11a65ba1-0a55-4478-b043-637c0a9d6d64" (UID: "11a65ba1-0a55-4478-b043-637c0a9d6d64"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:30:48.195884 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:48.195856    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/11a65ba1-0a55-4478-b043-637c0a9d6d64-istio-podinfo" pod "11a65ba1-0a55-4478-b043-637c0a9d6d64" (UID: "11a65ba1-0a55-4478-b043-637c0a9d6d64"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 15:30:48.197168 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:48.196100    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11a65ba1-0a55-4478-b043-637c0a9d6d64-istio-token" pod "11a65ba1-0a55-4478-b043-637c0a9d6d64" (UID: "11a65ba1-0a55-4478-b043-637c0a9d6d64"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:30:48.197168 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:48.196229    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11a65ba1-0a55-4478-b043-637c0a9d6d64-kube-api-access-wskwp" pod "11a65ba1-0a55-4478-b043-637c0a9d6d64" (UID: "11a65ba1-0a55-4478-b043-637c0a9d6d64"). InnerVolumeSpecName "kube-api-access-wskwp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:30:48.260245 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:48.260213    5304 patch_prober.go:29] interesting pod/e2e-inf-a-28f621-openshift-default-7dd8c5c789-jvmsw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" start-of-body=
Nov 02 15:30:48.260431 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:48.260267    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-28f621-openshift-default-7dd8c5c789-jvmsw" podUID="e4bb2a0e-a2da-44e1-ac19-ee296cfff71c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused"
Nov 02 15:30:48.294908 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:48.294819    5304 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11a65ba1-0a55-4478-b043-637c0a9d6d64-istio-envoy\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:30:48.294908 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:48.294847    5304 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11a65ba1-0a55-4478-b043-637c0a9d6d64-istio-podinfo\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:30:48.294908 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:48.294857    5304 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11a65ba1-0a55-4478-b043-637c0a9d6d64-workload-socket\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:30:48.294908 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:48.294867    5304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wskwp\" (UniqueName: \"kubernetes.io/projected/11a65ba1-0a55-4478-b043-637c0a9d6d64-kube-api-access-wskwp\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:30:48.294908 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:48.294877    5304 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11a65ba1-0a55-4478-b043-637c0a9d6d64-istio-token\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:30:48.294908 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:48.294886    5304 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11a65ba1-0a55-4478-b043-637c0a9d6d64-istio-data\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:30:48.624235 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:48.624196    5304 generic.go:356] "Generic (PLEG): container finished" podID="11a65ba1-0a55-4478-b043-637c0a9d6d64" containerID="f6c95516a630ebb1446e90f41aa47193036cdcdc56b49b12fa665ec5574f07af" exitCode=0
Nov 02 15:30:48.624456 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:48.624267    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-cp5ms"
Nov 02 15:30:48.624456 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:48.624281    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-cp5ms" event={"ID":"11a65ba1-0a55-4478-b043-637c0a9d6d64","Type":"ContainerDied","Data":"f6c95516a630ebb1446e90f41aa47193036cdcdc56b49b12fa665ec5574f07af"}
Nov 02 15:30:48.624456 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:48.624388    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-cp5ms" event={"ID":"11a65ba1-0a55-4478-b043-637c0a9d6d64","Type":"ContainerDied","Data":"25c68a804ac50dc74ee65fde529de4a6579b9bc0b46de19a466accbee2bfef04"}
Nov 02 15:30:48.624456 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:48.624420    5304 scope.go:122] "RemoveContainer" containerID="f6c95516a630ebb1446e90f41aa47193036cdcdc56b49b12fa665ec5574f07af"
Nov 02 15:30:48.652316 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:48.652285    5304 scope.go:122] "RemoveContainer" containerID="66e57b8107829805af57f0382cb5ffde3b62067aae95273892bb44947fb1304d"
Nov 02 15:30:48.666046 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:48.666020    5304 scope.go:122] "RemoveContainer" containerID="f6c95516a630ebb1446e90f41aa47193036cdcdc56b49b12fa665ec5574f07af"
Nov 02 15:30:48.666418 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:30:48.666391    5304 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f6c95516a630ebb1446e90f41aa47193036cdcdc56b49b12fa665ec5574f07af\": container with ID starting with f6c95516a630ebb1446e90f41aa47193036cdcdc56b49b12fa665ec5574f07af not found: ID does not exist" containerID="f6c95516a630ebb1446e90f41aa47193036cdcdc56b49b12fa665ec5574f07af"
Nov 02 15:30:48.666511 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:48.666430    5304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6c95516a630ebb1446e90f41aa47193036cdcdc56b49b12fa665ec5574f07af"} err="failed to get container status \"f6c95516a630ebb1446e90f41aa47193036cdcdc56b49b12fa665ec5574f07af\": rpc error: code = NotFound desc = could not find container \"f6c95516a630ebb1446e90f41aa47193036cdcdc56b49b12fa665ec5574f07af\": container with ID starting with f6c95516a630ebb1446e90f41aa47193036cdcdc56b49b12fa665ec5574f07af not found: ID does not exist"
Nov 02 15:30:48.666511 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:48.666451    5304 scope.go:122] "RemoveContainer" containerID="66e57b8107829805af57f0382cb5ffde3b62067aae95273892bb44947fb1304d"
Nov 02 15:30:48.666765 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:30:48.666736    5304 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"66e57b8107829805af57f0382cb5ffde3b62067aae95273892bb44947fb1304d\": container with ID starting with 66e57b8107829805af57f0382cb5ffde3b62067aae95273892bb44947fb1304d not found: ID does not exist" containerID="66e57b8107829805af57f0382cb5ffde3b62067aae95273892bb44947fb1304d"
Nov 02 15:30:48.666816 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:48.666775    5304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66e57b8107829805af57f0382cb5ffde3b62067aae95273892bb44947fb1304d"} err="failed to get container status \"66e57b8107829805af57f0382cb5ffde3b62067aae95273892bb44947fb1304d\": rpc error: code = NotFound desc = could not find container \"66e57b8107829805af57f0382cb5ffde3b62067aae95273892bb44947fb1304d\": container with ID starting with 66e57b8107829805af57f0382cb5ffde3b62067aae95273892bb44947fb1304d not found: ID does not exist"
Nov 02 15:30:48.685255 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:48.685226    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-cp5ms"]
Nov 02 15:30:48.691211 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:48.691175    5304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-cp5ms"]
Nov 02 15:30:49.265406 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:49.265374    5304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-28f621-openshift-default-7dd8c5c789-jvmsw"
Nov 02 15:30:49.266369 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:49.266345    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-28f621-openshift-default-7dd8c5c789-jvmsw"
Nov 02 15:30:50.120732 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:30:50.120702    5304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="11a65ba1-0a55-4478-b043-637c0a9d6d64" path="/var/lib/kubelet/pods/11a65ba1-0a55-4478-b043-637c0a9d6d64/volumes"
Nov 02 15:30:54.799866 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:30:54.799819    5304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k6fjk" containerName="extractor"
Nov 02 15:31:10.122871 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:31:10.122837    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-f975ed/test-model-f975ed-kserve-54cff7ccbc-9w5br"]
Nov 02 15:31:22.931866 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:31:22.931822    5304 generic.go:356] "Generic (PLEG): container finished" podID="783e5ed2-e421-4d98-87e2-6b57a5662b66" containerID="b76a218e8ab6e917ec5fcb6945b04f34c456fa3c5c5a1ed0b17b71a32a20b8c8" exitCode=0
Nov 02 15:31:22.932367 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:31:22.931892    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-b955c5-openshift-default-9fd99b497-tpwvk" event={"ID":"783e5ed2-e421-4d98-87e2-6b57a5662b66","Type":"ContainerDied","Data":"b76a218e8ab6e917ec5fcb6945b04f34c456fa3c5c5a1ed0b17b71a32a20b8c8"}
Nov 02 15:31:22.932416 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:31:22.932387    5304 scope.go:122] "RemoveContainer" containerID="b76a218e8ab6e917ec5fcb6945b04f34c456fa3c5c5a1ed0b17b71a32a20b8c8"
Nov 02 15:31:22.934487 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:31:22.934458    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:31:22.934572 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:31:22.934530    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:31:22.934572 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:31:22.934561    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:31:23.963108 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:31:23.963074    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-b955c5-openshift-default-9fd99b497-tpwvk" event={"ID":"783e5ed2-e421-4d98-87e2-6b57a5662b66","Type":"ContainerStarted","Data":"bbb27e8a8c3a112958382317700b3ddaaa5ed7a1515d2dc3af164ca7c82eba22"}
Nov 02 15:31:24.802009 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:31:24.801968    5304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k6fjk" containerName="extractor"
Nov 02 15:31:24.845532 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:31:24.845494    5304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w6-b955c5-openshift-default-9fd99b497-tpwvk"
Nov 02 15:31:24.855335 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:31:24.855293    5304 patch_prober.go:29] interesting pod/e2e-shared-a-w6-b955c5-openshift-default-9fd99b497-tpwvk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" start-of-body=
Nov 02 15:31:24.855527 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:31:24.855358    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-b955c5-openshift-default-9fd99b497-tpwvk" podUID="783e5ed2-e421-4d98-87e2-6b57a5662b66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused"
Nov 02 15:31:25.152510 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:31:25.152434    5304 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-f975ed/test-model-f975ed-kserve-54cff7ccbc-9w5br" podUID="02bfbeb4-6e08-4839-b1e3-822a84cbb9c9" containerName="main" containerID="cri-o://c23ac83b97b4b70e65e6fcae93bb5b61cbf6d3ea7111cde9b99c4534cb0a5888" gracePeriod=45
Nov 02 15:31:25.482943 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:31:25.482918    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-f975ed/test-model-f975ed-kserve-54cff7ccbc-9w5br"
Nov 02 15:31:25.598667 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:31:25.598633    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02bfbeb4-6e08-4839-b1e3-822a84cbb9c9-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/02bfbeb4-6e08-4839-b1e3-822a84cbb9c9-model-cache\") pod \"02bfbeb4-6e08-4839-b1e3-822a84cbb9c9\" (UID: \"02bfbeb4-6e08-4839-b1e3-822a84cbb9c9\") "
Nov 02 15:31:25.598869 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:31:25.598692    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02bfbeb4-6e08-4839-b1e3-822a84cbb9c9-dshm\" (UniqueName: \"kubernetes.io/empty-dir/02bfbeb4-6e08-4839-b1e3-822a84cbb9c9-dshm\") pod \"02bfbeb4-6e08-4839-b1e3-822a84cbb9c9\" (UID: \"02bfbeb4-6e08-4839-b1e3-822a84cbb9c9\") "
Nov 02 15:31:25.598869 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:31:25.598728    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/02bfbeb4-6e08-4839-b1e3-822a84cbb9c9-kube-api-access-rmgjl\" (UniqueName: \"kubernetes.io/projected/02bfbeb4-6e08-4839-b1e3-822a84cbb9c9-kube-api-access-rmgjl\") pod \"02bfbeb4-6e08-4839-b1e3-822a84cbb9c9\" (UID: \"02bfbeb4-6e08-4839-b1e3-822a84cbb9c9\") "
Nov 02 15:31:25.599006 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:31:25.598884    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/02bfbeb4-6e08-4839-b1e3-822a84cbb9c9-tls-certs\" (UniqueName: \"kubernetes.io/secret/02bfbeb4-6e08-4839-b1e3-822a84cbb9c9-tls-certs\") pod \"02bfbeb4-6e08-4839-b1e3-822a84cbb9c9\" (UID: \"02bfbeb4-6e08-4839-b1e3-822a84cbb9c9\") "
Nov 02 15:31:25.599006 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:31:25.598950    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02bfbeb4-6e08-4839-b1e3-822a84cbb9c9-model-cache" pod "02bfbeb4-6e08-4839-b1e3-822a84cbb9c9" (UID: "02bfbeb4-6e08-4839-b1e3-822a84cbb9c9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:31:25.599119 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:31:25.599005    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02bfbeb4-6e08-4839-b1e3-822a84cbb9c9-home\" (UniqueName: \"kubernetes.io/empty-dir/02bfbeb4-6e08-4839-b1e3-822a84cbb9c9-home\") pod \"02bfbeb4-6e08-4839-b1e3-822a84cbb9c9\" (UID: \"02bfbeb4-6e08-4839-b1e3-822a84cbb9c9\") "
Nov 02 15:31:25.599119 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:31:25.599064    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02bfbeb4-6e08-4839-b1e3-822a84cbb9c9-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/02bfbeb4-6e08-4839-b1e3-822a84cbb9c9-tmp-dir\") pod \"02bfbeb4-6e08-4839-b1e3-822a84cbb9c9\" (UID: \"02bfbeb4-6e08-4839-b1e3-822a84cbb9c9\") "
Nov 02 15:31:25.599347 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:31:25.599282    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02bfbeb4-6e08-4839-b1e3-822a84cbb9c9-home" pod "02bfbeb4-6e08-4839-b1e3-822a84cbb9c9" (UID: "02bfbeb4-6e08-4839-b1e3-822a84cbb9c9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:31:25.599532 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:31:25.599361    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02bfbeb4-6e08-4839-b1e3-822a84cbb9c9-tmp-dir" pod "02bfbeb4-6e08-4839-b1e3-822a84cbb9c9" (UID: "02bfbeb4-6e08-4839-b1e3-822a84cbb9c9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:31:25.599819 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:31:25.599800    5304 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/02bfbeb4-6e08-4839-b1e3-822a84cbb9c9-tmp-dir\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:31:25.599885 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:31:25.599825    5304 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/02bfbeb4-6e08-4839-b1e3-822a84cbb9c9-model-cache\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:31:25.599885 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:31:25.599840    5304 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/02bfbeb4-6e08-4839-b1e3-822a84cbb9c9-home\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:31:25.601350 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:31:25.601329    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02bfbeb4-6e08-4839-b1e3-822a84cbb9c9-dshm" pod "02bfbeb4-6e08-4839-b1e3-822a84cbb9c9" (UID: "02bfbeb4-6e08-4839-b1e3-822a84cbb9c9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:31:25.601488 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:31:25.601472    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/02bfbeb4-6e08-4839-b1e3-822a84cbb9c9-tls-certs" pod "02bfbeb4-6e08-4839-b1e3-822a84cbb9c9" (UID: "02bfbeb4-6e08-4839-b1e3-822a84cbb9c9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:31:25.601592 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:31:25.601577    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02bfbeb4-6e08-4839-b1e3-822a84cbb9c9-kube-api-access-rmgjl" pod "02bfbeb4-6e08-4839-b1e3-822a84cbb9c9" (UID: "02bfbeb4-6e08-4839-b1e3-822a84cbb9c9"). InnerVolumeSpecName "kube-api-access-rmgjl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:31:25.701088 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:31:25.701052    5304 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/02bfbeb4-6e08-4839-b1e3-822a84cbb9c9-dshm\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:31:25.701088 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:31:25.701084    5304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rmgjl\" (UniqueName: \"kubernetes.io/projected/02bfbeb4-6e08-4839-b1e3-822a84cbb9c9-kube-api-access-rmgjl\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:31:25.701299 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:31:25.701096    5304 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/02bfbeb4-6e08-4839-b1e3-822a84cbb9c9-tls-certs\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:31:25.855315 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:31:25.855274    5304 patch_prober.go:29] interesting pod/e2e-shared-a-w6-b955c5-openshift-default-9fd99b497-tpwvk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" start-of-body=
Nov 02 15:31:25.855515 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:31:25.855353    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-b955c5-openshift-default-9fd99b497-tpwvk" podUID="783e5ed2-e421-4d98-87e2-6b57a5662b66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused"
Nov 02 15:31:25.990057 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:31:25.990015    5304 generic.go:356] "Generic (PLEG): container finished" podID="02bfbeb4-6e08-4839-b1e3-822a84cbb9c9" containerID="c23ac83b97b4b70e65e6fcae93bb5b61cbf6d3ea7111cde9b99c4534cb0a5888" exitCode=0
Nov 02 15:31:25.990255 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:31:25.990069    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-f975ed/test-model-f975ed-kserve-54cff7ccbc-9w5br"
Nov 02 15:31:25.990255 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:31:25.990112    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-f975ed/test-model-f975ed-kserve-54cff7ccbc-9w5br" event={"ID":"02bfbeb4-6e08-4839-b1e3-822a84cbb9c9","Type":"ContainerDied","Data":"c23ac83b97b4b70e65e6fcae93bb5b61cbf6d3ea7111cde9b99c4534cb0a5888"}
Nov 02 15:31:25.990255 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:31:25.990178    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-f975ed/test-model-f975ed-kserve-54cff7ccbc-9w5br" event={"ID":"02bfbeb4-6e08-4839-b1e3-822a84cbb9c9","Type":"ContainerDied","Data":"2651182a9cbb5721bb4421244c6a155fd26be71bf9dde0fc7d6bb7ba393f4bfa"}
Nov 02 15:31:25.990255 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:31:25.990196    5304 scope.go:122] "RemoveContainer" containerID="c23ac83b97b4b70e65e6fcae93bb5b61cbf6d3ea7111cde9b99c4534cb0a5888"
Nov 02 15:31:26.006280 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:31:26.006257    5304 scope.go:122] "RemoveContainer" containerID="c23ac83b97b4b70e65e6fcae93bb5b61cbf6d3ea7111cde9b99c4534cb0a5888"
Nov 02 15:31:26.006628 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:31:26.006601    5304 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c23ac83b97b4b70e65e6fcae93bb5b61cbf6d3ea7111cde9b99c4534cb0a5888\": container with ID starting with c23ac83b97b4b70e65e6fcae93bb5b61cbf6d3ea7111cde9b99c4534cb0a5888 not found: ID does not exist" containerID="c23ac83b97b4b70e65e6fcae93bb5b61cbf6d3ea7111cde9b99c4534cb0a5888"
Nov 02 15:31:26.006685 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:31:26.006646    5304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c23ac83b97b4b70e65e6fcae93bb5b61cbf6d3ea7111cde9b99c4534cb0a5888"} err="failed to get container status \"c23ac83b97b4b70e65e6fcae93bb5b61cbf6d3ea7111cde9b99c4534cb0a5888\": rpc error: code = NotFound desc = could not find container \"c23ac83b97b4b70e65e6fcae93bb5b61cbf6d3ea7111cde9b99c4534cb0a5888\": container with ID starting with c23ac83b97b4b70e65e6fcae93bb5b61cbf6d3ea7111cde9b99c4534cb0a5888 not found: ID does not exist"
Nov 02 15:31:26.064996 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:31:26.064965    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-f975ed/test-model-f975ed-kserve-54cff7ccbc-9w5br"]
Nov 02 15:31:26.071823 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:31:26.071794    5304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-f975ed/test-model-f975ed-kserve-54cff7ccbc-9w5br"]
Nov 02 15:31:26.119388 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:31:26.119311    5304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="02bfbeb4-6e08-4839-b1e3-822a84cbb9c9" path="/var/lib/kubelet/pods/02bfbeb4-6e08-4839-b1e3-822a84cbb9c9/volumes"
Nov 02 15:31:26.852151 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:31:26.847582    5304 patch_prober.go:29] interesting pod/e2e-shared-a-w6-b955c5-openshift-default-9fd99b497-tpwvk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" start-of-body=
Nov 02 15:31:26.852151 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:31:26.847669    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-b955c5-openshift-default-9fd99b497-tpwvk" podUID="783e5ed2-e421-4d98-87e2-6b57a5662b66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused"
Nov 02 15:31:27.725525 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:31:27.725489    5304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w6-b55da1-openshift-default-787ff9f87c-cmxhv"]
Nov 02 15:31:27.726348 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:31:27.726322    5304 cpu_manager.go:414] "containerMap: removing container" podUID="11a65ba1-0a55-4478-b043-637c0a9d6d64" containerName="istio-proxy"
Nov 02 15:31:27.726348 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:31:27.726344    5304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11a65ba1-0a55-4478-b043-637c0a9d6d64" containerName="istio-proxy"
Nov 02 15:31:27.726543 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:31:27.726357    5304 cpu_manager.go:414] "containerMap: removing container" podUID="33921120-0f00-4584-9e1d-d486c475bd10" containerName="cleanup"
Nov 02 15:31:27.726543 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:31:27.726363    5304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33921120-0f00-4584-9e1d-d486c475bd10" containerName="cleanup"
Nov 02 15:31:27.726543 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:31:27.726376    5304 cpu_manager.go:414] "containerMap: removing container" podUID="5341d918-4b2b-4c8d-8898-0e764f23c230" containerName="cleanup"
Nov 02 15:31:27.726543 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:31:27.726384    5304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5341d918-4b2b-4c8d-8898-0e764f23c230" containerName="cleanup"
Nov 02 15:31:27.726543 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:31:27.726393    5304 cpu_manager.go:414] "containerMap: removing container" podUID="b2b513e5-c608-45e2-8dc6-d798bf902d83" containerName="cleanup"
Nov 02 15:31:27.726543 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:31:27.726398    5304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2b513e5-c608-45e2-8dc6-d798bf902d83" containerName="cleanup"
Nov 02 15:31:27.726543 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:31:27.726404    5304 cpu_manager.go:414] "containerMap: removing container" podUID="8f70c555-72cb-453b-8d47-8bdbe39222e7" containerName="cleanup"
Nov 02 15:31:27.726543 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:31:27.726410    5304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f70c555-72cb-453b-8d47-8bdbe39222e7" containerName="cleanup"
Nov 02 15:31:27.726543 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:31:27.726416    5304 cpu_manager.go:414] "containerMap: removing container" podUID="c0293ead-8a17-4860-8691-12a4bf2d0ed6" containerName="cleanup"
Nov 02 15:31:27.726543 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:31:27.726420    5304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0293ead-8a17-4860-8691-12a4bf2d0ed6" containerName="cleanup"
Nov 02 15:31:27.726543 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:31:27.726428    5304 cpu_manager.go:414] "containerMap: removing container" podUID="d6984b63-9bb9-4081-a3ba-51be6260e1ea" containerName="cleanup"
Nov 02 15:31:27.726543 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:31:27.726433    5304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6984b63-9bb9-4081-a3ba-51be6260e1ea" containerName="cleanup"
Nov 02 15:31:27.726543 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:31:27.726442    5304 cpu_manager.go:414] "containerMap: removing container" podUID="02bfbeb4-6e08-4839-b1e3-822a84cbb9c9" containerName="main"
Nov 02 15:31:27.726543 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:31:27.726450    5304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="02bfbeb4-6e08-4839-b1e3-822a84cbb9c9" containerName="main"
Nov 02 15:31:27.726543 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:31:27.726463    5304 cpu_manager.go:414] "containerMap: removing container" podUID="b2b513e5-c608-45e2-8dc6-d798bf902d83" containerName="cleanup"
Nov 02 15:31:27.726543 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:31:27.726470    5304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2b513e5-c608-45e2-8dc6-d798bf902d83" containerName="cleanup"
Nov 02 15:31:27.726543 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:31:27.726484    5304 cpu_manager.go:414] "containerMap: removing container" podUID="8b61992b-ffa1-4606-bf49-9efccefbc460" containerName="cleanup"
Nov 02 15:31:27.726543 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:31:27.726491    5304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b61992b-ffa1-4606-bf49-9efccefbc460" containerName="cleanup"
Nov 02 15:31:27.726543 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:31:27.726508    5304 cpu_manager.go:414] "containerMap: removing container" podUID="33921120-0f00-4584-9e1d-d486c475bd10" containerName="cleanup"
Nov 02 15:31:27.726543 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:31:27.726514    5304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33921120-0f00-4584-9e1d-d486c475bd10" containerName="cleanup"
Nov 02 15:31:27.726543 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:31:27.726533    5304 cpu_manager.go:414] "containerMap: removing container" podUID="c0293ead-8a17-4860-8691-12a4bf2d0ed6" containerName="cleanup"
Nov 02 15:31:27.726543 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:31:27.726541    5304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0293ead-8a17-4860-8691-12a4bf2d0ed6" containerName="cleanup"
Nov 02 15:31:27.726543 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:31:27.726549    5304 cpu_manager.go:414] "containerMap: removing container" podUID="875ff34a-5486-4647-bbc6-bbd0df2633e6" containerName="cleanup"
Nov 02 15:31:27.727641 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:31:27.726556    5304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="875ff34a-5486-4647-bbc6-bbd0df2633e6" containerName="cleanup"
Nov 02 15:31:27.727641 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:31:27.726570    5304 cpu_manager.go:414] "containerMap: removing container" podUID="3b856b33-22ae-40c4-b3a5-8d90c81a16de" containerName="cleanup"
Nov 02 15:31:27.727641 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:31:27.726578    5304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b856b33-22ae-40c4-b3a5-8d90c81a16de" containerName="cleanup"
Nov 02 15:31:27.727641 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:31:27.726588    5304 cpu_manager.go:414] "containerMap: removing container" podUID="fecd4670-5e5f-477b-9f3c-689d6bdb79e4" containerName="cleanup"
Nov 02 15:31:27.727641 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:31:27.726594    5304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fecd4670-5e5f-477b-9f3c-689d6bdb79e4" containerName="cleanup"
Nov 02 15:31:27.727641 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:31:27.726616    5304 cpu_manager.go:414] "containerMap: removing container" podUID="3b856b33-22ae-40c4-b3a5-8d90c81a16de" containerName="cleanup"
Nov 02 15:31:27.727641 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:31:27.726624    5304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b856b33-22ae-40c4-b3a5-8d90c81a16de" containerName="cleanup"
Nov 02 15:31:27.727641 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:31:27.726635    5304 cpu_manager.go:414] "containerMap: removing container" podUID="8f70c555-72cb-453b-8d47-8bdbe39222e7" containerName="cleanup"
Nov 02 15:31:27.727641 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:31:27.726642    5304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f70c555-72cb-453b-8d47-8bdbe39222e7" containerName="cleanup"
Nov 02 15:31:27.727641 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:31:27.726651    5304 cpu_manager.go:414] "containerMap: removing container" podUID="3b856b33-22ae-40c4-b3a5-8d90c81a16de" containerName="cleanup"
Nov 02 15:31:27.727641 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:31:27.726658    5304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b856b33-22ae-40c4-b3a5-8d90c81a16de" containerName="cleanup"
Nov 02 15:31:27.727641 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:31:27.726670    5304 cpu_manager.go:414] "containerMap: removing container" podUID="5341d918-4b2b-4c8d-8898-0e764f23c230" containerName="cleanup"
Nov 02 15:31:27.727641 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:31:27.726677    5304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5341d918-4b2b-4c8d-8898-0e764f23c230" containerName="cleanup"
Nov 02 15:31:27.727641 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:31:27.726689    5304 cpu_manager.go:414] "containerMap: removing container" podUID="b2b513e5-c608-45e2-8dc6-d798bf902d83" containerName="cleanup"
Nov 02 15:31:27.727641 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:31:27.726696    5304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2b513e5-c608-45e2-8dc6-d798bf902d83" containerName="cleanup"
Nov 02 15:31:27.727641 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:31:27.726710    5304 cpu_manager.go:414] "containerMap: removing container" podUID="fecd4670-5e5f-477b-9f3c-689d6bdb79e4" containerName="cleanup"
Nov 02 15:31:27.727641 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:31:27.726716    5304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fecd4670-5e5f-477b-9f3c-689d6bdb79e4" containerName="cleanup"
Nov 02 15:31:27.727641 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:31:27.726727    5304 cpu_manager.go:414] "containerMap: removing container" podUID="d6984b63-9bb9-4081-a3ba-51be6260e1ea" containerName="cleanup"
Nov 02 15:31:27.727641 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:31:27.726734    5304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6984b63-9bb9-4081-a3ba-51be6260e1ea" containerName="cleanup"
Nov 02 15:31:27.727641 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:31:27.726756    5304 cpu_manager.go:414] "containerMap: removing container" podUID="fecd4670-5e5f-477b-9f3c-689d6bdb79e4" containerName="cleanup"
Nov 02 15:31:27.727641 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:31:27.726762    5304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fecd4670-5e5f-477b-9f3c-689d6bdb79e4" containerName="cleanup"
Nov 02 15:31:27.727641 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:31:27.726788    5304 cpu_manager.go:414] "containerMap: removing container" podUID="8b61992b-ffa1-4606-bf49-9efccefbc460" containerName="cleanup"
Nov 02 15:31:27.727641 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:31:27.726795    5304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b61992b-ffa1-4606-bf49-9efccefbc460" containerName="cleanup"
Nov 02 15:31:27.727641 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:31:27.726805    5304 cpu_manager.go:414] "containerMap: removing container" podUID="875ff34a-5486-4647-bbc6-bbd0df2633e6" containerName="cleanup"
Nov 02 15:31:27.727641 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:31:27.726811    5304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="875ff34a-5486-4647-bbc6-bbd0df2633e6" containerName="cleanup"
Nov 02 15:31:27.727641 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:31:27.726951    5304 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0293ead-8a17-4860-8691-12a4bf2d0ed6" containerName="cleanup"
Nov 02 15:31:27.727641 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:31:27.726960    5304 memory_manager.go:365] "RemoveStaleState removing state" podUID="875ff34a-5486-4647-bbc6-bbd0df2633e6" containerName="cleanup"
Nov 02 15:31:27.727641 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:31:27.726966    5304 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b856b33-22ae-40c4-b3a5-8d90c81a16de" containerName="cleanup"
Nov 02 15:31:27.727641 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:31:27.726973    5304 memory_manager.go:365] "RemoveStaleState removing state" podUID="fecd4670-5e5f-477b-9f3c-689d6bdb79e4" containerName="cleanup"
Nov 02 15:31:27.727641 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:31:27.726981    5304 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b856b33-22ae-40c4-b3a5-8d90c81a16de" containerName="cleanup"
Nov 02 15:31:27.727641 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:31:27.726986    5304 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f70c555-72cb-453b-8d47-8bdbe39222e7" containerName="cleanup"
Nov 02 15:31:27.727641 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:31:27.726994    5304 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b856b33-22ae-40c4-b3a5-8d90c81a16de" containerName="cleanup"
Nov 02 15:31:27.727641 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:31:27.727004    5304 memory_manager.go:365] "RemoveStaleState removing state" podUID="5341d918-4b2b-4c8d-8898-0e764f23c230" containerName="cleanup"
Nov 02 15:31:27.727641 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:31:27.727013    5304 memory_manager.go:365] "RemoveStaleState removing state" podUID="b2b513e5-c608-45e2-8dc6-d798bf902d83" containerName="cleanup"
Nov 02 15:31:27.727641 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:31:27.727022    5304 memory_manager.go:365] "RemoveStaleState removing state" podUID="fecd4670-5e5f-477b-9f3c-689d6bdb79e4" containerName="cleanup"
Nov 02 15:31:27.727641 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:31:27.727030    5304 memory_manager.go:365] "RemoveStaleState removing state" podUID="d6984b63-9bb9-4081-a3ba-51be6260e1ea" containerName="cleanup"
Nov 02 15:31:27.727641 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:31:27.727040    5304 memory_manager.go:365] "RemoveStaleState removing state" podUID="fecd4670-5e5f-477b-9f3c-689d6bdb79e4" containerName="cleanup"
Nov 02 15:31:27.727641 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:31:27.727049    5304 memory_manager.go:365] "RemoveStaleState removing state" podUID="33921120-0f00-4584-9e1d-d486c475bd10" containerName="cleanup"
Nov 02 15:31:27.727641 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:31:27.727059    5304 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f70c555-72cb-453b-8d47-8bdbe39222e7" containerName="cleanup"
Nov 02 15:31:27.727641 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:31:27.727068    5304 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b61992b-ffa1-4606-bf49-9efccefbc460" containerName="cleanup"
Nov 02 15:31:27.727641 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:31:27.727075    5304 memory_manager.go:365] "RemoveStaleState removing state" podUID="875ff34a-5486-4647-bbc6-bbd0df2633e6" containerName="cleanup"
Nov 02 15:31:27.727641 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:31:27.727082    5304 memory_manager.go:365] "RemoveStaleState removing state" podUID="5341d918-4b2b-4c8d-8898-0e764f23c230" containerName="cleanup"
Nov 02 15:31:27.727641 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:31:27.727090    5304 memory_manager.go:365] "RemoveStaleState removing state" podUID="11a65ba1-0a55-4478-b043-637c0a9d6d64" containerName="istio-proxy"
Nov 02 15:31:27.727641 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:31:27.727098    5304 memory_manager.go:365] "RemoveStaleState removing state" podUID="33921120-0f00-4584-9e1d-d486c475bd10" containerName="cleanup"
Nov 02 15:31:27.727641 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:31:27.727106    5304 memory_manager.go:365] "RemoveStaleState removing state" podUID="5341d918-4b2b-4c8d-8898-0e764f23c230" containerName="cleanup"
Nov 02 15:31:27.727641 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:31:27.727117    5304 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0293ead-8a17-4860-8691-12a4bf2d0ed6" containerName="cleanup"
Nov 02 15:31:27.727641 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:31:27.727138    5304 memory_manager.go:365] "RemoveStaleState removing state" podUID="d6984b63-9bb9-4081-a3ba-51be6260e1ea" containerName="cleanup"
Nov 02 15:31:27.727641 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:31:27.727147    5304 memory_manager.go:365] "RemoveStaleState removing state" podUID="02bfbeb4-6e08-4839-b1e3-822a84cbb9c9" containerName="main"
Nov 02 15:31:27.727641 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:31:27.727157    5304 memory_manager.go:365] "RemoveStaleState removing state" podUID="b2b513e5-c608-45e2-8dc6-d798bf902d83" containerName="cleanup"
Nov 02 15:31:27.727641 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:31:27.727162    5304 memory_manager.go:365] "RemoveStaleState removing state" podUID="11a65ba1-0a55-4478-b043-637c0a9d6d64" containerName="istio-proxy"
Nov 02 15:31:27.727641 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:31:27.727169    5304 memory_manager.go:365] "RemoveStaleState removing state" podUID="33921120-0f00-4584-9e1d-d486c475bd10" containerName="cleanup"
Nov 02 15:31:27.727641 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:31:27.727178    5304 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b61992b-ffa1-4606-bf49-9efccefbc460" containerName="cleanup"
Nov 02 15:31:27.730903 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:31:27.730878    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-b55da1-openshift-default-787ff9f87c-cmxhv"
Nov 02 15:31:27.770159 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:31:27.763533    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-b55da1-openshift-default-787ff9f87c-cmxhv"]
Nov 02 15:31:27.786146 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:31:27.775040    5304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w6-b55da1-openshift-default-dockercfg-rpg55\""
Nov 02 15:31:27.874156 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:31:27.873628    5304 patch_prober.go:29] interesting pod/e2e-shared-a-w6-b955c5-openshift-default-9fd99b497-tpwvk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" start-of-body=
Nov 02 15:31:27.874156 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:31:27.873703    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-b955c5-openshift-default-9fd99b497-tpwvk" podUID="783e5ed2-e421-4d98-87e2-6b57a5662b66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused"
Nov 02 15:31:27.882169 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:31:27.878832    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67da2b58-a0f0-44f6-9258-b418a21e73bc-istio-envoy\") pod \"e2e-shared-b-w6-b55da1-openshift-default-787ff9f87c-cmxhv\" (UID: \"67da2b58-a0f0-44f6-9258-b418a21e73bc\") " pod="openshift-ingress/e2e-shared-b-w6-b55da1-openshift-default-787ff9f87c-cmxhv"
Nov 02 15:31:27.882169 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:31:27.878921    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67da2b58-a0f0-44f6-9258-b418a21e73bc-credential-socket\") pod \"e2e-shared-b-w6-b55da1-openshift-default-787ff9f87c-cmxhv\" (UID: \"67da2b58-a0f0-44f6-9258-b418a21e73bc\") " pod="openshift-ingress/e2e-shared-b-w6-b55da1-openshift-default-787ff9f87c-cmxhv"
Nov 02 15:31:27.882169 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:31:27.879058    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sr82w\" (UniqueName: \"kubernetes.io/projected/67da2b58-a0f0-44f6-9258-b418a21e73bc-kube-api-access-sr82w\") pod \"e2e-shared-b-w6-b55da1-openshift-default-787ff9f87c-cmxhv\" (UID: \"67da2b58-a0f0-44f6-9258-b418a21e73bc\") " pod="openshift-ingress/e2e-shared-b-w6-b55da1-openshift-default-787ff9f87c-cmxhv"
Nov 02 15:31:27.882169 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:31:27.879158    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67da2b58-a0f0-44f6-9258-b418a21e73bc-istio-podinfo\") pod \"e2e-shared-b-w6-b55da1-openshift-default-787ff9f87c-cmxhv\" (UID: \"67da2b58-a0f0-44f6-9258-b418a21e73bc\") " pod="openshift-ingress/e2e-shared-b-w6-b55da1-openshift-default-787ff9f87c-cmxhv"
Nov 02 15:31:27.882169 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:31:27.879206    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67da2b58-a0f0-44f6-9258-b418a21e73bc-workload-socket\") pod \"e2e-shared-b-w6-b55da1-openshift-default-787ff9f87c-cmxhv\" (UID: \"67da2b58-a0f0-44f6-9258-b418a21e73bc\") " pod="openshift-ingress/e2e-shared-b-w6-b55da1-openshift-default-787ff9f87c-cmxhv"
Nov 02 15:31:27.882169 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:31:27.879240    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67da2b58-a0f0-44f6-9258-b418a21e73bc-workload-certs\") pod \"e2e-shared-b-w6-b55da1-openshift-default-787ff9f87c-cmxhv\" (UID: \"67da2b58-a0f0-44f6-9258-b418a21e73bc\") " pod="openshift-ingress/e2e-shared-b-w6-b55da1-openshift-default-787ff9f87c-cmxhv"
Nov 02 15:31:27.882169 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:31:27.879324    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67da2b58-a0f0-44f6-9258-b418a21e73bc-istio-data\") pod \"e2e-shared-b-w6-b55da1-openshift-default-787ff9f87c-cmxhv\" (UID: \"67da2b58-a0f0-44f6-9258-b418a21e73bc\") " pod="openshift-ingress/e2e-shared-b-w6-b55da1-openshift-default-787ff9f87c-cmxhv"
Nov 02 15:31:27.882169 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:31:27.879350    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67da2b58-a0f0-44f6-9258-b418a21e73bc-istio-token\") pod \"e2e-shared-b-w6-b55da1-openshift-default-787ff9f87c-cmxhv\" (UID: \"67da2b58-a0f0-44f6-9258-b418a21e73bc\") " pod="openshift-ingress/e2e-shared-b-w6-b55da1-openshift-default-787ff9f87c-cmxhv"
Nov 02 15:31:27.882169 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:31:27.879413    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67da2b58-a0f0-44f6-9258-b418a21e73bc-istiod-ca-cert\") pod \"e2e-shared-b-w6-b55da1-openshift-default-787ff9f87c-cmxhv\" (UID: \"67da2b58-a0f0-44f6-9258-b418a21e73bc\") " pod="openshift-ingress/e2e-shared-b-w6-b55da1-openshift-default-787ff9f87c-cmxhv"
Nov 02 15:31:28.020793 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:31:27.997143    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67da2b58-a0f0-44f6-9258-b418a21e73bc-istio-token\") pod \"e2e-shared-b-w6-b55da1-openshift-default-787ff9f87c-cmxhv\" (UID: \"67da2b58-a0f0-44f6-9258-b418a21e73bc\") " pod="openshift-ingress/e2e-shared-b-w6-b55da1-openshift-default-787ff9f87c-cmxhv"
Nov 02 15:31:28.020793 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:31:27.997237    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67da2b58-a0f0-44f6-9258-b418a21e73bc-istiod-ca-cert\") pod \"e2e-shared-b-w6-b55da1-openshift-default-787ff9f87c-cmxhv\" (UID: \"67da2b58-a0f0-44f6-9258-b418a21e73bc\") " pod="openshift-ingress/e2e-shared-b-w6-b55da1-openshift-default-787ff9f87c-cmxhv"
Nov 02 15:31:28.020793 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:31:27.997280    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67da2b58-a0f0-44f6-9258-b418a21e73bc-istio-envoy\") pod \"e2e-shared-b-w6-b55da1-openshift-default-787ff9f87c-cmxhv\" (UID: \"67da2b58-a0f0-44f6-9258-b418a21e73bc\") " pod="openshift-ingress/e2e-shared-b-w6-b55da1-openshift-default-787ff9f87c-cmxhv"
Nov 02 15:31:28.020793 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:31:27.997365    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67da2b58-a0f0-44f6-9258-b418a21e73bc-credential-socket\") pod \"e2e-shared-b-w6-b55da1-openshift-default-787ff9f87c-cmxhv\" (UID: \"67da2b58-a0f0-44f6-9258-b418a21e73bc\") " pod="openshift-ingress/e2e-shared-b-w6-b55da1-openshift-default-787ff9f87c-cmxhv"
Nov 02 15:31:28.020793 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:31:27.997577    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sr82w\" (UniqueName: \"kubernetes.io/projected/67da2b58-a0f0-44f6-9258-b418a21e73bc-kube-api-access-sr82w\") pod \"e2e-shared-b-w6-b55da1-openshift-default-787ff9f87c-cmxhv\" (UID: \"67da2b58-a0f0-44f6-9258-b418a21e73bc\") " pod="openshift-ingress/e2e-shared-b-w6-b55da1-openshift-default-787ff9f87c-cmxhv"
Nov 02 15:31:28.020793 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:31:27.997653    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67da2b58-a0f0-44f6-9258-b418a21e73bc-istio-podinfo\") pod \"e2e-shared-b-w6-b55da1-openshift-default-787ff9f87c-cmxhv\" (UID: \"67da2b58-a0f0-44f6-9258-b418a21e73bc\") " pod="openshift-ingress/e2e-shared-b-w6-b55da1-openshift-default-787ff9f87c-cmxhv"
Nov 02 15:31:28.020793 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:31:27.997707    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67da2b58-a0f0-44f6-9258-b418a21e73bc-workload-socket\") pod \"e2e-shared-b-w6-b55da1-openshift-default-787ff9f87c-cmxhv\" (UID: \"67da2b58-a0f0-44f6-9258-b418a21e73bc\") " pod="openshift-ingress/e2e-shared-b-w6-b55da1-openshift-default-787ff9f87c-cmxhv"
Nov 02 15:31:28.020793 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:31:27.997747    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67da2b58-a0f0-44f6-9258-b418a21e73bc-workload-certs\") pod \"e2e-shared-b-w6-b55da1-openshift-default-787ff9f87c-cmxhv\" (UID: \"67da2b58-a0f0-44f6-9258-b418a21e73bc\") " pod="openshift-ingress/e2e-shared-b-w6-b55da1-openshift-default-787ff9f87c-cmxhv"
Nov 02 15:31:28.020793 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:31:27.997822    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67da2b58-a0f0-44f6-9258-b418a21e73bc-istio-data\") pod \"e2e-shared-b-w6-b55da1-openshift-default-787ff9f87c-cmxhv\" (UID: \"67da2b58-a0f0-44f6-9258-b418a21e73bc\") " pod="openshift-ingress/e2e-shared-b-w6-b55da1-openshift-default-787ff9f87c-cmxhv"
Nov 02 15:31:28.020793 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:31:27.998140    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67da2b58-a0f0-44f6-9258-b418a21e73bc-istiod-ca-cert\") pod \"e2e-shared-b-w6-b55da1-openshift-default-787ff9f87c-cmxhv\" (UID: \"67da2b58-a0f0-44f6-9258-b418a21e73bc\") " pod="openshift-ingress/e2e-shared-b-w6-b55da1-openshift-default-787ff9f87c-cmxhv"
Nov 02 15:31:28.020793 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:31:27.998450    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67da2b58-a0f0-44f6-9258-b418a21e73bc-istio-data\") pod \"e2e-shared-b-w6-b55da1-openshift-default-787ff9f87c-cmxhv\" (UID: \"67da2b58-a0f0-44f6-9258-b418a21e73bc\") " pod="openshift-ingress/e2e-shared-b-w6-b55da1-openshift-default-787ff9f87c-cmxhv"
Nov 02 15:31:28.021554 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:31:28.021506    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67da2b58-a0f0-44f6-9258-b418a21e73bc-istio-envoy\") pod \"e2e-shared-b-w6-b55da1-openshift-default-787ff9f87c-cmxhv\" (UID: \"67da2b58-a0f0-44f6-9258-b418a21e73bc\") " pod="openshift-ingress/e2e-shared-b-w6-b55da1-openshift-default-787ff9f87c-cmxhv"
Nov 02 15:31:28.033507 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:31:28.032989    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67da2b58-a0f0-44f6-9258-b418a21e73bc-istio-podinfo\") pod \"e2e-shared-b-w6-b55da1-openshift-default-787ff9f87c-cmxhv\" (UID: \"67da2b58-a0f0-44f6-9258-b418a21e73bc\") " pod="openshift-ingress/e2e-shared-b-w6-b55da1-openshift-default-787ff9f87c-cmxhv"
Nov 02 15:31:28.071156 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:31:28.054555    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67da2b58-a0f0-44f6-9258-b418a21e73bc-workload-certs\") pod \"e2e-shared-b-w6-b55da1-openshift-default-787ff9f87c-cmxhv\" (UID: \"67da2b58-a0f0-44f6-9258-b418a21e73bc\") " pod="openshift-ingress/e2e-shared-b-w6-b55da1-openshift-default-787ff9f87c-cmxhv"
Nov 02 15:31:28.071156 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:31:28.054681    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67da2b58-a0f0-44f6-9258-b418a21e73bc-workload-socket\") pod \"e2e-shared-b-w6-b55da1-openshift-default-787ff9f87c-cmxhv\" (UID: \"67da2b58-a0f0-44f6-9258-b418a21e73bc\") " pod="openshift-ingress/e2e-shared-b-w6-b55da1-openshift-default-787ff9f87c-cmxhv"
Nov 02 15:31:28.071156 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:31:28.058737    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67da2b58-a0f0-44f6-9258-b418a21e73bc-credential-socket\") pod \"e2e-shared-b-w6-b55da1-openshift-default-787ff9f87c-cmxhv\" (UID: \"67da2b58-a0f0-44f6-9258-b418a21e73bc\") " pod="openshift-ingress/e2e-shared-b-w6-b55da1-openshift-default-787ff9f87c-cmxhv"
Nov 02 15:31:28.092149 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:31:28.089815    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67da2b58-a0f0-44f6-9258-b418a21e73bc-istio-token\") pod \"e2e-shared-b-w6-b55da1-openshift-default-787ff9f87c-cmxhv\" (UID: \"67da2b58-a0f0-44f6-9258-b418a21e73bc\") " pod="openshift-ingress/e2e-shared-b-w6-b55da1-openshift-default-787ff9f87c-cmxhv"
Nov 02 15:31:28.109151 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:31:28.105474    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sr82w\" (UniqueName: \"kubernetes.io/projected/67da2b58-a0f0-44f6-9258-b418a21e73bc-kube-api-access-sr82w\") pod \"e2e-shared-b-w6-b55da1-openshift-default-787ff9f87c-cmxhv\" (UID: \"67da2b58-a0f0-44f6-9258-b418a21e73bc\") " pod="openshift-ingress/e2e-shared-b-w6-b55da1-openshift-default-787ff9f87c-cmxhv"
Nov 02 15:31:28.387893 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:31:28.387854    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-b55da1-openshift-default-787ff9f87c-cmxhv"
Nov 02 15:31:28.686793 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:31:28.686709    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w6-b955c5-openshift-default-9fd99b497-tpwvk"
Nov 02 15:31:28.760402 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:31:28.760366    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-b55da1-openshift-default-787ff9f87c-cmxhv"]
Nov 02 15:31:28.760819 ip-10-0-135-138 kubenswrapper[5304]: W1102 15:31:28.760793    5304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67da2b58_a0f0_44f6_9258_b418a21e73bc.slice/crio-a4c4ccc2c238a149676473f8f48d78673a61198c242464fd45d351410b2c1127 WatchSource:0}: Error finding container a4c4ccc2c238a149676473f8f48d78673a61198c242464fd45d351410b2c1127: Status 404 returned error can't find the container with id a4c4ccc2c238a149676473f8f48d78673a61198c242464fd45d351410b2c1127
Nov 02 15:31:28.764329 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:31:28.764286    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:31:28.764414 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:31:28.764378    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:31:28.764478 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:31:28.764421    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:31:28.842200 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:31:28.839590    5304 patch_prober.go:29] interesting pod/e2e-shared-a-w6-b955c5-openshift-default-9fd99b497-tpwvk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" start-of-body=
Nov 02 15:31:28.842200 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:31:28.839647    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-b955c5-openshift-default-9fd99b497-tpwvk" podUID="783e5ed2-e421-4d98-87e2-6b57a5662b66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused"
Nov 02 15:31:29.100198 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:31:29.100119    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-b55da1-openshift-default-787ff9f87c-cmxhv" event={"ID":"67da2b58-a0f0-44f6-9258-b418a21e73bc","Type":"ContainerStarted","Data":"aa7cb5d0093ee870a139ebc974db546ad5b647c091f54f48cddf10d8ddfc7646"}
Nov 02 15:31:29.100634 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:31:29.100223    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-b55da1-openshift-default-787ff9f87c-cmxhv" event={"ID":"67da2b58-a0f0-44f6-9258-b418a21e73bc","Type":"ContainerStarted","Data":"a4c4ccc2c238a149676473f8f48d78673a61198c242464fd45d351410b2c1127"}
Nov 02 15:31:29.147180 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:31:29.147082    5304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w6-b55da1-openshift-default-787ff9f87c-cmxhv" podStartSLOduration=2.147039868 podStartE2EDuration="2.147039868s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:31:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:31:29.14506022 +0000 UTC m=+2089.138884630" watchObservedRunningTime="2026-11-02 15:31:29.147039868 +0000 UTC m=+2089.140864279"
Nov 02 15:31:29.389161 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:31:29.389064    5304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w6-b55da1-openshift-default-787ff9f87c-cmxhv"
Nov 02 15:31:29.396944 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:31:29.396918    5304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w6-b55da1-openshift-default-787ff9f87c-cmxhv"
Nov 02 15:31:29.844560 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:31:29.844533    5304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w6-b955c5-openshift-default-9fd99b497-tpwvk"
Nov 02 15:31:29.845614 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:31:29.845587    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w6-b955c5-openshift-default-9fd99b497-tpwvk"
Nov 02 15:31:30.107172 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:31:30.107067    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w6-b55da1-openshift-default-787ff9f87c-cmxhv"
Nov 02 15:31:30.108209 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:31:30.108184    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w6-b55da1-openshift-default-787ff9f87c-cmxhv"
Nov 02 15:31:32.544869 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:31:32.544836    5304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-69ff88-openshift-default-866dfd5ff7-74bk2"]
Nov 02 15:31:32.545714 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:31:32.545690    5304 cpu_manager.go:414] "containerMap: removing container" podUID="8b61992b-ffa1-4606-bf49-9efccefbc460" containerName="cleanup"
Nov 02 15:31:32.545714 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:31:32.545713    5304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b61992b-ffa1-4606-bf49-9efccefbc460" containerName="cleanup"
Nov 02 15:31:32.545898 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:31:32.545734    5304 cpu_manager.go:414] "containerMap: removing container" podUID="33921120-0f00-4584-9e1d-d486c475bd10" containerName="cleanup"
Nov 02 15:31:32.545898 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:31:32.545740    5304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33921120-0f00-4584-9e1d-d486c475bd10" containerName="cleanup"
Nov 02 15:31:32.545898 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:31:32.545769    5304 cpu_manager.go:414] "containerMap: removing container" podUID="d6984b63-9bb9-4081-a3ba-51be6260e1ea" containerName="cleanup"
Nov 02 15:31:32.545898 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:31:32.545774    5304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6984b63-9bb9-4081-a3ba-51be6260e1ea" containerName="cleanup"
Nov 02 15:31:32.545898 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:31:32.545801    5304 cpu_manager.go:414] "containerMap: removing container" podUID="875ff34a-5486-4647-bbc6-bbd0df2633e6" containerName="cleanup"
Nov 02 15:31:32.545898 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:31:32.545807    5304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="875ff34a-5486-4647-bbc6-bbd0df2633e6" containerName="cleanup"
Nov 02 15:31:32.545898 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:31:32.545814    5304 cpu_manager.go:414] "containerMap: removing container" podUID="11a65ba1-0a55-4478-b043-637c0a9d6d64" containerName="istio-proxy"
Nov 02 15:31:32.545898 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:31:32.545821    5304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11a65ba1-0a55-4478-b043-637c0a9d6d64" containerName="istio-proxy"
Nov 02 15:31:32.545898 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:31:32.545834    5304 cpu_manager.go:414] "containerMap: removing container" podUID="5341d918-4b2b-4c8d-8898-0e764f23c230" containerName="cleanup"
Nov 02 15:31:32.545898 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:31:32.545839    5304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5341d918-4b2b-4c8d-8898-0e764f23c230" containerName="cleanup"
Nov 02 15:31:32.545898 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:31:32.545844    5304 cpu_manager.go:414] "containerMap: removing container" podUID="8f70c555-72cb-453b-8d47-8bdbe39222e7" containerName="cleanup"
Nov 02 15:31:32.545898 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:31:32.545849    5304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f70c555-72cb-453b-8d47-8bdbe39222e7" containerName="cleanup"
Nov 02 15:31:32.545898 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:31:32.545857    5304 cpu_manager.go:414] "containerMap: removing container" podUID="c0293ead-8a17-4860-8691-12a4bf2d0ed6" containerName="cleanup"
Nov 02 15:31:32.545898 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:31:32.545861    5304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0293ead-8a17-4860-8691-12a4bf2d0ed6" containerName="cleanup"
Nov 02 15:31:32.546499 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:31:32.545958    5304 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b61992b-ffa1-4606-bf49-9efccefbc460" containerName="cleanup"
Nov 02 15:31:32.546499 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:31:32.545972    5304 memory_manager.go:365] "RemoveStaleState removing state" podUID="b2b513e5-c608-45e2-8dc6-d798bf902d83" containerName="cleanup"
Nov 02 15:31:32.546499 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:31:32.545986    5304 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f70c555-72cb-453b-8d47-8bdbe39222e7" containerName="cleanup"
Nov 02 15:31:32.546499 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:31:32.545995    5304 memory_manager.go:365] "RemoveStaleState removing state" podUID="875ff34a-5486-4647-bbc6-bbd0df2633e6" containerName="cleanup"
Nov 02 15:31:32.546499 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:31:32.546003    5304 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0293ead-8a17-4860-8691-12a4bf2d0ed6" containerName="cleanup"
Nov 02 15:31:32.546499 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:31:32.546008    5304 memory_manager.go:365] "RemoveStaleState removing state" podUID="d6984b63-9bb9-4081-a3ba-51be6260e1ea" containerName="cleanup"
Nov 02 15:31:32.548931 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:31:32.548909    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-69ff88-openshift-default-866dfd5ff7-74bk2"
Nov 02 15:31:32.570414 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:31:32.570378    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-69ff88-openshift-default-866dfd5ff7-74bk2"]
Nov 02 15:31:32.578447 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:31:32.578417    5304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-69ff88-openshift-default-dockercfg-xb5gm\""
Nov 02 15:31:32.681483 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:31:32.681448    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/301b87ba-1380-4110-96cd-5e2a057cae4d-workload-socket\") pod \"e2e-inf-a-69ff88-openshift-default-866dfd5ff7-74bk2\" (UID: \"301b87ba-1380-4110-96cd-5e2a057cae4d\") " pod="openshift-ingress/e2e-inf-a-69ff88-openshift-default-866dfd5ff7-74bk2"
Nov 02 15:31:32.681697 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:31:32.681504    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/301b87ba-1380-4110-96cd-5e2a057cae4d-istio-envoy\") pod \"e2e-inf-a-69ff88-openshift-default-866dfd5ff7-74bk2\" (UID: \"301b87ba-1380-4110-96cd-5e2a057cae4d\") " pod="openshift-ingress/e2e-inf-a-69ff88-openshift-default-866dfd5ff7-74bk2"
Nov 02 15:31:32.681697 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:31:32.681661    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/301b87ba-1380-4110-96cd-5e2a057cae4d-istiod-ca-cert\") pod \"e2e-inf-a-69ff88-openshift-default-866dfd5ff7-74bk2\" (UID: \"301b87ba-1380-4110-96cd-5e2a057cae4d\") " pod="openshift-ingress/e2e-inf-a-69ff88-openshift-default-866dfd5ff7-74bk2"
Nov 02 15:31:32.681828 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:31:32.681711    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/301b87ba-1380-4110-96cd-5e2a057cae4d-workload-certs\") pod \"e2e-inf-a-69ff88-openshift-default-866dfd5ff7-74bk2\" (UID: \"301b87ba-1380-4110-96cd-5e2a057cae4d\") " pod="openshift-ingress/e2e-inf-a-69ff88-openshift-default-866dfd5ff7-74bk2"
Nov 02 15:31:32.681828 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:31:32.681783    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9fbwt\" (UniqueName: \"kubernetes.io/projected/301b87ba-1380-4110-96cd-5e2a057cae4d-kube-api-access-9fbwt\") pod \"e2e-inf-a-69ff88-openshift-default-866dfd5ff7-74bk2\" (UID: \"301b87ba-1380-4110-96cd-5e2a057cae4d\") " pod="openshift-ingress/e2e-inf-a-69ff88-openshift-default-866dfd5ff7-74bk2"
Nov 02 15:31:32.681942 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:31:32.681873    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/301b87ba-1380-4110-96cd-5e2a057cae4d-istio-data\") pod \"e2e-inf-a-69ff88-openshift-default-866dfd5ff7-74bk2\" (UID: \"301b87ba-1380-4110-96cd-5e2a057cae4d\") " pod="openshift-ingress/e2e-inf-a-69ff88-openshift-default-866dfd5ff7-74bk2"
Nov 02 15:31:32.682084 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:31:32.682067    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/301b87ba-1380-4110-96cd-5e2a057cae4d-istio-token\") pod \"e2e-inf-a-69ff88-openshift-default-866dfd5ff7-74bk2\" (UID: \"301b87ba-1380-4110-96cd-5e2a057cae4d\") " pod="openshift-ingress/e2e-inf-a-69ff88-openshift-default-866dfd5ff7-74bk2"
Nov 02 15:31:32.682180 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:31:32.682118    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/301b87ba-1380-4110-96cd-5e2a057cae4d-istio-podinfo\") pod \"e2e-inf-a-69ff88-openshift-default-866dfd5ff7-74bk2\" (UID: \"301b87ba-1380-4110-96cd-5e2a057cae4d\") " pod="openshift-ingress/e2e-inf-a-69ff88-openshift-default-866dfd5ff7-74bk2"
Nov 02 15:31:32.682242 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:31:32.682220    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/301b87ba-1380-4110-96cd-5e2a057cae4d-credential-socket\") pod \"e2e-inf-a-69ff88-openshift-default-866dfd5ff7-74bk2\" (UID: \"301b87ba-1380-4110-96cd-5e2a057cae4d\") " pod="openshift-ingress/e2e-inf-a-69ff88-openshift-default-866dfd5ff7-74bk2"
Nov 02 15:31:32.783535 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:31:32.783501    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/301b87ba-1380-4110-96cd-5e2a057cae4d-istio-podinfo\") pod \"e2e-inf-a-69ff88-openshift-default-866dfd5ff7-74bk2\" (UID: \"301b87ba-1380-4110-96cd-5e2a057cae4d\") " pod="openshift-ingress/e2e-inf-a-69ff88-openshift-default-866dfd5ff7-74bk2"
Nov 02 15:31:32.783745 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:31:32.783551    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/301b87ba-1380-4110-96cd-5e2a057cae4d-credential-socket\") pod \"e2e-inf-a-69ff88-openshift-default-866dfd5ff7-74bk2\" (UID: \"301b87ba-1380-4110-96cd-5e2a057cae4d\") " pod="openshift-ingress/e2e-inf-a-69ff88-openshift-default-866dfd5ff7-74bk2"
Nov 02 15:31:32.783745 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:31:32.783583    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/301b87ba-1380-4110-96cd-5e2a057cae4d-workload-socket\") pod \"e2e-inf-a-69ff88-openshift-default-866dfd5ff7-74bk2\" (UID: \"301b87ba-1380-4110-96cd-5e2a057cae4d\") " pod="openshift-ingress/e2e-inf-a-69ff88-openshift-default-866dfd5ff7-74bk2"
Nov 02 15:31:32.783745 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:31:32.783625    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/301b87ba-1380-4110-96cd-5e2a057cae4d-istio-envoy\") pod \"e2e-inf-a-69ff88-openshift-default-866dfd5ff7-74bk2\" (UID: \"301b87ba-1380-4110-96cd-5e2a057cae4d\") " pod="openshift-ingress/e2e-inf-a-69ff88-openshift-default-866dfd5ff7-74bk2"
Nov 02 15:31:32.783745 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:31:32.783699    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/301b87ba-1380-4110-96cd-5e2a057cae4d-istiod-ca-cert\") pod \"e2e-inf-a-69ff88-openshift-default-866dfd5ff7-74bk2\" (UID: \"301b87ba-1380-4110-96cd-5e2a057cae4d\") " pod="openshift-ingress/e2e-inf-a-69ff88-openshift-default-866dfd5ff7-74bk2"
Nov 02 15:31:32.783745 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:31:32.783738    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/301b87ba-1380-4110-96cd-5e2a057cae4d-workload-certs\") pod \"e2e-inf-a-69ff88-openshift-default-866dfd5ff7-74bk2\" (UID: \"301b87ba-1380-4110-96cd-5e2a057cae4d\") " pod="openshift-ingress/e2e-inf-a-69ff88-openshift-default-866dfd5ff7-74bk2"
Nov 02 15:31:32.784047 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:31:32.783775    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9fbwt\" (UniqueName: \"kubernetes.io/projected/301b87ba-1380-4110-96cd-5e2a057cae4d-kube-api-access-9fbwt\") pod \"e2e-inf-a-69ff88-openshift-default-866dfd5ff7-74bk2\" (UID: \"301b87ba-1380-4110-96cd-5e2a057cae4d\") " pod="openshift-ingress/e2e-inf-a-69ff88-openshift-default-866dfd5ff7-74bk2"
Nov 02 15:31:32.784047 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:31:32.783844    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/301b87ba-1380-4110-96cd-5e2a057cae4d-istio-data\") pod \"e2e-inf-a-69ff88-openshift-default-866dfd5ff7-74bk2\" (UID: \"301b87ba-1380-4110-96cd-5e2a057cae4d\") " pod="openshift-ingress/e2e-inf-a-69ff88-openshift-default-866dfd5ff7-74bk2"
Nov 02 15:31:32.784047 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:31:32.784019    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/301b87ba-1380-4110-96cd-5e2a057cae4d-credential-socket\") pod \"e2e-inf-a-69ff88-openshift-default-866dfd5ff7-74bk2\" (UID: \"301b87ba-1380-4110-96cd-5e2a057cae4d\") " pod="openshift-ingress/e2e-inf-a-69ff88-openshift-default-866dfd5ff7-74bk2"
Nov 02 15:31:32.784047 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:31:32.784039    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/301b87ba-1380-4110-96cd-5e2a057cae4d-istio-token\") pod \"e2e-inf-a-69ff88-openshift-default-866dfd5ff7-74bk2\" (UID: \"301b87ba-1380-4110-96cd-5e2a057cae4d\") " pod="openshift-ingress/e2e-inf-a-69ff88-openshift-default-866dfd5ff7-74bk2"
Nov 02 15:31:32.784286 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:31:32.784046    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/301b87ba-1380-4110-96cd-5e2a057cae4d-workload-socket\") pod \"e2e-inf-a-69ff88-openshift-default-866dfd5ff7-74bk2\" (UID: \"301b87ba-1380-4110-96cd-5e2a057cae4d\") " pod="openshift-ingress/e2e-inf-a-69ff88-openshift-default-866dfd5ff7-74bk2"
Nov 02 15:31:32.784342 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:31:32.784308    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/301b87ba-1380-4110-96cd-5e2a057cae4d-workload-certs\") pod \"e2e-inf-a-69ff88-openshift-default-866dfd5ff7-74bk2\" (UID: \"301b87ba-1380-4110-96cd-5e2a057cae4d\") " pod="openshift-ingress/e2e-inf-a-69ff88-openshift-default-866dfd5ff7-74bk2"
Nov 02 15:31:32.784519 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:31:32.784487    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/301b87ba-1380-4110-96cd-5e2a057cae4d-istio-data\") pod \"e2e-inf-a-69ff88-openshift-default-866dfd5ff7-74bk2\" (UID: \"301b87ba-1380-4110-96cd-5e2a057cae4d\") " pod="openshift-ingress/e2e-inf-a-69ff88-openshift-default-866dfd5ff7-74bk2"
Nov 02 15:31:32.784598 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:31:32.784528    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/301b87ba-1380-4110-96cd-5e2a057cae4d-istiod-ca-cert\") pod \"e2e-inf-a-69ff88-openshift-default-866dfd5ff7-74bk2\" (UID: \"301b87ba-1380-4110-96cd-5e2a057cae4d\") " pod="openshift-ingress/e2e-inf-a-69ff88-openshift-default-866dfd5ff7-74bk2"
Nov 02 15:31:32.786222 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:31:32.786198    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/301b87ba-1380-4110-96cd-5e2a057cae4d-istio-envoy\") pod \"e2e-inf-a-69ff88-openshift-default-866dfd5ff7-74bk2\" (UID: \"301b87ba-1380-4110-96cd-5e2a057cae4d\") " pod="openshift-ingress/e2e-inf-a-69ff88-openshift-default-866dfd5ff7-74bk2"
Nov 02 15:31:32.786412 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:31:32.786393    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/301b87ba-1380-4110-96cd-5e2a057cae4d-istio-podinfo\") pod \"e2e-inf-a-69ff88-openshift-default-866dfd5ff7-74bk2\" (UID: \"301b87ba-1380-4110-96cd-5e2a057cae4d\") " pod="openshift-ingress/e2e-inf-a-69ff88-openshift-default-866dfd5ff7-74bk2"
Nov 02 15:31:32.809659 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:31:32.809604    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/301b87ba-1380-4110-96cd-5e2a057cae4d-istio-token\") pod \"e2e-inf-a-69ff88-openshift-default-866dfd5ff7-74bk2\" (UID: \"301b87ba-1380-4110-96cd-5e2a057cae4d\") " pod="openshift-ingress/e2e-inf-a-69ff88-openshift-default-866dfd5ff7-74bk2"
Nov 02 15:31:32.811292 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:31:32.811271    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9fbwt\" (UniqueName: \"kubernetes.io/projected/301b87ba-1380-4110-96cd-5e2a057cae4d-kube-api-access-9fbwt\") pod \"e2e-inf-a-69ff88-openshift-default-866dfd5ff7-74bk2\" (UID: \"301b87ba-1380-4110-96cd-5e2a057cae4d\") " pod="openshift-ingress/e2e-inf-a-69ff88-openshift-default-866dfd5ff7-74bk2"
Nov 02 15:31:32.864062 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:31:32.864031    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-69ff88-openshift-default-866dfd5ff7-74bk2"
Nov 02 15:31:33.188787 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:31:33.188754    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-69ff88-openshift-default-866dfd5ff7-74bk2"]
Nov 02 15:31:33.194269 ip-10-0-135-138 kubenswrapper[5304]: W1102 15:31:33.194237    5304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod301b87ba_1380_4110_96cd_5e2a057cae4d.slice/crio-c2c6f4d0f21c59b911436725df9698c09933bbf38e942cbf595faa9cbf988e35 WatchSource:0}: Error finding container c2c6f4d0f21c59b911436725df9698c09933bbf38e942cbf595faa9cbf988e35: Status 404 returned error can't find the container with id c2c6f4d0f21c59b911436725df9698c09933bbf38e942cbf595faa9cbf988e35
Nov 02 15:31:33.197532 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:31:33.197495    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:31:33.197645 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:31:33.197592    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:31:33.197711 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:31:33.197640    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:31:33.624440 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:31:33.621320    5304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-b55da1-7757f8c8bc-l8dpc"]
Nov 02 15:31:33.625731 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:31:33.625691    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-b55da1-7757f8c8bc-l8dpc"
Nov 02 15:31:33.658210 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:31:33.658180    5304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w6-b55da1-dockercfg-cbfgb\""
Nov 02 15:31:33.658541 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:31:33.658518    5304 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w6-b55da1\""
Nov 02 15:31:33.816186 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:31:33.767063    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-b55da1-7757f8c8bc-l8dpc"]
Nov 02 15:31:33.822303 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:31:33.820146    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j95fz\" (UniqueName: \"kubernetes.io/projected/1ec89e8e-8ec5-497f-b3ad-1570c6d0d3dd-kube-api-access-j95fz\") pod \"payload-processing-e2e-shared-b-w6-b55da1-7757f8c8bc-l8dpc\" (UID: \"1ec89e8e-8ec5-497f-b3ad-1570c6d0d3dd\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-b55da1-7757f8c8bc-l8dpc"
Nov 02 15:31:33.822495 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:31:33.822454    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ec89e8e-8ec5-497f-b3ad-1570c6d0d3dd-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w6-b55da1-7757f8c8bc-l8dpc\" (UID: \"1ec89e8e-8ec5-497f-b3ad-1570c6d0d3dd\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-b55da1-7757f8c8bc-l8dpc"
Nov 02 15:31:33.833033 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:31:33.833003    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-f975ed-fcd5fdf77-sqnvq"]
Nov 02 15:31:33.878562 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:31:33.878476    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-f975ed-6c55746559-gc62v"]
Nov 02 15:31:33.923924 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:31:33.923887    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j95fz\" (UniqueName: \"kubernetes.io/projected/1ec89e8e-8ec5-497f-b3ad-1570c6d0d3dd-kube-api-access-j95fz\") pod \"payload-processing-e2e-shared-b-w6-b55da1-7757f8c8bc-l8dpc\" (UID: \"1ec89e8e-8ec5-497f-b3ad-1570c6d0d3dd\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-b55da1-7757f8c8bc-l8dpc"
Nov 02 15:31:33.924147 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:31:33.924060    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ec89e8e-8ec5-497f-b3ad-1570c6d0d3dd-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w6-b55da1-7757f8c8bc-l8dpc\" (UID: \"1ec89e8e-8ec5-497f-b3ad-1570c6d0d3dd\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-b55da1-7757f8c8bc-l8dpc"
Nov 02 15:31:33.925160 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:31:33.925113    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ec89e8e-8ec5-497f-b3ad-1570c6d0d3dd-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w6-b55da1-7757f8c8bc-l8dpc\" (UID: \"1ec89e8e-8ec5-497f-b3ad-1570c6d0d3dd\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-b55da1-7757f8c8bc-l8dpc"
Nov 02 15:31:33.957188 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:31:33.956140    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j95fz\" (UniqueName: \"kubernetes.io/projected/1ec89e8e-8ec5-497f-b3ad-1570c6d0d3dd-kube-api-access-j95fz\") pod \"payload-processing-e2e-shared-b-w6-b55da1-7757f8c8bc-l8dpc\" (UID: \"1ec89e8e-8ec5-497f-b3ad-1570c6d0d3dd\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-b55da1-7757f8c8bc-l8dpc"
Nov 02 15:31:34.126685 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:31:34.126648    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-b55da1-7757f8c8bc-l8dpc"
Nov 02 15:31:34.148751 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:31:34.142298    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-69ff88-openshift-default-866dfd5ff7-74bk2" event={"ID":"301b87ba-1380-4110-96cd-5e2a057cae4d","Type":"ContainerStarted","Data":"c52095f80a6a806c607449558de0019e7eba5c9efca4153c3fcfc3a77a707a9c"}
Nov 02 15:31:34.148751 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:31:34.142371    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-69ff88-openshift-default-866dfd5ff7-74bk2" event={"ID":"301b87ba-1380-4110-96cd-5e2a057cae4d","Type":"ContainerStarted","Data":"c2c6f4d0f21c59b911436725df9698c09933bbf38e942cbf595faa9cbf988e35"}
Nov 02 15:31:34.193350 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:31:34.193296    5304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-69ff88-openshift-default-866dfd5ff7-74bk2" podStartSLOduration=2.193277755 podStartE2EDuration="2.193277755s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:31:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:31:34.192353527 +0000 UTC m=+2094.186177936" watchObservedRunningTime="2026-11-02 15:31:34.193277755 +0000 UTC m=+2094.187102165"
Nov 02 15:31:34.462005 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:31:34.461970    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-b55da1-7757f8c8bc-l8dpc"]
Nov 02 15:31:34.464115 ip-10-0-135-138 kubenswrapper[5304]: W1102 15:31:34.464088    5304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ec89e8e_8ec5_497f_b3ad_1570c6d0d3dd.slice/crio-62bd5e3dde0b4cb4588504469b7943bde27c5aa33b10f6a142099c91966798a8 WatchSource:0}: Error finding container 62bd5e3dde0b4cb4588504469b7943bde27c5aa33b10f6a142099c91966798a8: Status 404 returned error can't find the container with id 62bd5e3dde0b4cb4588504469b7943bde27c5aa33b10f6a142099c91966798a8
Nov 02 15:31:34.870676 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:31:34.868704    5304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-69ff88-openshift-default-866dfd5ff7-74bk2"
Nov 02 15:31:34.871328 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:31:34.871300    5304 patch_prober.go:29] interesting pod/e2e-inf-a-69ff88-openshift-default-866dfd5ff7-74bk2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" start-of-body=
Nov 02 15:31:34.871488 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:31:34.871464    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-69ff88-openshift-default-866dfd5ff7-74bk2" podUID="301b87ba-1380-4110-96cd-5e2a057cae4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused"
Nov 02 15:31:35.162514 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:31:35.162408    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-b55da1-7757f8c8bc-l8dpc" event={"ID":"1ec89e8e-8ec5-497f-b3ad-1570c6d0d3dd","Type":"ContainerStarted","Data":"9eb97d07737b81346510341054528b102c06371546d047c73cdca7b1186e881f"}
Nov 02 15:31:35.162514 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:31:35.162475    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-b55da1-7757f8c8bc-l8dpc" event={"ID":"1ec89e8e-8ec5-497f-b3ad-1570c6d0d3dd","Type":"ContainerStarted","Data":"62bd5e3dde0b4cb4588504469b7943bde27c5aa33b10f6a142099c91966798a8"}
Nov 02 15:31:35.162758 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:31:35.162650    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-b55da1-7757f8c8bc-l8dpc"
Nov 02 15:31:35.205428 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:31:35.205373    5304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-b55da1-7757f8c8bc-l8dpc" podStartSLOduration=2.205356487 podStartE2EDuration="2.205356487s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:31:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:31:35.20231386 +0000 UTC m=+2095.196138276" watchObservedRunningTime="2026-11-02 15:31:35.205356487 +0000 UTC m=+2095.199180896"
Nov 02 15:31:35.864724 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:31:35.864690    5304 patch_prober.go:29] interesting pod/e2e-inf-a-69ff88-openshift-default-866dfd5ff7-74bk2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" start-of-body=
Nov 02 15:31:35.864909 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:31:35.864742    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-69ff88-openshift-default-866dfd5ff7-74bk2" podUID="301b87ba-1380-4110-96cd-5e2a057cae4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused"
Nov 02 15:31:36.864706 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:31:36.864670    5304 patch_prober.go:29] interesting pod/e2e-inf-a-69ff88-openshift-default-866dfd5ff7-74bk2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" start-of-body=
Nov 02 15:31:36.865102 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:31:36.864734    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-69ff88-openshift-default-866dfd5ff7-74bk2" podUID="301b87ba-1380-4110-96cd-5e2a057cae4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused"
Nov 02 15:31:37.864987 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:31:37.864946    5304 patch_prober.go:29] interesting pod/e2e-inf-a-69ff88-openshift-default-866dfd5ff7-74bk2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" start-of-body=
Nov 02 15:31:37.865412 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:31:37.865001    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-69ff88-openshift-default-866dfd5ff7-74bk2" podUID="301b87ba-1380-4110-96cd-5e2a057cae4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused"
Nov 02 15:31:38.855261 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:31:38.855208    5304 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-f975ed-fcd5fdf77-sqnvq" podUID="f2c78d1b-1497-4fe3-bfcc-f8eecb41f621" containerName="payload-processing" containerID="cri-o://3d1c1209af7649e3620435383e5e47b91bf5ee9cd6ed0b6b1a49c1d705ad700d" gracePeriod=25
Nov 02 15:31:38.865046 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:31:38.865017    5304 patch_prober.go:29] interesting pod/e2e-inf-a-69ff88-openshift-default-866dfd5ff7-74bk2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" start-of-body=
Nov 02 15:31:38.865429 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:31:38.865068    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-69ff88-openshift-default-866dfd5ff7-74bk2" podUID="301b87ba-1380-4110-96cd-5e2a057cae4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused"
Nov 02 15:31:38.879015 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:31:38.878987    5304 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f975ed-6c55746559-gc62v" podUID="1f4f76a0-e6ea-4515-9543-e7720ae0a2c6" containerName="payload-pre-processing" containerID="cri-o://cc1105ad8af35c1a3d1da216f47955f5b45f10cc33915131f8a3a646393847d8" gracePeriod=25
Nov 02 15:31:39.191032 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:31:39.191001    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-f975ed-fcd5fdf77-sqnvq"
Nov 02 15:31:39.219604 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:31:39.219567    5304 generic.go:356] "Generic (PLEG): container finished" podID="f2c78d1b-1497-4fe3-bfcc-f8eecb41f621" containerID="3d1c1209af7649e3620435383e5e47b91bf5ee9cd6ed0b6b1a49c1d705ad700d" exitCode=0
Nov 02 15:31:39.219769 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:31:39.219680    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-f975ed-fcd5fdf77-sqnvq"
Nov 02 15:31:39.219769 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:31:39.219720    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-f975ed-fcd5fdf77-sqnvq" event={"ID":"f2c78d1b-1497-4fe3-bfcc-f8eecb41f621","Type":"ContainerDied","Data":"3d1c1209af7649e3620435383e5e47b91bf5ee9cd6ed0b6b1a49c1d705ad700d"}
Nov 02 15:31:39.219864 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:31:39.219788    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-f975ed-fcd5fdf77-sqnvq" event={"ID":"f2c78d1b-1497-4fe3-bfcc-f8eecb41f621","Type":"ContainerDied","Data":"4439f12cb99278388cfd819be2f17e51d780167a8106ad19bb64985d216db92a"}
Nov 02 15:31:39.219864 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:31:39.219811    5304 scope.go:122] "RemoveContainer" containerID="3d1c1209af7649e3620435383e5e47b91bf5ee9cd6ed0b6b1a49c1d705ad700d"
Nov 02 15:31:39.260390 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:31:39.260325    5304 scope.go:122] "RemoveContainer" containerID="3d1c1209af7649e3620435383e5e47b91bf5ee9cd6ed0b6b1a49c1d705ad700d"
Nov 02 15:31:39.261622 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:31:39.261543    5304 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d1c1209af7649e3620435383e5e47b91bf5ee9cd6ed0b6b1a49c1d705ad700d\": container with ID starting with 3d1c1209af7649e3620435383e5e47b91bf5ee9cd6ed0b6b1a49c1d705ad700d not found: ID does not exist" containerID="3d1c1209af7649e3620435383e5e47b91bf5ee9cd6ed0b6b1a49c1d705ad700d"
Nov 02 15:31:39.261622 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:31:39.261590    5304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d1c1209af7649e3620435383e5e47b91bf5ee9cd6ed0b6b1a49c1d705ad700d"} err="failed to get container status \"3d1c1209af7649e3620435383e5e47b91bf5ee9cd6ed0b6b1a49c1d705ad700d\": rpc error: code = NotFound desc = could not find container \"3d1c1209af7649e3620435383e5e47b91bf5ee9cd6ed0b6b1a49c1d705ad700d\": container with ID starting with 3d1c1209af7649e3620435383e5e47b91bf5ee9cd6ed0b6b1a49c1d705ad700d not found: ID does not exist"
Nov 02 15:31:39.280982 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:31:39.280349    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f2c78d1b-1497-4fe3-bfcc-f8eecb41f621-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2c78d1b-1497-4fe3-bfcc-f8eecb41f621-plugins-config-volume\") pod \"f2c78d1b-1497-4fe3-bfcc-f8eecb41f621\" (UID: \"f2c78d1b-1497-4fe3-bfcc-f8eecb41f621\") "
Nov 02 15:31:39.280982 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:31:39.280587    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2c78d1b-1497-4fe3-bfcc-f8eecb41f621-kube-api-access-qkzhb\" (UniqueName: \"kubernetes.io/projected/f2c78d1b-1497-4fe3-bfcc-f8eecb41f621-kube-api-access-qkzhb\") pod \"f2c78d1b-1497-4fe3-bfcc-f8eecb41f621\" (UID: \"f2c78d1b-1497-4fe3-bfcc-f8eecb41f621\") "
Nov 02 15:31:39.281341 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:31:39.281303    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2c78d1b-1497-4fe3-bfcc-f8eecb41f621-plugins-config-volume" pod "f2c78d1b-1497-4fe3-bfcc-f8eecb41f621" (UID: "f2c78d1b-1497-4fe3-bfcc-f8eecb41f621"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:31:39.281714 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:31:39.281685    5304 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2c78d1b-1497-4fe3-bfcc-f8eecb41f621-plugins-config-volume\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:31:39.283668 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:31:39.283646    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2c78d1b-1497-4fe3-bfcc-f8eecb41f621-kube-api-access-qkzhb" pod "f2c78d1b-1497-4fe3-bfcc-f8eecb41f621" (UID: "f2c78d1b-1497-4fe3-bfcc-f8eecb41f621"). InnerVolumeSpecName "kube-api-access-qkzhb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:31:39.392975 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:31:39.391954    5304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qkzhb\" (UniqueName: \"kubernetes.io/projected/f2c78d1b-1497-4fe3-bfcc-f8eecb41f621-kube-api-access-qkzhb\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:31:39.580713 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:31:39.580679    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-f975ed-fcd5fdf77-sqnvq"]
Nov 02 15:31:39.591757 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:31:39.591728    5304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-f975ed-fcd5fdf77-sqnvq"]
Nov 02 15:31:39.865449 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:31:39.865415    5304 patch_prober.go:29] interesting pod/e2e-inf-a-69ff88-openshift-default-866dfd5ff7-74bk2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" start-of-body=
Nov 02 15:31:39.865871 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:31:39.865478    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-69ff88-openshift-default-866dfd5ff7-74bk2" podUID="301b87ba-1380-4110-96cd-5e2a057cae4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused"
Nov 02 15:31:40.120143 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:31:40.120044    5304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f2c78d1b-1497-4fe3-bfcc-f8eecb41f621" path="/var/lib/kubelet/pods/f2c78d1b-1497-4fe3-bfcc-f8eecb41f621/volumes"
Nov 02 15:31:40.870468 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:31:40.870432    5304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-69ff88-openshift-default-866dfd5ff7-74bk2"
Nov 02 15:31:40.870850 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:31:40.870824    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-69ff88-openshift-default-866dfd5ff7-74bk2"
Nov 02 15:31:40.871501 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:31:40.871478    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-69ff88-openshift-default-866dfd5ff7-74bk2"
Nov 02 15:31:41.719311 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:31:41.719277    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f975ed-6c55746559-gc62v" podUID="1f4f76a0-e6ea-4515-9543-e7720ae0a2c6" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 15:31:46.205711 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:31:46.205677    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-b55da1-7757f8c8bc-l8dpc"
Nov 02 15:31:51.719030 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:31:51.718995    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f975ed-6c55746559-gc62v" podUID="1f4f76a0-e6ea-4515-9543-e7720ae0a2c6" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 15:31:54.340650 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:31:54.340613    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-a03e84/test-model-a03e84-kserve-554589b56f-tq4bb"]
Nov 02 15:31:54.801386 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:31:54.801301    5304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k6fjk" containerName="extractor"
Nov 02 15:31:59.164975 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:31:59.164886    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f975ed-6c55746559-gc62v"
Nov 02 15:31:59.183895 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:31:59.183859    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f4f76a0-e6ea-4515-9543-e7720ae0a2c6-kube-api-access-4bqw5\" (UniqueName: \"kubernetes.io/projected/1f4f76a0-e6ea-4515-9543-e7720ae0a2c6-kube-api-access-4bqw5\") pod \"1f4f76a0-e6ea-4515-9543-e7720ae0a2c6\" (UID: \"1f4f76a0-e6ea-4515-9543-e7720ae0a2c6\") "
Nov 02 15:31:59.184060 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:31:59.183997    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1f4f76a0-e6ea-4515-9543-e7720ae0a2c6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f4f76a0-e6ea-4515-9543-e7720ae0a2c6-plugins-config-volume\") pod \"1f4f76a0-e6ea-4515-9543-e7720ae0a2c6\" (UID: \"1f4f76a0-e6ea-4515-9543-e7720ae0a2c6\") "
Nov 02 15:31:59.184483 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:31:59.184441    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f4f76a0-e6ea-4515-9543-e7720ae0a2c6-plugins-config-volume" pod "1f4f76a0-e6ea-4515-9543-e7720ae0a2c6" (UID: "1f4f76a0-e6ea-4515-9543-e7720ae0a2c6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:31:59.185426 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:31:59.185392    5304 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f4f76a0-e6ea-4515-9543-e7720ae0a2c6-plugins-config-volume\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:31:59.186539 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:31:59.186514    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f4f76a0-e6ea-4515-9543-e7720ae0a2c6-kube-api-access-4bqw5" pod "1f4f76a0-e6ea-4515-9543-e7720ae0a2c6" (UID: "1f4f76a0-e6ea-4515-9543-e7720ae0a2c6"). InnerVolumeSpecName "kube-api-access-4bqw5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:31:59.287277 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:31:59.287238    5304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4bqw5\" (UniqueName: \"kubernetes.io/projected/1f4f76a0-e6ea-4515-9543-e7720ae0a2c6-kube-api-access-4bqw5\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:31:59.478352 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:31:59.478255    5304 generic.go:356] "Generic (PLEG): container finished" podID="1f4f76a0-e6ea-4515-9543-e7720ae0a2c6" containerID="cc1105ad8af35c1a3d1da216f47955f5b45f10cc33915131f8a3a646393847d8" exitCode=0
Nov 02 15:31:59.478943 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:31:59.478795    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f975ed-6c55746559-gc62v" event={"ID":"1f4f76a0-e6ea-4515-9543-e7720ae0a2c6","Type":"ContainerDied","Data":"cc1105ad8af35c1a3d1da216f47955f5b45f10cc33915131f8a3a646393847d8"}
Nov 02 15:31:59.479262 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:31:59.479241    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f975ed-6c55746559-gc62v" event={"ID":"1f4f76a0-e6ea-4515-9543-e7720ae0a2c6","Type":"ContainerDied","Data":"67f68acd4c10d91c7def5d3ef3456c16ea78e132ac19f4dd47bfdbac8f0798cf"}
Nov 02 15:31:59.479404 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:31:59.479390    5304 scope.go:122] "RemoveContainer" containerID="cc1105ad8af35c1a3d1da216f47955f5b45f10cc33915131f8a3a646393847d8"
Nov 02 15:31:59.479622 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:31:59.479047    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f975ed-6c55746559-gc62v"
Nov 02 15:31:59.505302 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:31:59.504692    5304 scope.go:122] "RemoveContainer" containerID="cc1105ad8af35c1a3d1da216f47955f5b45f10cc33915131f8a3a646393847d8"
Nov 02 15:31:59.505302 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:31:59.505073    5304 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc1105ad8af35c1a3d1da216f47955f5b45f10cc33915131f8a3a646393847d8\": container with ID starting with cc1105ad8af35c1a3d1da216f47955f5b45f10cc33915131f8a3a646393847d8 not found: ID does not exist" containerID="cc1105ad8af35c1a3d1da216f47955f5b45f10cc33915131f8a3a646393847d8"
Nov 02 15:31:59.505302 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:31:59.505109    5304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc1105ad8af35c1a3d1da216f47955f5b45f10cc33915131f8a3a646393847d8"} err="failed to get container status \"cc1105ad8af35c1a3d1da216f47955f5b45f10cc33915131f8a3a646393847d8\": rpc error: code = NotFound desc = could not find container \"cc1105ad8af35c1a3d1da216f47955f5b45f10cc33915131f8a3a646393847d8\": container with ID starting with cc1105ad8af35c1a3d1da216f47955f5b45f10cc33915131f8a3a646393847d8 not found: ID does not exist"
Nov 02 15:31:59.564759 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:31:59.561730    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-f975ed-6c55746559-gc62v"]
Nov 02 15:31:59.570204 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:31:59.570169    5304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-f975ed-6c55746559-gc62v"]
Nov 02 15:32:00.128729 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:00.128699    5304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1f4f76a0-e6ea-4515-9543-e7720ae0a2c6" path="/var/lib/kubelet/pods/1f4f76a0-e6ea-4515-9543-e7720ae0a2c6/volumes"
Nov 02 15:32:09.286227 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:09.286192    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-tl8dm"]
Nov 02 15:32:09.286667 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:09.286419    5304 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-tl8dm" podUID="7a09b277-1d1f-45c3-a18a-d393acfa5882" containerName="istio-proxy" containerID="cri-o://1e9030e051ac48ab0cc6e29ac000907223c70f491d6fdcb0e7bcf3c49454816d" gracePeriod=30
Nov 02 15:32:09.297784 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:09.297729    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-qm7zm"]
Nov 02 15:32:09.297972 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:09.297945    5304 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-qm7zm" podUID="4d0dd04f-b8ff-4ecd-a1fd-ec15ce4dea5a" containerName="istio-proxy" containerID="cri-o://0ee06f62c3aa25c2759c1c9f53bfefcd5eb79054b406994dd091bfc80e08aeb0" gracePeriod=30
Nov 02 15:32:09.302060 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:09.302031    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-4cc4m"]
Nov 02 15:32:09.302300 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:09.302258    5304 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-4cc4m" podUID="af975b55-20fa-459f-8659-a7a69f2d2248" containerName="istio-proxy" containerID="cri-o://8c317367ce8a8c092812e99fe4869b78b9b68f8abb841513c7e4d33790c8c4e9" gracePeriod=30
Nov 02 15:32:09.311447 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:09.311420    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-r7jjp"]
Nov 02 15:32:09.311665 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:09.311639    5304 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-r7jjp" podUID="707b5eda-0575-4b3d-b1d8-f18005ddaa07" containerName="istio-proxy" containerID="cri-o://a5494422ed014731629f8914216f2679ff9aa7d7c0532e007350dab4d603eb5c" gracePeriod=30
Nov 02 15:32:09.362358 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:09.362319    5304 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-a03e84/test-model-a03e84-kserve-554589b56f-tq4bb" podUID="917f905c-457e-401b-af26-2af1201b5d28" containerName="main" containerID="cri-o://70310ece1db88ecf64ebb7bde2c00a3726f33ade6fff8e2464b1a2e0b8056332" gracePeriod=45
Nov 02 15:32:09.557552 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:09.557513    5304 generic.go:356] "Generic (PLEG): container finished" podID="917f905c-457e-401b-af26-2af1201b5d28" containerID="70310ece1db88ecf64ebb7bde2c00a3726f33ade6fff8e2464b1a2e0b8056332" exitCode=0
Nov 02 15:32:09.557736 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:09.557565    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-a03e84/test-model-a03e84-kserve-554589b56f-tq4bb" event={"ID":"917f905c-457e-401b-af26-2af1201b5d28","Type":"ContainerDied","Data":"70310ece1db88ecf64ebb7bde2c00a3726f33ade6fff8e2464b1a2e0b8056332"}
Nov 02 15:32:09.665980 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:09.665944    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-a03e84/test-model-a03e84-kserve-554589b56f-tq4bb"
Nov 02 15:32:09.767671 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:09.767636    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/917f905c-457e-401b-af26-2af1201b5d28-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/917f905c-457e-401b-af26-2af1201b5d28-tmp-dir\") pod \"917f905c-457e-401b-af26-2af1201b5d28\" (UID: \"917f905c-457e-401b-af26-2af1201b5d28\") "
Nov 02 15:32:09.767872 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:09.767720    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/917f905c-457e-401b-af26-2af1201b5d28-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/917f905c-457e-401b-af26-2af1201b5d28-model-cache\") pod \"917f905c-457e-401b-af26-2af1201b5d28\" (UID: \"917f905c-457e-401b-af26-2af1201b5d28\") "
Nov 02 15:32:09.767872 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:09.767792    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/917f905c-457e-401b-af26-2af1201b5d28-tls-certs\" (UniqueName: \"kubernetes.io/secret/917f905c-457e-401b-af26-2af1201b5d28-tls-certs\") pod \"917f905c-457e-401b-af26-2af1201b5d28\" (UID: \"917f905c-457e-401b-af26-2af1201b5d28\") "
Nov 02 15:32:09.768000 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:09.767918    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/917f905c-457e-401b-af26-2af1201b5d28-dshm\" (UniqueName: \"kubernetes.io/empty-dir/917f905c-457e-401b-af26-2af1201b5d28-dshm\") pod \"917f905c-457e-401b-af26-2af1201b5d28\" (UID: \"917f905c-457e-401b-af26-2af1201b5d28\") "
Nov 02 15:32:09.768057 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:09.767989    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/917f905c-457e-401b-af26-2af1201b5d28-tmp-dir" pod "917f905c-457e-401b-af26-2af1201b5d28" (UID: "917f905c-457e-401b-af26-2af1201b5d28"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:32:09.768057 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:09.768018    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/917f905c-457e-401b-af26-2af1201b5d28-home\" (UniqueName: \"kubernetes.io/empty-dir/917f905c-457e-401b-af26-2af1201b5d28-home\") pod \"917f905c-457e-401b-af26-2af1201b5d28\" (UID: \"917f905c-457e-401b-af26-2af1201b5d28\") "
Nov 02 15:32:09.768204 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:09.768057    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/917f905c-457e-401b-af26-2af1201b5d28-kube-api-access-7hlhd\" (UniqueName: \"kubernetes.io/projected/917f905c-457e-401b-af26-2af1201b5d28-kube-api-access-7hlhd\") pod \"917f905c-457e-401b-af26-2af1201b5d28\" (UID: \"917f905c-457e-401b-af26-2af1201b5d28\") "
Nov 02 15:32:09.768204 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:09.768053    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/917f905c-457e-401b-af26-2af1201b5d28-model-cache" pod "917f905c-457e-401b-af26-2af1201b5d28" (UID: "917f905c-457e-401b-af26-2af1201b5d28"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:32:09.768316 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:09.768283    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/917f905c-457e-401b-af26-2af1201b5d28-home" pod "917f905c-457e-401b-af26-2af1201b5d28" (UID: "917f905c-457e-401b-af26-2af1201b5d28"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:32:09.768741 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:09.768719    5304 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/917f905c-457e-401b-af26-2af1201b5d28-home\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:32:09.768741 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:09.768739    5304 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/917f905c-457e-401b-af26-2af1201b5d28-tmp-dir\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:32:09.768910 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:09.768750    5304 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/917f905c-457e-401b-af26-2af1201b5d28-model-cache\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:32:09.773162 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:09.770241    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/917f905c-457e-401b-af26-2af1201b5d28-tls-certs" pod "917f905c-457e-401b-af26-2af1201b5d28" (UID: "917f905c-457e-401b-af26-2af1201b5d28"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:32:09.773162 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:09.770317    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/917f905c-457e-401b-af26-2af1201b5d28-dshm" pod "917f905c-457e-401b-af26-2af1201b5d28" (UID: "917f905c-457e-401b-af26-2af1201b5d28"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:32:09.773162 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:09.770518    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/917f905c-457e-401b-af26-2af1201b5d28-kube-api-access-7hlhd" pod "917f905c-457e-401b-af26-2af1201b5d28" (UID: "917f905c-457e-401b-af26-2af1201b5d28"). InnerVolumeSpecName "kube-api-access-7hlhd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:32:09.870860 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:09.870822    5304 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/917f905c-457e-401b-af26-2af1201b5d28-dshm\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:32:09.870860 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:09.870859    5304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7hlhd\" (UniqueName: \"kubernetes.io/projected/917f905c-457e-401b-af26-2af1201b5d28-kube-api-access-7hlhd\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:32:09.871142 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:09.870872    5304 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/917f905c-457e-401b-af26-2af1201b5d28-tls-certs\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:32:10.566071 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:10.565988    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-a03e84/test-model-a03e84-kserve-554589b56f-tq4bb"
Nov 02 15:32:10.566071 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:10.566009    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-a03e84/test-model-a03e84-kserve-554589b56f-tq4bb" event={"ID":"917f905c-457e-401b-af26-2af1201b5d28","Type":"ContainerDied","Data":"c712556767213cf9104ac9ade1c46d7caf28bcea3d75f8c5f7859080ba931b0a"}
Nov 02 15:32:10.566071 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:10.566064    5304 scope.go:122] "RemoveContainer" containerID="70310ece1db88ecf64ebb7bde2c00a3726f33ade6fff8e2464b1a2e0b8056332"
Nov 02 15:32:10.618517 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:10.618474    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-a03e84/test-model-a03e84-kserve-554589b56f-tq4bb"]
Nov 02 15:32:10.630499 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:10.630465    5304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-a03e84/test-model-a03e84-kserve-554589b56f-tq4bb"]
Nov 02 15:32:11.185289 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:11.185253    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-3daba5/test-model-3daba5-kserve-6f59d8687-qp658"]
Nov 02 15:32:12.119889 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:12.119858    5304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="917f905c-457e-401b-af26-2af1201b5d28" path="/var/lib/kubelet/pods/917f905c-457e-401b-af26-2af1201b5d28/volumes"
Nov 02 15:32:14.618317 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:14.618283    5304 generic.go:356] "Generic (PLEG): container finished" podID="707b5eda-0575-4b3d-b1d8-f18005ddaa07" containerID="a5494422ed014731629f8914216f2679ff9aa7d7c0532e007350dab4d603eb5c" exitCode=0
Nov 02 15:32:14.618712 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:14.618447    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-r7jjp" event={"ID":"707b5eda-0575-4b3d-b1d8-f18005ddaa07","Type":"ContainerDied","Data":"a5494422ed014731629f8914216f2679ff9aa7d7c0532e007350dab4d603eb5c"}
Nov 02 15:32:14.621657 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:14.621627    5304 generic.go:356] "Generic (PLEG): container finished" podID="7a09b277-1d1f-45c3-a18a-d393acfa5882" containerID="1e9030e051ac48ab0cc6e29ac000907223c70f491d6fdcb0e7bcf3c49454816d" exitCode=0
Nov 02 15:32:14.621802 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:14.621691    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-tl8dm" event={"ID":"7a09b277-1d1f-45c3-a18a-d393acfa5882","Type":"ContainerDied","Data":"1e9030e051ac48ab0cc6e29ac000907223c70f491d6fdcb0e7bcf3c49454816d"}
Nov 02 15:32:14.621802 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:14.621757    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-tl8dm" event={"ID":"7a09b277-1d1f-45c3-a18a-d393acfa5882","Type":"ContainerDied","Data":"306e799f8eddb68e1c13320353fa91e151833b88cb1ebd54a9429b0f21d01558"}
Nov 02 15:32:14.621802 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:14.621785    5304 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="306e799f8eddb68e1c13320353fa91e151833b88cb1ebd54a9429b0f21d01558"
Nov 02 15:32:14.624256 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:14.624234    5304 generic.go:356] "Generic (PLEG): container finished" podID="af975b55-20fa-459f-8659-a7a69f2d2248" containerID="8c317367ce8a8c092812e99fe4869b78b9b68f8abb841513c7e4d33790c8c4e9" exitCode=0
Nov 02 15:32:14.624543 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:14.624265    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-4cc4m" event={"ID":"af975b55-20fa-459f-8659-a7a69f2d2248","Type":"ContainerDied","Data":"8c317367ce8a8c092812e99fe4869b78b9b68f8abb841513c7e4d33790c8c4e9"}
Nov 02 15:32:14.626393 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:14.626373    5304 generic.go:356] "Generic (PLEG): container finished" podID="4d0dd04f-b8ff-4ecd-a1fd-ec15ce4dea5a" containerID="0ee06f62c3aa25c2759c1c9f53bfefcd5eb79054b406994dd091bfc80e08aeb0" exitCode=0
Nov 02 15:32:14.626507 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:14.626438    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-qm7zm" event={"ID":"4d0dd04f-b8ff-4ecd-a1fd-ec15ce4dea5a","Type":"ContainerDied","Data":"0ee06f62c3aa25c2759c1c9f53bfefcd5eb79054b406994dd091bfc80e08aeb0"}
Nov 02 15:32:14.644334 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:14.644308    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-tl8dm"
Nov 02 15:32:14.648876 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:14.648851    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-qm7zm"
Nov 02 15:32:14.662411 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:14.662385    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a09b277-1d1f-45c3-a18a-d393acfa5882-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a09b277-1d1f-45c3-a18a-d393acfa5882-istio-envoy\") pod \"7a09b277-1d1f-45c3-a18a-d393acfa5882\" (UID: \"7a09b277-1d1f-45c3-a18a-d393acfa5882\") "
Nov 02 15:32:14.662526 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:14.662437    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4d0dd04f-b8ff-4ecd-a1fd-ec15ce4dea5a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d0dd04f-b8ff-4ecd-a1fd-ec15ce4dea5a-istio-podinfo\") pod \"4d0dd04f-b8ff-4ecd-a1fd-ec15ce4dea5a\" (UID: \"4d0dd04f-b8ff-4ecd-a1fd-ec15ce4dea5a\") "
Nov 02 15:32:14.662526 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:14.662516    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d0dd04f-b8ff-4ecd-a1fd-ec15ce4dea5a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d0dd04f-b8ff-4ecd-a1fd-ec15ce4dea5a-istio-envoy\") pod \"4d0dd04f-b8ff-4ecd-a1fd-ec15ce4dea5a\" (UID: \"4d0dd04f-b8ff-4ecd-a1fd-ec15ce4dea5a\") "
Nov 02 15:32:14.662653 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:14.662581    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d0dd04f-b8ff-4ecd-a1fd-ec15ce4dea5a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d0dd04f-b8ff-4ecd-a1fd-ec15ce4dea5a-istio-data\") pod \"4d0dd04f-b8ff-4ecd-a1fd-ec15ce4dea5a\" (UID: \"4d0dd04f-b8ff-4ecd-a1fd-ec15ce4dea5a\") "
Nov 02 15:32:14.662653 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:14.662619    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a09b277-1d1f-45c3-a18a-d393acfa5882-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a09b277-1d1f-45c3-a18a-d393acfa5882-credential-socket\") pod \"7a09b277-1d1f-45c3-a18a-d393acfa5882\" (UID: \"7a09b277-1d1f-45c3-a18a-d393acfa5882\") "
Nov 02 15:32:14.662653 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:14.662649    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a09b277-1d1f-45c3-a18a-d393acfa5882-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a09b277-1d1f-45c3-a18a-d393acfa5882-workload-certs\") pod \"7a09b277-1d1f-45c3-a18a-d393acfa5882\" (UID: \"7a09b277-1d1f-45c3-a18a-d393acfa5882\") "
Nov 02 15:32:14.662801 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:14.662755    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d0dd04f-b8ff-4ecd-a1fd-ec15ce4dea5a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d0dd04f-b8ff-4ecd-a1fd-ec15ce4dea5a-workload-socket\") pod \"4d0dd04f-b8ff-4ecd-a1fd-ec15ce4dea5a\" (UID: \"4d0dd04f-b8ff-4ecd-a1fd-ec15ce4dea5a\") "
Nov 02 15:32:14.662801 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:14.662783    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7a09b277-1d1f-45c3-a18a-d393acfa5882-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a09b277-1d1f-45c3-a18a-d393acfa5882-istiod-ca-cert\") pod \"7a09b277-1d1f-45c3-a18a-d393acfa5882\" (UID: \"7a09b277-1d1f-45c3-a18a-d393acfa5882\") "
Nov 02 15:32:14.662883 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:14.662814    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4d0dd04f-b8ff-4ecd-a1fd-ec15ce4dea5a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d0dd04f-b8ff-4ecd-a1fd-ec15ce4dea5a-istiod-ca-cert\") pod \"4d0dd04f-b8ff-4ecd-a1fd-ec15ce4dea5a\" (UID: \"4d0dd04f-b8ff-4ecd-a1fd-ec15ce4dea5a\") "
Nov 02 15:32:14.662883 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:14.662842    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d0dd04f-b8ff-4ecd-a1fd-ec15ce4dea5a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d0dd04f-b8ff-4ecd-a1fd-ec15ce4dea5a-workload-certs\") pod \"4d0dd04f-b8ff-4ecd-a1fd-ec15ce4dea5a\" (UID: \"4d0dd04f-b8ff-4ecd-a1fd-ec15ce4dea5a\") "
Nov 02 15:32:14.662998 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:14.662903    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7a09b277-1d1f-45c3-a18a-d393acfa5882-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a09b277-1d1f-45c3-a18a-d393acfa5882-istio-podinfo\") pod \"7a09b277-1d1f-45c3-a18a-d393acfa5882\" (UID: \"7a09b277-1d1f-45c3-a18a-d393acfa5882\") "
Nov 02 15:32:14.662998 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:14.662963    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d0dd04f-b8ff-4ecd-a1fd-ec15ce4dea5a-kube-api-access-9g4sx\" (UniqueName: \"kubernetes.io/projected/4d0dd04f-b8ff-4ecd-a1fd-ec15ce4dea5a-kube-api-access-9g4sx\") pod \"4d0dd04f-b8ff-4ecd-a1fd-ec15ce4dea5a\" (UID: \"4d0dd04f-b8ff-4ecd-a1fd-ec15ce4dea5a\") "
Nov 02 15:32:14.662998 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:14.662993    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a09b277-1d1f-45c3-a18a-d393acfa5882-kube-api-access-jnx24\" (UniqueName: \"kubernetes.io/projected/7a09b277-1d1f-45c3-a18a-d393acfa5882-kube-api-access-jnx24\") pod \"7a09b277-1d1f-45c3-a18a-d393acfa5882\" (UID: \"7a09b277-1d1f-45c3-a18a-d393acfa5882\") "
Nov 02 15:32:14.663213 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:14.663004    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d0dd04f-b8ff-4ecd-a1fd-ec15ce4dea5a-workload-socket" pod "4d0dd04f-b8ff-4ecd-a1fd-ec15ce4dea5a" (UID: "4d0dd04f-b8ff-4ecd-a1fd-ec15ce4dea5a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:32:14.663213 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:14.663000    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a09b277-1d1f-45c3-a18a-d393acfa5882-credential-socket" pod "7a09b277-1d1f-45c3-a18a-d393acfa5882" (UID: "7a09b277-1d1f-45c3-a18a-d393acfa5882"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:32:14.663213 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:14.663091    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a09b277-1d1f-45c3-a18a-d393acfa5882-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a09b277-1d1f-45c3-a18a-d393acfa5882-workload-socket\") pod \"7a09b277-1d1f-45c3-a18a-d393acfa5882\" (UID: \"7a09b277-1d1f-45c3-a18a-d393acfa5882\") "
Nov 02 15:32:14.663213 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:14.663093    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a09b277-1d1f-45c3-a18a-d393acfa5882-workload-certs" pod "7a09b277-1d1f-45c3-a18a-d393acfa5882" (UID: "7a09b277-1d1f-45c3-a18a-d393acfa5882"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:32:14.663213 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:14.663159    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d0dd04f-b8ff-4ecd-a1fd-ec15ce4dea5a-istio-token\" (UniqueName: \"kubernetes.io/projected/4d0dd04f-b8ff-4ecd-a1fd-ec15ce4dea5a-istio-token\") pod \"4d0dd04f-b8ff-4ecd-a1fd-ec15ce4dea5a\" (UID: \"4d0dd04f-b8ff-4ecd-a1fd-ec15ce4dea5a\") "
Nov 02 15:32:14.663213 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:14.663205    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a09b277-1d1f-45c3-a18a-d393acfa5882-istio-token\" (UniqueName: \"kubernetes.io/projected/7a09b277-1d1f-45c3-a18a-d393acfa5882-istio-token\") pod \"7a09b277-1d1f-45c3-a18a-d393acfa5882\" (UID: \"7a09b277-1d1f-45c3-a18a-d393acfa5882\") "
Nov 02 15:32:14.663555 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:14.663223    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d0dd04f-b8ff-4ecd-a1fd-ec15ce4dea5a-istio-data" pod "4d0dd04f-b8ff-4ecd-a1fd-ec15ce4dea5a" (UID: "4d0dd04f-b8ff-4ecd-a1fd-ec15ce4dea5a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:32:14.663555 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:14.663231    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d0dd04f-b8ff-4ecd-a1fd-ec15ce4dea5a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d0dd04f-b8ff-4ecd-a1fd-ec15ce4dea5a-credential-socket\") pod \"4d0dd04f-b8ff-4ecd-a1fd-ec15ce4dea5a\" (UID: \"4d0dd04f-b8ff-4ecd-a1fd-ec15ce4dea5a\") "
Nov 02 15:32:14.663555 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:14.663282    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d0dd04f-b8ff-4ecd-a1fd-ec15ce4dea5a-workload-certs" pod "4d0dd04f-b8ff-4ecd-a1fd-ec15ce4dea5a" (UID: "4d0dd04f-b8ff-4ecd-a1fd-ec15ce4dea5a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:32:14.663555 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:14.663333    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a09b277-1d1f-45c3-a18a-d393acfa5882-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a09b277-1d1f-45c3-a18a-d393acfa5882-istio-data\") pod \"7a09b277-1d1f-45c3-a18a-d393acfa5882\" (UID: \"7a09b277-1d1f-45c3-a18a-d393acfa5882\") "
Nov 02 15:32:14.663555 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:14.663453    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d0dd04f-b8ff-4ecd-a1fd-ec15ce4dea5a-credential-socket" pod "4d0dd04f-b8ff-4ecd-a1fd-ec15ce4dea5a" (UID: "4d0dd04f-b8ff-4ecd-a1fd-ec15ce4dea5a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:32:14.663803 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:14.663567    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d0dd04f-b8ff-4ecd-a1fd-ec15ce4dea5a-istiod-ca-cert" pod "4d0dd04f-b8ff-4ecd-a1fd-ec15ce4dea5a" (UID: "4d0dd04f-b8ff-4ecd-a1fd-ec15ce4dea5a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:32:14.663857 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:14.663804    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a09b277-1d1f-45c3-a18a-d393acfa5882-istiod-ca-cert" pod "7a09b277-1d1f-45c3-a18a-d393acfa5882" (UID: "7a09b277-1d1f-45c3-a18a-d393acfa5882"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:32:14.664076 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:14.664058    5304 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d0dd04f-b8ff-4ecd-a1fd-ec15ce4dea5a-credential-socket\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:32:14.664076 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:14.664078    5304 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d0dd04f-b8ff-4ecd-a1fd-ec15ce4dea5a-istio-data\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:32:14.664278 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:14.664091    5304 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a09b277-1d1f-45c3-a18a-d393acfa5882-credential-socket\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:32:14.664278 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:14.664104    5304 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a09b277-1d1f-45c3-a18a-d393acfa5882-workload-certs\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:32:14.664278 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:14.664117    5304 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d0dd04f-b8ff-4ecd-a1fd-ec15ce4dea5a-workload-socket\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:32:14.664278 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:14.664145    5304 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a09b277-1d1f-45c3-a18a-d393acfa5882-istiod-ca-cert\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:32:14.664278 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:14.664157    5304 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d0dd04f-b8ff-4ecd-a1fd-ec15ce4dea5a-istiod-ca-cert\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:32:14.664278 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:14.664170    5304 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d0dd04f-b8ff-4ecd-a1fd-ec15ce4dea5a-workload-certs\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:32:14.664756 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:14.664726    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a09b277-1d1f-45c3-a18a-d393acfa5882-workload-socket" pod "7a09b277-1d1f-45c3-a18a-d393acfa5882" (UID: "7a09b277-1d1f-45c3-a18a-d393acfa5882"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:32:14.664954 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:14.664929    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a09b277-1d1f-45c3-a18a-d393acfa5882-istio-data" pod "7a09b277-1d1f-45c3-a18a-d393acfa5882" (UID: "7a09b277-1d1f-45c3-a18a-d393acfa5882"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:32:14.665859 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:14.665826    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d0dd04f-b8ff-4ecd-a1fd-ec15ce4dea5a-kube-api-access-9g4sx" pod "4d0dd04f-b8ff-4ecd-a1fd-ec15ce4dea5a" (UID: "4d0dd04f-b8ff-4ecd-a1fd-ec15ce4dea5a"). InnerVolumeSpecName "kube-api-access-9g4sx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:32:14.666383 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:14.666352    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a09b277-1d1f-45c3-a18a-d393acfa5882-kube-api-access-jnx24" pod "7a09b277-1d1f-45c3-a18a-d393acfa5882" (UID: "7a09b277-1d1f-45c3-a18a-d393acfa5882"). InnerVolumeSpecName "kube-api-access-jnx24". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:32:14.666510 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:14.666480    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d0dd04f-b8ff-4ecd-a1fd-ec15ce4dea5a-istio-envoy" pod "4d0dd04f-b8ff-4ecd-a1fd-ec15ce4dea5a" (UID: "4d0dd04f-b8ff-4ecd-a1fd-ec15ce4dea5a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:32:14.666643 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:14.666618    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7a09b277-1d1f-45c3-a18a-d393acfa5882-istio-podinfo" pod "7a09b277-1d1f-45c3-a18a-d393acfa5882" (UID: "7a09b277-1d1f-45c3-a18a-d393acfa5882"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 15:32:14.667072 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:14.667055    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4d0dd04f-b8ff-4ecd-a1fd-ec15ce4dea5a-istio-podinfo" pod "4d0dd04f-b8ff-4ecd-a1fd-ec15ce4dea5a" (UID: "4d0dd04f-b8ff-4ecd-a1fd-ec15ce4dea5a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 15:32:14.667209 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:14.667087    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a09b277-1d1f-45c3-a18a-d393acfa5882-istio-envoy" pod "7a09b277-1d1f-45c3-a18a-d393acfa5882" (UID: "7a09b277-1d1f-45c3-a18a-d393acfa5882"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:32:14.667970 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:14.667865    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a09b277-1d1f-45c3-a18a-d393acfa5882-istio-token" pod "7a09b277-1d1f-45c3-a18a-d393acfa5882" (UID: "7a09b277-1d1f-45c3-a18a-d393acfa5882"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:32:14.668443 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:14.668426    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d0dd04f-b8ff-4ecd-a1fd-ec15ce4dea5a-istio-token" pod "4d0dd04f-b8ff-4ecd-a1fd-ec15ce4dea5a" (UID: "4d0dd04f-b8ff-4ecd-a1fd-ec15ce4dea5a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:32:14.687318 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:14.687293    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-4cc4m"
Nov 02 15:32:14.770678 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:14.770644    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af975b55-20fa-459f-8659-a7a69f2d2248-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af975b55-20fa-459f-8659-a7a69f2d2248-workload-certs\") pod \"af975b55-20fa-459f-8659-a7a69f2d2248\" (UID: \"af975b55-20fa-459f-8659-a7a69f2d2248\") "
Nov 02 15:32:14.770878 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:14.770714    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af975b55-20fa-459f-8659-a7a69f2d2248-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af975b55-20fa-459f-8659-a7a69f2d2248-istio-data\") pod \"af975b55-20fa-459f-8659-a7a69f2d2248\" (UID: \"af975b55-20fa-459f-8659-a7a69f2d2248\") "
Nov 02 15:32:14.770878 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:14.770752    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af975b55-20fa-459f-8659-a7a69f2d2248-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af975b55-20fa-459f-8659-a7a69f2d2248-credential-socket\") pod \"af975b55-20fa-459f-8659-a7a69f2d2248\" (UID: \"af975b55-20fa-459f-8659-a7a69f2d2248\") "
Nov 02 15:32:14.770878 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:14.770868    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/af975b55-20fa-459f-8659-a7a69f2d2248-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af975b55-20fa-459f-8659-a7a69f2d2248-istio-podinfo\") pod \"af975b55-20fa-459f-8659-a7a69f2d2248\" (UID: \"af975b55-20fa-459f-8659-a7a69f2d2248\") "
Nov 02 15:32:14.771061 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:14.770917    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af975b55-20fa-459f-8659-a7a69f2d2248-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af975b55-20fa-459f-8659-a7a69f2d2248-workload-socket\") pod \"af975b55-20fa-459f-8659-a7a69f2d2248\" (UID: \"af975b55-20fa-459f-8659-a7a69f2d2248\") "
Nov 02 15:32:14.771061 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:14.770948    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af975b55-20fa-459f-8659-a7a69f2d2248-istio-token\" (UniqueName: \"kubernetes.io/projected/af975b55-20fa-459f-8659-a7a69f2d2248-istio-token\") pod \"af975b55-20fa-459f-8659-a7a69f2d2248\" (UID: \"af975b55-20fa-459f-8659-a7a69f2d2248\") "
Nov 02 15:32:14.771061 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:14.771015    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af975b55-20fa-459f-8659-a7a69f2d2248-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af975b55-20fa-459f-8659-a7a69f2d2248-istio-envoy\") pod \"af975b55-20fa-459f-8659-a7a69f2d2248\" (UID: \"af975b55-20fa-459f-8659-a7a69f2d2248\") "
Nov 02 15:32:14.771061 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:14.771049    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af975b55-20fa-459f-8659-a7a69f2d2248-kube-api-access-pzqvg\" (UniqueName: \"kubernetes.io/projected/af975b55-20fa-459f-8659-a7a69f2d2248-kube-api-access-pzqvg\") pod \"af975b55-20fa-459f-8659-a7a69f2d2248\" (UID: \"af975b55-20fa-459f-8659-a7a69f2d2248\") "
Nov 02 15:32:14.771347 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:14.771104    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/af975b55-20fa-459f-8659-a7a69f2d2248-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af975b55-20fa-459f-8659-a7a69f2d2248-istiod-ca-cert\") pod \"af975b55-20fa-459f-8659-a7a69f2d2248\" (UID: \"af975b55-20fa-459f-8659-a7a69f2d2248\") "
Nov 02 15:32:14.771828 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:14.771796    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af975b55-20fa-459f-8659-a7a69f2d2248-workload-socket" pod "af975b55-20fa-459f-8659-a7a69f2d2248" (UID: "af975b55-20fa-459f-8659-a7a69f2d2248"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:32:14.771942 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:14.771810    5304 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a09b277-1d1f-45c3-a18a-d393acfa5882-istio-podinfo\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:32:14.771942 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:14.771855    5304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9g4sx\" (UniqueName: \"kubernetes.io/projected/4d0dd04f-b8ff-4ecd-a1fd-ec15ce4dea5a-kube-api-access-9g4sx\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:32:14.771942 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:14.771875    5304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jnx24\" (UniqueName: \"kubernetes.io/projected/7a09b277-1d1f-45c3-a18a-d393acfa5882-kube-api-access-jnx24\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:32:14.771942 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:14.771890    5304 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a09b277-1d1f-45c3-a18a-d393acfa5882-workload-socket\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:32:14.771942 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:14.771906    5304 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d0dd04f-b8ff-4ecd-a1fd-ec15ce4dea5a-istio-token\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:32:14.771942 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:14.771919    5304 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a09b277-1d1f-45c3-a18a-d393acfa5882-istio-token\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:32:14.771942 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:14.771932    5304 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a09b277-1d1f-45c3-a18a-d393acfa5882-istio-data\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:32:14.771942 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:14.771944    5304 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a09b277-1d1f-45c3-a18a-d393acfa5882-istio-envoy\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:32:14.772391 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:14.771957    5304 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d0dd04f-b8ff-4ecd-a1fd-ec15ce4dea5a-istio-podinfo\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:32:14.772391 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:14.771995    5304 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d0dd04f-b8ff-4ecd-a1fd-ec15ce4dea5a-istio-envoy\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:32:14.772391 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:14.772213    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af975b55-20fa-459f-8659-a7a69f2d2248-istio-data" pod "af975b55-20fa-459f-8659-a7a69f2d2248" (UID: "af975b55-20fa-459f-8659-a7a69f2d2248"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:32:14.772391 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:14.772237    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af975b55-20fa-459f-8659-a7a69f2d2248-istiod-ca-cert" pod "af975b55-20fa-459f-8659-a7a69f2d2248" (UID: "af975b55-20fa-459f-8659-a7a69f2d2248"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:32:14.772613 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:14.772414    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af975b55-20fa-459f-8659-a7a69f2d2248-workload-certs" pod "af975b55-20fa-459f-8659-a7a69f2d2248" (UID: "af975b55-20fa-459f-8659-a7a69f2d2248"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:32:14.772949 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:14.772925    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af975b55-20fa-459f-8659-a7a69f2d2248-credential-socket" pod "af975b55-20fa-459f-8659-a7a69f2d2248" (UID: "af975b55-20fa-459f-8659-a7a69f2d2248"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:32:14.774510 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:14.774489    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af975b55-20fa-459f-8659-a7a69f2d2248-istio-token" pod "af975b55-20fa-459f-8659-a7a69f2d2248" (UID: "af975b55-20fa-459f-8659-a7a69f2d2248"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:32:14.774665 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:14.774545    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af975b55-20fa-459f-8659-a7a69f2d2248-istio-envoy" pod "af975b55-20fa-459f-8659-a7a69f2d2248" (UID: "af975b55-20fa-459f-8659-a7a69f2d2248"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:32:14.774902 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:14.774871    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/af975b55-20fa-459f-8659-a7a69f2d2248-istio-podinfo" pod "af975b55-20fa-459f-8659-a7a69f2d2248" (UID: "af975b55-20fa-459f-8659-a7a69f2d2248"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 15:32:14.775997 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:14.775970    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af975b55-20fa-459f-8659-a7a69f2d2248-kube-api-access-pzqvg" pod "af975b55-20fa-459f-8659-a7a69f2d2248" (UID: "af975b55-20fa-459f-8659-a7a69f2d2248"). InnerVolumeSpecName "kube-api-access-pzqvg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:32:14.874058 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:14.873964    5304 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af975b55-20fa-459f-8659-a7a69f2d2248-workload-certs\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:32:14.874058 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:14.873995    5304 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af975b55-20fa-459f-8659-a7a69f2d2248-istio-data\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:32:14.874058 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:14.874005    5304 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af975b55-20fa-459f-8659-a7a69f2d2248-credential-socket\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:32:14.874058 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:14.874015    5304 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af975b55-20fa-459f-8659-a7a69f2d2248-istio-podinfo\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:32:14.874058 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:14.874024    5304 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af975b55-20fa-459f-8659-a7a69f2d2248-workload-socket\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:32:14.874058 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:14.874033    5304 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af975b55-20fa-459f-8659-a7a69f2d2248-istio-token\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:32:14.874058 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:14.874041    5304 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af975b55-20fa-459f-8659-a7a69f2d2248-istio-envoy\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:32:14.874058 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:14.874049    5304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pzqvg\" (UniqueName: \"kubernetes.io/projected/af975b55-20fa-459f-8659-a7a69f2d2248-kube-api-access-pzqvg\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:32:14.874058 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:14.874058    5304 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af975b55-20fa-459f-8659-a7a69f2d2248-istiod-ca-cert\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:32:14.992540 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:14.992513    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-r7jjp"
Nov 02 15:32:15.076863 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:15.076820    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/707b5eda-0575-4b3d-b1d8-f18005ddaa07-istio-token\" (UniqueName: \"kubernetes.io/projected/707b5eda-0575-4b3d-b1d8-f18005ddaa07-istio-token\") pod \"707b5eda-0575-4b3d-b1d8-f18005ddaa07\" (UID: \"707b5eda-0575-4b3d-b1d8-f18005ddaa07\") "
Nov 02 15:32:15.077059 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:15.076955    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/707b5eda-0575-4b3d-b1d8-f18005ddaa07-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/707b5eda-0575-4b3d-b1d8-f18005ddaa07-istio-envoy\") pod \"707b5eda-0575-4b3d-b1d8-f18005ddaa07\" (UID: \"707b5eda-0575-4b3d-b1d8-f18005ddaa07\") "
Nov 02 15:32:15.077279 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:15.077215    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/707b5eda-0575-4b3d-b1d8-f18005ddaa07-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/707b5eda-0575-4b3d-b1d8-f18005ddaa07-istiod-ca-cert\") pod \"707b5eda-0575-4b3d-b1d8-f18005ddaa07\" (UID: \"707b5eda-0575-4b3d-b1d8-f18005ddaa07\") "
Nov 02 15:32:15.077279 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:15.077264    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/707b5eda-0575-4b3d-b1d8-f18005ddaa07-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/707b5eda-0575-4b3d-b1d8-f18005ddaa07-istio-podinfo\") pod \"707b5eda-0575-4b3d-b1d8-f18005ddaa07\" (UID: \"707b5eda-0575-4b3d-b1d8-f18005ddaa07\") "
Nov 02 15:32:15.077467 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:15.077329    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/707b5eda-0575-4b3d-b1d8-f18005ddaa07-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/707b5eda-0575-4b3d-b1d8-f18005ddaa07-credential-socket\") pod \"707b5eda-0575-4b3d-b1d8-f18005ddaa07\" (UID: \"707b5eda-0575-4b3d-b1d8-f18005ddaa07\") "
Nov 02 15:32:15.077467 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:15.077372    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/707b5eda-0575-4b3d-b1d8-f18005ddaa07-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/707b5eda-0575-4b3d-b1d8-f18005ddaa07-workload-certs\") pod \"707b5eda-0575-4b3d-b1d8-f18005ddaa07\" (UID: \"707b5eda-0575-4b3d-b1d8-f18005ddaa07\") "
Nov 02 15:32:15.077579 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:15.077491    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/707b5eda-0575-4b3d-b1d8-f18005ddaa07-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/707b5eda-0575-4b3d-b1d8-f18005ddaa07-workload-socket\") pod \"707b5eda-0575-4b3d-b1d8-f18005ddaa07\" (UID: \"707b5eda-0575-4b3d-b1d8-f18005ddaa07\") "
Nov 02 15:32:15.077579 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:15.077550    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/707b5eda-0575-4b3d-b1d8-f18005ddaa07-kube-api-access-rmtml\" (UniqueName: \"kubernetes.io/projected/707b5eda-0575-4b3d-b1d8-f18005ddaa07-kube-api-access-rmtml\") pod \"707b5eda-0575-4b3d-b1d8-f18005ddaa07\" (UID: \"707b5eda-0575-4b3d-b1d8-f18005ddaa07\") "
Nov 02 15:32:15.077696 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:15.077575    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/707b5eda-0575-4b3d-b1d8-f18005ddaa07-istiod-ca-cert" pod "707b5eda-0575-4b3d-b1d8-f18005ddaa07" (UID: "707b5eda-0575-4b3d-b1d8-f18005ddaa07"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:32:15.077696 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:15.077618    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/707b5eda-0575-4b3d-b1d8-f18005ddaa07-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/707b5eda-0575-4b3d-b1d8-f18005ddaa07-istio-data\") pod \"707b5eda-0575-4b3d-b1d8-f18005ddaa07\" (UID: \"707b5eda-0575-4b3d-b1d8-f18005ddaa07\") "
Nov 02 15:32:15.078428 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:15.077860    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/707b5eda-0575-4b3d-b1d8-f18005ddaa07-workload-certs" pod "707b5eda-0575-4b3d-b1d8-f18005ddaa07" (UID: "707b5eda-0575-4b3d-b1d8-f18005ddaa07"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:32:15.078428 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:15.078066    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/707b5eda-0575-4b3d-b1d8-f18005ddaa07-credential-socket" pod "707b5eda-0575-4b3d-b1d8-f18005ddaa07" (UID: "707b5eda-0575-4b3d-b1d8-f18005ddaa07"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:32:15.078428 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:15.078334    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/707b5eda-0575-4b3d-b1d8-f18005ddaa07-istio-data" pod "707b5eda-0575-4b3d-b1d8-f18005ddaa07" (UID: "707b5eda-0575-4b3d-b1d8-f18005ddaa07"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:32:15.078428 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:15.078351    5304 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/707b5eda-0575-4b3d-b1d8-f18005ddaa07-workload-certs\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:32:15.078428 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:15.078374    5304 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/707b5eda-0575-4b3d-b1d8-f18005ddaa07-istiod-ca-cert\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:32:15.078428 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:15.078374    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/707b5eda-0575-4b3d-b1d8-f18005ddaa07-workload-socket" pod "707b5eda-0575-4b3d-b1d8-f18005ddaa07" (UID: "707b5eda-0575-4b3d-b1d8-f18005ddaa07"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:32:15.078428 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:15.078388    5304 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/707b5eda-0575-4b3d-b1d8-f18005ddaa07-credential-socket\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:32:15.079751 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:15.079708    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/707b5eda-0575-4b3d-b1d8-f18005ddaa07-istio-envoy" pod "707b5eda-0575-4b3d-b1d8-f18005ddaa07" (UID: "707b5eda-0575-4b3d-b1d8-f18005ddaa07"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:32:15.079859 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:15.079819    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/707b5eda-0575-4b3d-b1d8-f18005ddaa07-istio-token" pod "707b5eda-0575-4b3d-b1d8-f18005ddaa07" (UID: "707b5eda-0575-4b3d-b1d8-f18005ddaa07"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:32:15.080509 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:15.080473    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/707b5eda-0575-4b3d-b1d8-f18005ddaa07-kube-api-access-rmtml" pod "707b5eda-0575-4b3d-b1d8-f18005ddaa07" (UID: "707b5eda-0575-4b3d-b1d8-f18005ddaa07"). InnerVolumeSpecName "kube-api-access-rmtml". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:32:15.080598 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:15.080534    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/707b5eda-0575-4b3d-b1d8-f18005ddaa07-istio-podinfo" pod "707b5eda-0575-4b3d-b1d8-f18005ddaa07" (UID: "707b5eda-0575-4b3d-b1d8-f18005ddaa07"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 15:32:15.180287 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:15.180192    5304 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/707b5eda-0575-4b3d-b1d8-f18005ddaa07-workload-socket\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:32:15.180287 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:15.180221    5304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rmtml\" (UniqueName: \"kubernetes.io/projected/707b5eda-0575-4b3d-b1d8-f18005ddaa07-kube-api-access-rmtml\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:32:15.180287 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:15.180232    5304 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/707b5eda-0575-4b3d-b1d8-f18005ddaa07-istio-data\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:32:15.180287 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:15.180246    5304 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/707b5eda-0575-4b3d-b1d8-f18005ddaa07-istio-token\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:32:15.180287 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:15.180254    5304 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/707b5eda-0575-4b3d-b1d8-f18005ddaa07-istio-envoy\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:32:15.180287 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:15.180262    5304 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/707b5eda-0575-4b3d-b1d8-f18005ddaa07-istio-podinfo\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:32:15.636002 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:15.635975    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-4cc4m"
Nov 02 15:32:15.636537 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:15.636012    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-4cc4m" event={"ID":"af975b55-20fa-459f-8659-a7a69f2d2248","Type":"ContainerDied","Data":"60eba5d5dfba1e40a3e26ad3337aa40605ee28514ff5a1636e54a748bc1c509a"}
Nov 02 15:32:15.636537 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:15.636087    5304 scope.go:122] "RemoveContainer" containerID="8c317367ce8a8c092812e99fe4869b78b9b68f8abb841513c7e4d33790c8c4e9"
Nov 02 15:32:15.637685 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:15.637665    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-qm7zm"
Nov 02 15:32:15.637800 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:15.637737    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-qm7zm" event={"ID":"4d0dd04f-b8ff-4ecd-a1fd-ec15ce4dea5a","Type":"ContainerDied","Data":"2fc33fa7dd0d06b45af899e3333defb184801230bdda38f9634bcd9702fe5f4a"}
Nov 02 15:32:15.643142 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:15.639810    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-r7jjp"
Nov 02 15:32:15.643142 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:15.639837    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-r7jjp" event={"ID":"707b5eda-0575-4b3d-b1d8-f18005ddaa07","Type":"ContainerDied","Data":"2abd8aaf6940c452481162ce3c00b147c8d0869a6025a9f276a5bef035d47354"}
Nov 02 15:32:15.643142 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:15.640057    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-tl8dm"
Nov 02 15:32:15.653312 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:15.653294    5304 scope.go:122] "RemoveContainer" containerID="0ee06f62c3aa25c2759c1c9f53bfefcd5eb79054b406994dd091bfc80e08aeb0"
Nov 02 15:32:15.663542 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:15.663523    5304 scope.go:122] "RemoveContainer" containerID="a5494422ed014731629f8914216f2679ff9aa7d7c0532e007350dab4d603eb5c"
Nov 02 15:32:15.693904 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:15.693876    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-4cc4m"]
Nov 02 15:32:15.706159 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:15.706119    5304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-4cc4m"]
Nov 02 15:32:15.767266 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:15.767233    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-qm7zm"]
Nov 02 15:32:15.770299 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:15.770274    5304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-qm7zm"]
Nov 02 15:32:15.802815 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:15.802783    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-tl8dm"]
Nov 02 15:32:15.807099 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:15.807072    5304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-tl8dm"]
Nov 02 15:32:15.841140 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:15.841096    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-r7jjp"]
Nov 02 15:32:15.851106 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:15.851074    5304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-r7jjp"]
Nov 02 15:32:15.969828 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:15.969744    5304 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-r7jjp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 15:32:15.969828 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:15.969818    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-r7jjp" podUID="707b5eda-0575-4b3d-b1d8-f18005ddaa07" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:32:16.119765 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:16.119733    5304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4d0dd04f-b8ff-4ecd-a1fd-ec15ce4dea5a" path="/var/lib/kubelet/pods/4d0dd04f-b8ff-4ecd-a1fd-ec15ce4dea5a/volumes"
Nov 02 15:32:16.120346 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:16.120330    5304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="707b5eda-0575-4b3d-b1d8-f18005ddaa07" path="/var/lib/kubelet/pods/707b5eda-0575-4b3d-b1d8-f18005ddaa07/volumes"
Nov 02 15:32:16.120813 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:16.120799    5304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7a09b277-1d1f-45c3-a18a-d393acfa5882" path="/var/lib/kubelet/pods/7a09b277-1d1f-45c3-a18a-d393acfa5882/volumes"
Nov 02 15:32:16.121553 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:16.121535    5304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="af975b55-20fa-459f-8659-a7a69f2d2248" path="/var/lib/kubelet/pods/af975b55-20fa-459f-8659-a7a69f2d2248/volumes"
Nov 02 15:32:19.528246 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:19.528208    5304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-b55da1-687f46949-z4mjq"]
Nov 02 15:32:19.529141 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:19.529109    5304 cpu_manager.go:414] "containerMap: removing container" podUID="4d0dd04f-b8ff-4ecd-a1fd-ec15ce4dea5a" containerName="istio-proxy"
Nov 02 15:32:19.529252 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:19.529145    5304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d0dd04f-b8ff-4ecd-a1fd-ec15ce4dea5a" containerName="istio-proxy"
Nov 02 15:32:19.529252 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:19.529158    5304 cpu_manager.go:414] "containerMap: removing container" podUID="917f905c-457e-401b-af26-2af1201b5d28" containerName="main"
Nov 02 15:32:19.529252 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:19.529166    5304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="917f905c-457e-401b-af26-2af1201b5d28" containerName="main"
Nov 02 15:32:19.529252 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:19.529178    5304 cpu_manager.go:414] "containerMap: removing container" podUID="1f4f76a0-e6ea-4515-9543-e7720ae0a2c6" containerName="payload-pre-processing"
Nov 02 15:32:19.529252 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:19.529188    5304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f4f76a0-e6ea-4515-9543-e7720ae0a2c6" containerName="payload-pre-processing"
Nov 02 15:32:19.529252 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:19.529215    5304 cpu_manager.go:414] "containerMap: removing container" podUID="f2c78d1b-1497-4fe3-bfcc-f8eecb41f621" containerName="payload-processing"
Nov 02 15:32:19.529252 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:19.529222    5304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2c78d1b-1497-4fe3-bfcc-f8eecb41f621" containerName="payload-processing"
Nov 02 15:32:19.529252 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:19.529239    5304 cpu_manager.go:414] "containerMap: removing container" podUID="707b5eda-0575-4b3d-b1d8-f18005ddaa07" containerName="istio-proxy"
Nov 02 15:32:19.529252 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:19.529247    5304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="707b5eda-0575-4b3d-b1d8-f18005ddaa07" containerName="istio-proxy"
Nov 02 15:32:19.529536 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:19.529271    5304 cpu_manager.go:414] "containerMap: removing container" podUID="af975b55-20fa-459f-8659-a7a69f2d2248" containerName="istio-proxy"
Nov 02 15:32:19.529536 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:19.529279    5304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af975b55-20fa-459f-8659-a7a69f2d2248" containerName="istio-proxy"
Nov 02 15:32:19.529536 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:19.529300    5304 cpu_manager.go:414] "containerMap: removing container" podUID="7a09b277-1d1f-45c3-a18a-d393acfa5882" containerName="istio-proxy"
Nov 02 15:32:19.529536 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:19.529308    5304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a09b277-1d1f-45c3-a18a-d393acfa5882" containerName="istio-proxy"
Nov 02 15:32:19.529536 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:19.529464    5304 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d0dd04f-b8ff-4ecd-a1fd-ec15ce4dea5a" containerName="istio-proxy"
Nov 02 15:32:19.529536 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:19.529477    5304 memory_manager.go:365] "RemoveStaleState removing state" podUID="917f905c-457e-401b-af26-2af1201b5d28" containerName="main"
Nov 02 15:32:19.529536 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:19.529483    5304 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f4f76a0-e6ea-4515-9543-e7720ae0a2c6" containerName="payload-pre-processing"
Nov 02 15:32:19.529536 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:19.529492    5304 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2c78d1b-1497-4fe3-bfcc-f8eecb41f621" containerName="payload-processing"
Nov 02 15:32:19.529536 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:19.529501    5304 memory_manager.go:365] "RemoveStaleState removing state" podUID="707b5eda-0575-4b3d-b1d8-f18005ddaa07" containerName="istio-proxy"
Nov 02 15:32:19.529536 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:19.529508    5304 memory_manager.go:365] "RemoveStaleState removing state" podUID="af975b55-20fa-459f-8659-a7a69f2d2248" containerName="istio-proxy"
Nov 02 15:32:19.529536 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:19.529516    5304 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a09b277-1d1f-45c3-a18a-d393acfa5882" containerName="istio-proxy"
Nov 02 15:32:19.534506 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:19.534486    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-b55da1-687f46949-z4mjq"
Nov 02 15:32:19.561500 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:19.561464    5304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w6-b55da1-serving-cert\""
Nov 02 15:32:19.564984 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:19.564958    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-b55da1-687f46949-z4mjq"]
Nov 02 15:32:19.647499 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:19.647464    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7c093afd-5109-4891-a40c-868a053da719-metrics-certs\") pod \"maas-api-e2e-shared-b-w6-b55da1-687f46949-z4mjq\" (UID: \"7c093afd-5109-4891-a40c-868a053da719\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-b55da1-687f46949-z4mjq"
Nov 02 15:32:19.647717 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:19.647632    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7c093afd-5109-4891-a40c-868a053da719-maas-api-tls\") pod \"maas-api-e2e-shared-b-w6-b55da1-687f46949-z4mjq\" (UID: \"7c093afd-5109-4891-a40c-868a053da719\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-b55da1-687f46949-z4mjq"
Nov 02 15:32:19.647793 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:19.647757    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8wxw\" (UniqueName: \"kubernetes.io/projected/7c093afd-5109-4891-a40c-868a053da719-kube-api-access-m8wxw\") pod \"maas-api-e2e-shared-b-w6-b55da1-687f46949-z4mjq\" (UID: \"7c093afd-5109-4891-a40c-868a053da719\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-b55da1-687f46949-z4mjq"
Nov 02 15:32:19.647906 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:19.647890    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c093afd-5109-4891-a40c-868a053da719-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w6-b55da1-687f46949-z4mjq\" (UID: \"7c093afd-5109-4891-a40c-868a053da719\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-b55da1-687f46949-z4mjq"
Nov 02 15:32:19.755111 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:19.755070    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c093afd-5109-4891-a40c-868a053da719-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w6-b55da1-687f46949-z4mjq\" (UID: \"7c093afd-5109-4891-a40c-868a053da719\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-b55da1-687f46949-z4mjq"
Nov 02 15:32:19.755329 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:19.755200    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7c093afd-5109-4891-a40c-868a053da719-metrics-certs\") pod \"maas-api-e2e-shared-b-w6-b55da1-687f46949-z4mjq\" (UID: \"7c093afd-5109-4891-a40c-868a053da719\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-b55da1-687f46949-z4mjq"
Nov 02 15:32:19.755878 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:19.755834    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7c093afd-5109-4891-a40c-868a053da719-maas-api-tls\") pod \"maas-api-e2e-shared-b-w6-b55da1-687f46949-z4mjq\" (UID: \"7c093afd-5109-4891-a40c-868a053da719\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-b55da1-687f46949-z4mjq"
Nov 02 15:32:19.756005 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:19.755992    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8wxw\" (UniqueName: \"kubernetes.io/projected/7c093afd-5109-4891-a40c-868a053da719-kube-api-access-m8wxw\") pod \"maas-api-e2e-shared-b-w6-b55da1-687f46949-z4mjq\" (UID: \"7c093afd-5109-4891-a40c-868a053da719\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-b55da1-687f46949-z4mjq"
Nov 02 15:32:19.757298 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:19.756156    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c093afd-5109-4891-a40c-868a053da719-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w6-b55da1-687f46949-z4mjq\" (UID: \"7c093afd-5109-4891-a40c-868a053da719\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-b55da1-687f46949-z4mjq"
Nov 02 15:32:19.757298 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:32:19.756264    5304 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-b55da1-serving-cert: secret "maas-api-e2e-shared-b-w6-b55da1-serving-cert" not found
Nov 02 15:32:19.757298 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:32:19.756366    5304 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7c093afd-5109-4891-a40c-868a053da719-maas-api-tls podName:7c093afd-5109-4891-a40c-868a053da719 nodeName:}" failed. No retries permitted until 2026-11-02 15:32:20.256345303 +0000 UTC m=+2140.250169705 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/7c093afd-5109-4891-a40c-868a053da719-maas-api-tls") pod "maas-api-e2e-shared-b-w6-b55da1-687f46949-z4mjq" (UID: "7c093afd-5109-4891-a40c-868a053da719") : secret "maas-api-e2e-shared-b-w6-b55da1-serving-cert" not found
Nov 02 15:32:19.759913 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:19.759876    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7c093afd-5109-4891-a40c-868a053da719-metrics-certs\") pod \"maas-api-e2e-shared-b-w6-b55da1-687f46949-z4mjq\" (UID: \"7c093afd-5109-4891-a40c-868a053da719\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-b55da1-687f46949-z4mjq"
Nov 02 15:32:19.780598 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:19.780520    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8wxw\" (UniqueName: \"kubernetes.io/projected/7c093afd-5109-4891-a40c-868a053da719-kube-api-access-m8wxw\") pod \"maas-api-e2e-shared-b-w6-b55da1-687f46949-z4mjq\" (UID: \"7c093afd-5109-4891-a40c-868a053da719\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-b55da1-687f46949-z4mjq"
Nov 02 15:32:20.263137 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:20.263101    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7c093afd-5109-4891-a40c-868a053da719-maas-api-tls\") pod \"maas-api-e2e-shared-b-w6-b55da1-687f46949-z4mjq\" (UID: \"7c093afd-5109-4891-a40c-868a053da719\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-b55da1-687f46949-z4mjq"
Nov 02 15:32:20.265523 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:20.265496    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7c093afd-5109-4891-a40c-868a053da719-maas-api-tls\") pod \"maas-api-e2e-shared-b-w6-b55da1-687f46949-z4mjq\" (UID: \"7c093afd-5109-4891-a40c-868a053da719\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-b55da1-687f46949-z4mjq"
Nov 02 15:32:20.448831 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:20.448796    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-b55da1-687f46949-z4mjq"
Nov 02 15:32:20.810707 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:20.810682    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-b55da1-687f46949-z4mjq"]
Nov 02 15:32:20.813279 ip-10-0-135-138 kubenswrapper[5304]: W1102 15:32:20.813247    5304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c093afd_5109_4891_a40c_868a053da719.slice/crio-13ca995481f3e26b142a145d40807892a880f6567ee1890d049267949b9f2c72 WatchSource:0}: Error finding container 13ca995481f3e26b142a145d40807892a880f6567ee1890d049267949b9f2c72: Status 404 returned error can't find the container with id 13ca995481f3e26b142a145d40807892a880f6567ee1890d049267949b9f2c72
Nov 02 15:32:21.744176 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:21.740099    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-b55da1-687f46949-z4mjq" event={"ID":"7c093afd-5109-4891-a40c-868a053da719","Type":"ContainerStarted","Data":"32f42b3c511838439227acf76e715246eddff1c27695e530f66076e2523648d0"}
Nov 02 15:32:21.744176 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:21.740225    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-b55da1-687f46949-z4mjq" event={"ID":"7c093afd-5109-4891-a40c-868a053da719","Type":"ContainerStarted","Data":"13ca995481f3e26b142a145d40807892a880f6567ee1890d049267949b9f2c72"}
Nov 02 15:32:21.744176 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:21.740507    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-b55da1-687f46949-z4mjq"
Nov 02 15:32:21.783228 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:21.783175    5304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-b55da1-687f46949-z4mjq" podStartSLOduration=2.443969564 podStartE2EDuration="2.783163009s" totalImagesPullingTime="339.193445ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:32:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:32:21.782289669 +0000 UTC m=+2141.776114079" watchObservedRunningTime="2026-11-02 15:32:21.783163009 +0000 UTC m=+2141.776987417"
Nov 02 15:32:21.938260 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:21.938223    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-a03e84-85f8449979-gd8nh"]
Nov 02 15:32:21.957621 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:21.957590    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-a03e84-65c9f56b59-jxxpb"]
Nov 02 15:32:24.819534 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:32:24.819496    5304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k6fjk" containerName="extractor"
Nov 02 15:32:26.207721 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:26.207684    5304 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-3daba5/test-model-3daba5-kserve-6f59d8687-qp658" podUID="33fc6836-e163-4b43-8a3b-b04ce4e6f4bc" containerName="main" containerID="cri-o://4f2309313dd11f79152f30c171cf02a7c24d76179f07bd06e3923d0f3ec584e1" gracePeriod=45
Nov 02 15:32:26.336951 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:26.335220    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-3daba5/test-model-3daba5-kserve-6f59d8687-qp658" podUID="33fc6836-e163-4b43-8a3b-b04ce4e6f4bc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.127:8000/ready\": dial tcp 10.133.0.127:8000: connect: connection refused"
Nov 02 15:32:26.521118 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:26.521090    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-3daba5/test-model-3daba5-kserve-6f59d8687-qp658"
Nov 02 15:32:26.691720 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:26.691688    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33fc6836-e163-4b43-8a3b-b04ce4e6f4bc-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/33fc6836-e163-4b43-8a3b-b04ce4e6f4bc-tmp-dir\") pod \"33fc6836-e163-4b43-8a3b-b04ce4e6f4bc\" (UID: \"33fc6836-e163-4b43-8a3b-b04ce4e6f4bc\") "
Nov 02 15:32:26.691895 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:26.691742    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33fc6836-e163-4b43-8a3b-b04ce4e6f4bc-kube-api-access-wl2ht\" (UniqueName: \"kubernetes.io/projected/33fc6836-e163-4b43-8a3b-b04ce4e6f4bc-kube-api-access-wl2ht\") pod \"33fc6836-e163-4b43-8a3b-b04ce4e6f4bc\" (UID: \"33fc6836-e163-4b43-8a3b-b04ce4e6f4bc\") "
Nov 02 15:32:26.691895 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:26.691813    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33fc6836-e163-4b43-8a3b-b04ce4e6f4bc-dshm\" (UniqueName: \"kubernetes.io/empty-dir/33fc6836-e163-4b43-8a3b-b04ce4e6f4bc-dshm\") pod \"33fc6836-e163-4b43-8a3b-b04ce4e6f4bc\" (UID: \"33fc6836-e163-4b43-8a3b-b04ce4e6f4bc\") "
Nov 02 15:32:26.692068 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:26.691938    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33fc6836-e163-4b43-8a3b-b04ce4e6f4bc-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/33fc6836-e163-4b43-8a3b-b04ce4e6f4bc-model-cache\") pod \"33fc6836-e163-4b43-8a3b-b04ce4e6f4bc\" (UID: \"33fc6836-e163-4b43-8a3b-b04ce4e6f4bc\") "
Nov 02 15:32:26.692068 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:26.692013    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33fc6836-e163-4b43-8a3b-b04ce4e6f4bc-tmp-dir" pod "33fc6836-e163-4b43-8a3b-b04ce4e6f4bc" (UID: "33fc6836-e163-4b43-8a3b-b04ce4e6f4bc"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:32:26.692213 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:26.692170    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33fc6836-e163-4b43-8a3b-b04ce4e6f4bc-home\" (UniqueName: \"kubernetes.io/empty-dir/33fc6836-e163-4b43-8a3b-b04ce4e6f4bc-home\") pod \"33fc6836-e163-4b43-8a3b-b04ce4e6f4bc\" (UID: \"33fc6836-e163-4b43-8a3b-b04ce4e6f4bc\") "
Nov 02 15:32:26.692269 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:26.692215    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33fc6836-e163-4b43-8a3b-b04ce4e6f4bc-model-cache" pod "33fc6836-e163-4b43-8a3b-b04ce4e6f4bc" (UID: "33fc6836-e163-4b43-8a3b-b04ce4e6f4bc"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:32:26.692331 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:26.692290    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/33fc6836-e163-4b43-8a3b-b04ce4e6f4bc-tls-certs\" (UniqueName: \"kubernetes.io/secret/33fc6836-e163-4b43-8a3b-b04ce4e6f4bc-tls-certs\") pod \"33fc6836-e163-4b43-8a3b-b04ce4e6f4bc\" (UID: \"33fc6836-e163-4b43-8a3b-b04ce4e6f4bc\") "
Nov 02 15:32:26.692407 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:26.692387    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33fc6836-e163-4b43-8a3b-b04ce4e6f4bc-home" pod "33fc6836-e163-4b43-8a3b-b04ce4e6f4bc" (UID: "33fc6836-e163-4b43-8a3b-b04ce4e6f4bc"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:32:26.692935 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:26.692909    5304 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/33fc6836-e163-4b43-8a3b-b04ce4e6f4bc-model-cache\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:32:26.692935 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:26.692937    5304 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/33fc6836-e163-4b43-8a3b-b04ce4e6f4bc-home\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:32:26.693149 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:26.692978    5304 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/33fc6836-e163-4b43-8a3b-b04ce4e6f4bc-tmp-dir\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:32:26.694260 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:26.694233    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33fc6836-e163-4b43-8a3b-b04ce4e6f4bc-kube-api-access-wl2ht" pod "33fc6836-e163-4b43-8a3b-b04ce4e6f4bc" (UID: "33fc6836-e163-4b43-8a3b-b04ce4e6f4bc"). InnerVolumeSpecName "kube-api-access-wl2ht". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:32:26.694359 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:26.694276    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33fc6836-e163-4b43-8a3b-b04ce4e6f4bc-dshm" pod "33fc6836-e163-4b43-8a3b-b04ce4e6f4bc" (UID: "33fc6836-e163-4b43-8a3b-b04ce4e6f4bc"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:32:26.694359 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:26.694278    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/33fc6836-e163-4b43-8a3b-b04ce4e6f4bc-tls-certs" pod "33fc6836-e163-4b43-8a3b-b04ce4e6f4bc" (UID: "33fc6836-e163-4b43-8a3b-b04ce4e6f4bc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:32:26.793953 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:26.793873    5304 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/33fc6836-e163-4b43-8a3b-b04ce4e6f4bc-tls-certs\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:32:26.793953 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:26.793903    5304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wl2ht\" (UniqueName: \"kubernetes.io/projected/33fc6836-e163-4b43-8a3b-b04ce4e6f4bc-kube-api-access-wl2ht\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:32:26.793953 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:26.793915    5304 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/33fc6836-e163-4b43-8a3b-b04ce4e6f4bc-dshm\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:32:26.814275 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:26.814242    5304 generic.go:356] "Generic (PLEG): container finished" podID="33fc6836-e163-4b43-8a3b-b04ce4e6f4bc" containerID="4f2309313dd11f79152f30c171cf02a7c24d76179f07bd06e3923d0f3ec584e1" exitCode=0
Nov 02 15:32:26.814458 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:26.814286    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-3daba5/test-model-3daba5-kserve-6f59d8687-qp658"
Nov 02 15:32:26.814458 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:26.814332    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-3daba5/test-model-3daba5-kserve-6f59d8687-qp658" event={"ID":"33fc6836-e163-4b43-8a3b-b04ce4e6f4bc","Type":"ContainerDied","Data":"4f2309313dd11f79152f30c171cf02a7c24d76179f07bd06e3923d0f3ec584e1"}
Nov 02 15:32:26.814458 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:26.814373    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-3daba5/test-model-3daba5-kserve-6f59d8687-qp658" event={"ID":"33fc6836-e163-4b43-8a3b-b04ce4e6f4bc","Type":"ContainerDied","Data":"fd2d28715baa23d76d42bef7f21accead8c7491b3ea2d9bdd272b5d0db4930a2"}
Nov 02 15:32:26.814458 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:26.814390    5304 scope.go:122] "RemoveContainer" containerID="4f2309313dd11f79152f30c171cf02a7c24d76179f07bd06e3923d0f3ec584e1"
Nov 02 15:32:26.829233 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:26.829207    5304 scope.go:122] "RemoveContainer" containerID="4f2309313dd11f79152f30c171cf02a7c24d76179f07bd06e3923d0f3ec584e1"
Nov 02 15:32:26.829542 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:32:26.829519    5304 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f2309313dd11f79152f30c171cf02a7c24d76179f07bd06e3923d0f3ec584e1\": container with ID starting with 4f2309313dd11f79152f30c171cf02a7c24d76179f07bd06e3923d0f3ec584e1 not found: ID does not exist" containerID="4f2309313dd11f79152f30c171cf02a7c24d76179f07bd06e3923d0f3ec584e1"
Nov 02 15:32:26.829631 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:26.829547    5304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f2309313dd11f79152f30c171cf02a7c24d76179f07bd06e3923d0f3ec584e1"} err="failed to get container status \"4f2309313dd11f79152f30c171cf02a7c24d76179f07bd06e3923d0f3ec584e1\": rpc error: code = NotFound desc = could not find container \"4f2309313dd11f79152f30c171cf02a7c24d76179f07bd06e3923d0f3ec584e1\": container with ID starting with 4f2309313dd11f79152f30c171cf02a7c24d76179f07bd06e3923d0f3ec584e1 not found: ID does not exist"
Nov 02 15:32:26.867622 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:26.867591    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-3daba5/test-model-3daba5-kserve-6f59d8687-qp658"]
Nov 02 15:32:26.872157 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:26.872093    5304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-3daba5/test-model-3daba5-kserve-6f59d8687-qp658"]
Nov 02 15:32:26.939191 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:26.939109    5304 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-a03e84-85f8449979-gd8nh" podUID="29b94683-c6c9-495d-a0bf-1036f9b5bc3b" containerName="payload-processing" containerID="cri-o://319b401588ef2445d3a689ea6983dce27c9de3526f4c0ffdeb1bbf72fbc5b19a" gracePeriod=25
Nov 02 15:32:26.958725 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:26.958688    5304 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a03e84-65c9f56b59-jxxpb" podUID="7f8f4139-61bb-41be-828c-b1d6a7692ee2" containerName="payload-pre-processing" containerID="cri-o://f1d9eb9cc04eb0d2e0bf4f3a27c9aa986b120c6af79498d15b93b690a812382c" gracePeriod=25
Nov 02 15:32:27.200994 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:27.200967    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-a03e84-85f8449979-gd8nh"
Nov 02 15:32:27.301907 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:27.301858    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/29b94683-c6c9-495d-a0bf-1036f9b5bc3b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29b94683-c6c9-495d-a0bf-1036f9b5bc3b-plugins-config-volume\") pod \"29b94683-c6c9-495d-a0bf-1036f9b5bc3b\" (UID: \"29b94683-c6c9-495d-a0bf-1036f9b5bc3b\") "
Nov 02 15:32:27.302404 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:27.301956    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/29b94683-c6c9-495d-a0bf-1036f9b5bc3b-kube-api-access-dvh5p\" (UniqueName: \"kubernetes.io/projected/29b94683-c6c9-495d-a0bf-1036f9b5bc3b-kube-api-access-dvh5p\") pod \"29b94683-c6c9-495d-a0bf-1036f9b5bc3b\" (UID: \"29b94683-c6c9-495d-a0bf-1036f9b5bc3b\") "
Nov 02 15:32:27.302404 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:27.302331    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29b94683-c6c9-495d-a0bf-1036f9b5bc3b-plugins-config-volume" pod "29b94683-c6c9-495d-a0bf-1036f9b5bc3b" (UID: "29b94683-c6c9-495d-a0bf-1036f9b5bc3b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:32:27.302631 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:27.302612    5304 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29b94683-c6c9-495d-a0bf-1036f9b5bc3b-plugins-config-volume\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:32:27.304765 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:27.304734    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29b94683-c6c9-495d-a0bf-1036f9b5bc3b-kube-api-access-dvh5p" pod "29b94683-c6c9-495d-a0bf-1036f9b5bc3b" (UID: "29b94683-c6c9-495d-a0bf-1036f9b5bc3b"). InnerVolumeSpecName "kube-api-access-dvh5p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:32:27.404058 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:27.404022    5304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dvh5p\" (UniqueName: \"kubernetes.io/projected/29b94683-c6c9-495d-a0bf-1036f9b5bc3b-kube-api-access-dvh5p\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:32:27.775653 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:27.775621    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-b55da1-687f46949-z4mjq"
Nov 02 15:32:27.825662 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:27.825627    5304 generic.go:356] "Generic (PLEG): container finished" podID="29b94683-c6c9-495d-a0bf-1036f9b5bc3b" containerID="319b401588ef2445d3a689ea6983dce27c9de3526f4c0ffdeb1bbf72fbc5b19a" exitCode=0
Nov 02 15:32:27.825836 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:27.825675    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-a03e84-85f8449979-gd8nh"
Nov 02 15:32:27.825836 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:27.825700    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-a03e84-85f8449979-gd8nh" event={"ID":"29b94683-c6c9-495d-a0bf-1036f9b5bc3b","Type":"ContainerDied","Data":"319b401588ef2445d3a689ea6983dce27c9de3526f4c0ffdeb1bbf72fbc5b19a"}
Nov 02 15:32:27.825836 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:27.825751    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-a03e84-85f8449979-gd8nh" event={"ID":"29b94683-c6c9-495d-a0bf-1036f9b5bc3b","Type":"ContainerDied","Data":"b5e763437e62308b5b46508a4b3c7b683a2e461c9cc0249024b65484495d3c50"}
Nov 02 15:32:27.825836 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:27.825773    5304 scope.go:122] "RemoveContainer" containerID="319b401588ef2445d3a689ea6983dce27c9de3526f4c0ffdeb1bbf72fbc5b19a"
Nov 02 15:32:27.839327 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:27.839300    5304 scope.go:122] "RemoveContainer" containerID="319b401588ef2445d3a689ea6983dce27c9de3526f4c0ffdeb1bbf72fbc5b19a"
Nov 02 15:32:27.839680 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:32:27.839659    5304 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"319b401588ef2445d3a689ea6983dce27c9de3526f4c0ffdeb1bbf72fbc5b19a\": container with ID starting with 319b401588ef2445d3a689ea6983dce27c9de3526f4c0ffdeb1bbf72fbc5b19a not found: ID does not exist" containerID="319b401588ef2445d3a689ea6983dce27c9de3526f4c0ffdeb1bbf72fbc5b19a"
Nov 02 15:32:27.839760 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:27.839687    5304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"319b401588ef2445d3a689ea6983dce27c9de3526f4c0ffdeb1bbf72fbc5b19a"} err="failed to get container status \"319b401588ef2445d3a689ea6983dce27c9de3526f4c0ffdeb1bbf72fbc5b19a\": rpc error: code = NotFound desc = could not find container \"319b401588ef2445d3a689ea6983dce27c9de3526f4c0ffdeb1bbf72fbc5b19a\": container with ID starting with 319b401588ef2445d3a689ea6983dce27c9de3526f4c0ffdeb1bbf72fbc5b19a not found: ID does not exist"
Nov 02 15:32:27.879955 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:27.879923    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-a03e84-85f8449979-gd8nh"]
Nov 02 15:32:27.882957 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:27.882933    5304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-a03e84-85f8449979-gd8nh"]
Nov 02 15:32:28.119595 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:28.119565    5304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="29b94683-c6c9-495d-a0bf-1036f9b5bc3b" path="/var/lib/kubelet/pods/29b94683-c6c9-495d-a0bf-1036f9b5bc3b/volumes"
Nov 02 15:32:28.120054 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:28.120022    5304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="33fc6836-e163-4b43-8a3b-b04ce4e6f4bc" path="/var/lib/kubelet/pods/33fc6836-e163-4b43-8a3b-b04ce4e6f4bc/volumes"
Nov 02 15:32:32.312614 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:32.312576    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a03e84-65c9f56b59-jxxpb" podUID="7f8f4139-61bb-41be-828c-b1d6a7692ee2" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 15:32:40.358332 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:40.358296    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-a03e84-openshift-default-8f4cc949-4flt2"]
Nov 02 15:32:40.358736 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:40.358498    5304 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-a03e84-openshift-default-8f4cc949-4flt2" podUID="c109588b-9544-4a37-bb13-d81572067546" containerName="istio-proxy" containerID="cri-o://428106e52b4fde94e8785db87868da1cd0730d9066963e510fa436919ebcd580" gracePeriod=30
Nov 02 15:32:41.700938 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:41.700903    5304 scope.go:122] "RemoveContainer" containerID="1e9030e051ac48ab0cc6e29ac000907223c70f491d6fdcb0e7bcf3c49454816d"
Nov 02 15:32:42.312651 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:42.312608    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a03e84-65c9f56b59-jxxpb" podUID="7f8f4139-61bb-41be-828c-b1d6a7692ee2" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 15:32:45.641067 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:45.640996    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-a03e84-openshift-default-8f4cc949-4flt2"
Nov 02 15:32:45.697787 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:45.697759    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c109588b-9544-4a37-bb13-d81572067546-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c109588b-9544-4a37-bb13-d81572067546-istio-envoy\") pod \"c109588b-9544-4a37-bb13-d81572067546\" (UID: \"c109588b-9544-4a37-bb13-d81572067546\") "
Nov 02 15:32:45.697957 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:45.697845    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c109588b-9544-4a37-bb13-d81572067546-istio-token\" (UniqueName: \"kubernetes.io/projected/c109588b-9544-4a37-bb13-d81572067546-istio-token\") pod \"c109588b-9544-4a37-bb13-d81572067546\" (UID: \"c109588b-9544-4a37-bb13-d81572067546\") "
Nov 02 15:32:45.697957 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:45.697917    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c109588b-9544-4a37-bb13-d81572067546-kube-api-access-xs8cd\" (UniqueName: \"kubernetes.io/projected/c109588b-9544-4a37-bb13-d81572067546-kube-api-access-xs8cd\") pod \"c109588b-9544-4a37-bb13-d81572067546\" (UID: \"c109588b-9544-4a37-bb13-d81572067546\") "
Nov 02 15:32:45.697957 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:45.697948    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c109588b-9544-4a37-bb13-d81572067546-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c109588b-9544-4a37-bb13-d81572067546-workload-certs\") pod \"c109588b-9544-4a37-bb13-d81572067546\" (UID: \"c109588b-9544-4a37-bb13-d81572067546\") "
Nov 02 15:32:45.698160 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:45.697980    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c109588b-9544-4a37-bb13-d81572067546-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c109588b-9544-4a37-bb13-d81572067546-credential-socket\") pod \"c109588b-9544-4a37-bb13-d81572067546\" (UID: \"c109588b-9544-4a37-bb13-d81572067546\") "
Nov 02 15:32:45.698160 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:45.698021    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c109588b-9544-4a37-bb13-d81572067546-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c109588b-9544-4a37-bb13-d81572067546-istio-data\") pod \"c109588b-9544-4a37-bb13-d81572067546\" (UID: \"c109588b-9544-4a37-bb13-d81572067546\") "
Nov 02 15:32:45.698160 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:45.698077    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c109588b-9544-4a37-bb13-d81572067546-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c109588b-9544-4a37-bb13-d81572067546-workload-socket\") pod \"c109588b-9544-4a37-bb13-d81572067546\" (UID: \"c109588b-9544-4a37-bb13-d81572067546\") "
Nov 02 15:32:45.698330 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:45.698243    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c109588b-9544-4a37-bb13-d81572067546-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c109588b-9544-4a37-bb13-d81572067546-istiod-ca-cert\") pod \"c109588b-9544-4a37-bb13-d81572067546\" (UID: \"c109588b-9544-4a37-bb13-d81572067546\") "
Nov 02 15:32:45.698330 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:45.698269    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c109588b-9544-4a37-bb13-d81572067546-workload-certs" pod "c109588b-9544-4a37-bb13-d81572067546" (UID: "c109588b-9544-4a37-bb13-d81572067546"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:32:45.698330 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:45.698287    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c109588b-9544-4a37-bb13-d81572067546-credential-socket" pod "c109588b-9544-4a37-bb13-d81572067546" (UID: "c109588b-9544-4a37-bb13-d81572067546"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:32:45.698330 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:45.698292    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c109588b-9544-4a37-bb13-d81572067546-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c109588b-9544-4a37-bb13-d81572067546-istio-podinfo\") pod \"c109588b-9544-4a37-bb13-d81572067546\" (UID: \"c109588b-9544-4a37-bb13-d81572067546\") "
Nov 02 15:32:45.698601 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:45.698321    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c109588b-9544-4a37-bb13-d81572067546-istio-data" pod "c109588b-9544-4a37-bb13-d81572067546" (UID: "c109588b-9544-4a37-bb13-d81572067546"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:32:45.698662 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:45.698463    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c109588b-9544-4a37-bb13-d81572067546-workload-socket" pod "c109588b-9544-4a37-bb13-d81572067546" (UID: "c109588b-9544-4a37-bb13-d81572067546"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:32:45.698662 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:45.698614    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c109588b-9544-4a37-bb13-d81572067546-istiod-ca-cert" pod "c109588b-9544-4a37-bb13-d81572067546" (UID: "c109588b-9544-4a37-bb13-d81572067546"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:32:45.699037 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:45.699015    5304 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c109588b-9544-4a37-bb13-d81572067546-istiod-ca-cert\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:32:45.699120 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:45.699042    5304 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c109588b-9544-4a37-bb13-d81572067546-workload-certs\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:32:45.699120 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:45.699056    5304 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c109588b-9544-4a37-bb13-d81572067546-credential-socket\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:32:45.699120 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:45.699068    5304 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c109588b-9544-4a37-bb13-d81572067546-istio-data\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:32:45.699120 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:45.699082    5304 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c109588b-9544-4a37-bb13-d81572067546-workload-socket\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:32:45.700062 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:45.700035    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c109588b-9544-4a37-bb13-d81572067546-istio-envoy" pod "c109588b-9544-4a37-bb13-d81572067546" (UID: "c109588b-9544-4a37-bb13-d81572067546"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:32:45.700372 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:45.700353    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c109588b-9544-4a37-bb13-d81572067546-istio-podinfo" pod "c109588b-9544-4a37-bb13-d81572067546" (UID: "c109588b-9544-4a37-bb13-d81572067546"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 15:32:45.700590 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:45.700571    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c109588b-9544-4a37-bb13-d81572067546-kube-api-access-xs8cd" pod "c109588b-9544-4a37-bb13-d81572067546" (UID: "c109588b-9544-4a37-bb13-d81572067546"). InnerVolumeSpecName "kube-api-access-xs8cd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:32:45.700678 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:45.700606    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c109588b-9544-4a37-bb13-d81572067546-istio-token" pod "c109588b-9544-4a37-bb13-d81572067546" (UID: "c109588b-9544-4a37-bb13-d81572067546"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:32:45.800823 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:45.800795    5304 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c109588b-9544-4a37-bb13-d81572067546-istio-token\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:32:45.800823 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:45.800821    5304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xs8cd\" (UniqueName: \"kubernetes.io/projected/c109588b-9544-4a37-bb13-d81572067546-kube-api-access-xs8cd\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:32:45.800823 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:45.800831    5304 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c109588b-9544-4a37-bb13-d81572067546-istio-podinfo\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:32:45.801063 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:45.800840    5304 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c109588b-9544-4a37-bb13-d81572067546-istio-envoy\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:32:46.001800 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:46.001710    5304 generic.go:356] "Generic (PLEG): container finished" podID="c109588b-9544-4a37-bb13-d81572067546" containerID="428106e52b4fde94e8785db87868da1cd0730d9066963e510fa436919ebcd580" exitCode=0
Nov 02 15:32:46.001800 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:46.001760    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-a03e84-openshift-default-8f4cc949-4flt2"
Nov 02 15:32:46.002052 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:46.001795    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a03e84-openshift-default-8f4cc949-4flt2" event={"ID":"c109588b-9544-4a37-bb13-d81572067546","Type":"ContainerDied","Data":"428106e52b4fde94e8785db87868da1cd0730d9066963e510fa436919ebcd580"}
Nov 02 15:32:46.002052 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:46.001857    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a03e84-openshift-default-8f4cc949-4flt2" event={"ID":"c109588b-9544-4a37-bb13-d81572067546","Type":"ContainerDied","Data":"196549cb08ecda30881e5ff5f8c990e688694927fb0ee40c480f19d0fb6821dc"}
Nov 02 15:32:46.002052 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:46.001888    5304 scope.go:122] "RemoveContainer" containerID="428106e52b4fde94e8785db87868da1cd0730d9066963e510fa436919ebcd580"
Nov 02 15:32:46.013338 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:46.013267    5304 scope.go:122] "RemoveContainer" containerID="8d68a037a516dffec694fa5bc9008369c95dbe007c3efb2c181b9dc643c4597a"
Nov 02 15:32:46.022864 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:46.022846    5304 scope.go:122] "RemoveContainer" containerID="428106e52b4fde94e8785db87868da1cd0730d9066963e510fa436919ebcd580"
Nov 02 15:32:46.023181 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:32:46.023159    5304 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"428106e52b4fde94e8785db87868da1cd0730d9066963e510fa436919ebcd580\": container with ID starting with 428106e52b4fde94e8785db87868da1cd0730d9066963e510fa436919ebcd580 not found: ID does not exist" containerID="428106e52b4fde94e8785db87868da1cd0730d9066963e510fa436919ebcd580"
Nov 02 15:32:46.023250 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:46.023190    5304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"428106e52b4fde94e8785db87868da1cd0730d9066963e510fa436919ebcd580"} err="failed to get container status \"428106e52b4fde94e8785db87868da1cd0730d9066963e510fa436919ebcd580\": rpc error: code = NotFound desc = could not find container \"428106e52b4fde94e8785db87868da1cd0730d9066963e510fa436919ebcd580\": container with ID starting with 428106e52b4fde94e8785db87868da1cd0730d9066963e510fa436919ebcd580 not found: ID does not exist"
Nov 02 15:32:46.023250 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:46.023211    5304 scope.go:122] "RemoveContainer" containerID="8d68a037a516dffec694fa5bc9008369c95dbe007c3efb2c181b9dc643c4597a"
Nov 02 15:32:46.023501 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:32:46.023481    5304 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8d68a037a516dffec694fa5bc9008369c95dbe007c3efb2c181b9dc643c4597a\": container with ID starting with 8d68a037a516dffec694fa5bc9008369c95dbe007c3efb2c181b9dc643c4597a not found: ID does not exist" containerID="8d68a037a516dffec694fa5bc9008369c95dbe007c3efb2c181b9dc643c4597a"
Nov 02 15:32:46.023556 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:46.023509    5304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d68a037a516dffec694fa5bc9008369c95dbe007c3efb2c181b9dc643c4597a"} err="failed to get container status \"8d68a037a516dffec694fa5bc9008369c95dbe007c3efb2c181b9dc643c4597a\": rpc error: code = NotFound desc = could not find container \"8d68a037a516dffec694fa5bc9008369c95dbe007c3efb2c181b9dc643c4597a\": container with ID starting with 8d68a037a516dffec694fa5bc9008369c95dbe007c3efb2c181b9dc643c4597a not found: ID does not exist"
Nov 02 15:32:46.057570 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:46.057542    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-a03e84-openshift-default-8f4cc949-4flt2"]
Nov 02 15:32:46.066327 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:46.066302    5304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-a03e84-openshift-default-8f4cc949-4flt2"]
Nov 02 15:32:46.119543 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:46.119504    5304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c109588b-9544-4a37-bb13-d81572067546" path="/var/lib/kubelet/pods/c109588b-9544-4a37-bb13-d81572067546/volumes"
Nov 02 15:32:47.404692 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:47.404623    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a03e84-65c9f56b59-jxxpb"
Nov 02 15:32:47.560631 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:47.560536    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7f8f4139-61bb-41be-828c-b1d6a7692ee2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f8f4139-61bb-41be-828c-b1d6a7692ee2-plugins-config-volume\") pod \"7f8f4139-61bb-41be-828c-b1d6a7692ee2\" (UID: \"7f8f4139-61bb-41be-828c-b1d6a7692ee2\") "
Nov 02 15:32:47.560822 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:47.560745    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f8f4139-61bb-41be-828c-b1d6a7692ee2-kube-api-access-f6287\" (UniqueName: \"kubernetes.io/projected/7f8f4139-61bb-41be-828c-b1d6a7692ee2-kube-api-access-f6287\") pod \"7f8f4139-61bb-41be-828c-b1d6a7692ee2\" (UID: \"7f8f4139-61bb-41be-828c-b1d6a7692ee2\") "
Nov 02 15:32:47.560978 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:47.560947    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f8f4139-61bb-41be-828c-b1d6a7692ee2-plugins-config-volume" pod "7f8f4139-61bb-41be-828c-b1d6a7692ee2" (UID: "7f8f4139-61bb-41be-828c-b1d6a7692ee2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:32:47.561330 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:47.561308    5304 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f8f4139-61bb-41be-828c-b1d6a7692ee2-plugins-config-volume\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:32:47.562902 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:47.562879    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f8f4139-61bb-41be-828c-b1d6a7692ee2-kube-api-access-f6287" pod "7f8f4139-61bb-41be-828c-b1d6a7692ee2" (UID: "7f8f4139-61bb-41be-828c-b1d6a7692ee2"). InnerVolumeSpecName "kube-api-access-f6287". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:32:47.662847 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:47.662811    5304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f6287\" (UniqueName: \"kubernetes.io/projected/7f8f4139-61bb-41be-828c-b1d6a7692ee2-kube-api-access-f6287\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:32:48.017820 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:48.017784    5304 generic.go:356] "Generic (PLEG): container finished" podID="7f8f4139-61bb-41be-828c-b1d6a7692ee2" containerID="f1d9eb9cc04eb0d2e0bf4f3a27c9aa986b120c6af79498d15b93b690a812382c" exitCode=0
Nov 02 15:32:48.018000 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:48.017837    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a03e84-65c9f56b59-jxxpb"
Nov 02 15:32:48.018000 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:48.017871    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a03e84-65c9f56b59-jxxpb" event={"ID":"7f8f4139-61bb-41be-828c-b1d6a7692ee2","Type":"ContainerDied","Data":"f1d9eb9cc04eb0d2e0bf4f3a27c9aa986b120c6af79498d15b93b690a812382c"}
Nov 02 15:32:48.018000 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:48.017929    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a03e84-65c9f56b59-jxxpb" event={"ID":"7f8f4139-61bb-41be-828c-b1d6a7692ee2","Type":"ContainerDied","Data":"3acb31e09ee4b7502d1bf20bae046c1cccdcd6eb2179524028e39168a05fb9bc"}
Nov 02 15:32:48.018000 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:48.017956    5304 scope.go:122] "RemoveContainer" containerID="f1d9eb9cc04eb0d2e0bf4f3a27c9aa986b120c6af79498d15b93b690a812382c"
Nov 02 15:32:48.028832 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:48.028807    5304 scope.go:122] "RemoveContainer" containerID="f1d9eb9cc04eb0d2e0bf4f3a27c9aa986b120c6af79498d15b93b690a812382c"
Nov 02 15:32:48.029177 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:32:48.029150    5304 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f1d9eb9cc04eb0d2e0bf4f3a27c9aa986b120c6af79498d15b93b690a812382c\": container with ID starting with f1d9eb9cc04eb0d2e0bf4f3a27c9aa986b120c6af79498d15b93b690a812382c not found: ID does not exist" containerID="f1d9eb9cc04eb0d2e0bf4f3a27c9aa986b120c6af79498d15b93b690a812382c"
Nov 02 15:32:48.029270 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:48.029187    5304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1d9eb9cc04eb0d2e0bf4f3a27c9aa986b120c6af79498d15b93b690a812382c"} err="failed to get container status \"f1d9eb9cc04eb0d2e0bf4f3a27c9aa986b120c6af79498d15b93b690a812382c\": rpc error: code = NotFound desc = could not find container \"f1d9eb9cc04eb0d2e0bf4f3a27c9aa986b120c6af79498d15b93b690a812382c\": container with ID starting with f1d9eb9cc04eb0d2e0bf4f3a27c9aa986b120c6af79498d15b93b690a812382c not found: ID does not exist"
Nov 02 15:32:48.070246 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:48.070210    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-a03e84-65c9f56b59-jxxpb"]
Nov 02 15:32:48.082354 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:48.082313    5304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-a03e84-65c9f56b59-jxxpb"]
Nov 02 15:32:48.120852 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:48.120817    5304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7f8f4139-61bb-41be-828c-b1d6a7692ee2" path="/var/lib/kubelet/pods/7f8f4139-61bb-41be-828c-b1d6a7692ee2/volumes"
Nov 02 15:32:54.799535 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:32:54.799450    5304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k6fjk" containerName="extractor"
Nov 02 15:32:59.884636 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:59.884597    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-2abeaa-56f4d84db5-5nll8"]
Nov 02 15:32:59.885101 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:32:59.884830    5304 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2abeaa-56f4d84db5-5nll8" podUID="e0cf2360-2112-4333-8fd9-e597685ea1cf" containerName="maas-api" containerID="cri-o://88bdbced8ee085ce743909d97f75472a6f0b048b70da3346a80db388a97a91ae" gracePeriod=30
Nov 02 15:33:00.122681 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:33:00.122647    5304 generic.go:356] "Generic (PLEG): container finished" podID="e0cf2360-2112-4333-8fd9-e597685ea1cf" containerID="88bdbced8ee085ce743909d97f75472a6f0b048b70da3346a80db388a97a91ae" exitCode=0
Nov 02 15:33:00.123229 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:33:00.123199    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2abeaa-56f4d84db5-5nll8" event={"ID":"e0cf2360-2112-4333-8fd9-e597685ea1cf","Type":"ContainerDied","Data":"88bdbced8ee085ce743909d97f75472a6f0b048b70da3346a80db388a97a91ae"}
Nov 02 15:33:00.255198 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:33:00.255166    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2abeaa-56f4d84db5-5nll8"
Nov 02 15:33:00.383463 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:33:00.381620    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e0cf2360-2112-4333-8fd9-e597685ea1cf-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e0cf2360-2112-4333-8fd9-e597685ea1cf-maas-api-tls\") pod \"e0cf2360-2112-4333-8fd9-e597685ea1cf\" (UID: \"e0cf2360-2112-4333-8fd9-e597685ea1cf\") "
Nov 02 15:33:00.384169 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:33:00.384141    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e0cf2360-2112-4333-8fd9-e597685ea1cf-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0cf2360-2112-4333-8fd9-e597685ea1cf-openshift-service-ca\") pod \"e0cf2360-2112-4333-8fd9-e597685ea1cf\" (UID: \"e0cf2360-2112-4333-8fd9-e597685ea1cf\") "
Nov 02 15:33:00.384421 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:33:00.384398    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e0cf2360-2112-4333-8fd9-e597685ea1cf-kube-api-access-ccd5x\" (UniqueName: \"kubernetes.io/projected/e0cf2360-2112-4333-8fd9-e597685ea1cf-kube-api-access-ccd5x\") pod \"e0cf2360-2112-4333-8fd9-e597685ea1cf\" (UID: \"e0cf2360-2112-4333-8fd9-e597685ea1cf\") "
Nov 02 15:33:00.384621 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:33:00.384603    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e0cf2360-2112-4333-8fd9-e597685ea1cf-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e0cf2360-2112-4333-8fd9-e597685ea1cf-metrics-certs\") pod \"e0cf2360-2112-4333-8fd9-e597685ea1cf\" (UID: \"e0cf2360-2112-4333-8fd9-e597685ea1cf\") "
Nov 02 15:33:00.386403 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:33:00.386376    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0cf2360-2112-4333-8fd9-e597685ea1cf-openshift-service-ca" pod "e0cf2360-2112-4333-8fd9-e597685ea1cf" (UID: "e0cf2360-2112-4333-8fd9-e597685ea1cf"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:33:00.388934 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:33:00.388911    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e0cf2360-2112-4333-8fd9-e597685ea1cf-maas-api-tls" pod "e0cf2360-2112-4333-8fd9-e597685ea1cf" (UID: "e0cf2360-2112-4333-8fd9-e597685ea1cf"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:33:00.419334 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:33:00.419288    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e0cf2360-2112-4333-8fd9-e597685ea1cf-metrics-certs" pod "e0cf2360-2112-4333-8fd9-e597685ea1cf" (UID: "e0cf2360-2112-4333-8fd9-e597685ea1cf"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:33:00.432374 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:33:00.430364    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0cf2360-2112-4333-8fd9-e597685ea1cf-kube-api-access-ccd5x" pod "e0cf2360-2112-4333-8fd9-e597685ea1cf" (UID: "e0cf2360-2112-4333-8fd9-e597685ea1cf"). InnerVolumeSpecName "kube-api-access-ccd5x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:33:00.487287 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:33:00.487202    5304 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e0cf2360-2112-4333-8fd9-e597685ea1cf-maas-api-tls\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:33:00.487287 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:33:00.487239    5304 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0cf2360-2112-4333-8fd9-e597685ea1cf-openshift-service-ca\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:33:00.487287 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:33:00.487253    5304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ccd5x\" (UniqueName: \"kubernetes.io/projected/e0cf2360-2112-4333-8fd9-e597685ea1cf-kube-api-access-ccd5x\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:33:00.487287 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:33:00.487265    5304 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e0cf2360-2112-4333-8fd9-e597685ea1cf-metrics-certs\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:33:01.130918 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:33:01.130888    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2abeaa-56f4d84db5-5nll8"
Nov 02 15:33:01.131446 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:33:01.130934    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2abeaa-56f4d84db5-5nll8" event={"ID":"e0cf2360-2112-4333-8fd9-e597685ea1cf","Type":"ContainerDied","Data":"405dce5cee62346e6705e77aa5c691dd4839fe3936f0330480762303a102a1b8"}
Nov 02 15:33:01.131446 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:33:01.131179    5304 scope.go:122] "RemoveContainer" containerID="88bdbced8ee085ce743909d97f75472a6f0b048b70da3346a80db388a97a91ae"
Nov 02 15:33:01.183313 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:33:01.183283    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-2abeaa-56f4d84db5-5nll8"]
Nov 02 15:33:01.195156 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:33:01.195111    5304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-2abeaa-56f4d84db5-5nll8"]
Nov 02 15:33:02.120641 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:33:02.120602    5304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e0cf2360-2112-4333-8fd9-e597685ea1cf" path="/var/lib/kubelet/pods/e0cf2360-2112-4333-8fd9-e597685ea1cf/volumes"
Nov 02 15:33:08.264183 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:33:08.264100    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-2abeaa-openshift-default-ccdd5d949-6fmzt"]
Nov 02 15:33:08.264843 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:33:08.264376    5304 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-2abeaa-openshift-default-ccdd5d949-6fmzt" podUID="aa31b68f-cd14-41d9-92ca-2d1df613f263" containerName="istio-proxy" containerID="cri-o://cacbcf86b32c08d520cac3f78168b566738508a647c8224c74f5c07655bf3b7e" gracePeriod=30
Nov 02 15:33:10.039155 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:33:10.022538    5304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w4-501738/auth-test-model-501738-kserve-85cfb4c9cc-xkzq5"]
Nov 02 15:33:10.039155 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:33:10.023515    5304 cpu_manager.go:414] "containerMap: removing container" podUID="29b94683-c6c9-495d-a0bf-1036f9b5bc3b" containerName="payload-processing"
Nov 02 15:33:10.039155 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:33:10.023536    5304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29b94683-c6c9-495d-a0bf-1036f9b5bc3b" containerName="payload-processing"
Nov 02 15:33:10.039155 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:33:10.023551    5304 cpu_manager.go:414] "containerMap: removing container" podUID="33fc6836-e163-4b43-8a3b-b04ce4e6f4bc" containerName="main"
Nov 02 15:33:10.039155 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:33:10.023559    5304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33fc6836-e163-4b43-8a3b-b04ce4e6f4bc" containerName="main"
Nov 02 15:33:10.039155 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:33:10.023588    5304 cpu_manager.go:414] "containerMap: removing container" podUID="e0cf2360-2112-4333-8fd9-e597685ea1cf" containerName="maas-api"
Nov 02 15:33:10.039155 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:33:10.023596    5304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e0cf2360-2112-4333-8fd9-e597685ea1cf" containerName="maas-api"
Nov 02 15:33:10.039155 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:33:10.023629    5304 cpu_manager.go:414] "containerMap: removing container" podUID="7f8f4139-61bb-41be-828c-b1d6a7692ee2" containerName="payload-pre-processing"
Nov 02 15:33:10.039155 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:33:10.023636    5304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f8f4139-61bb-41be-828c-b1d6a7692ee2" containerName="payload-pre-processing"
Nov 02 15:33:10.039155 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:33:10.023651    5304 cpu_manager.go:414] "containerMap: removing container" podUID="c109588b-9544-4a37-bb13-d81572067546" containerName="istio-proxy"
Nov 02 15:33:10.039155 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:33:10.023660    5304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c109588b-9544-4a37-bb13-d81572067546" containerName="istio-proxy"
Nov 02 15:33:10.039155 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:33:10.023851    5304 memory_manager.go:365] "RemoveStaleState removing state" podUID="29b94683-c6c9-495d-a0bf-1036f9b5bc3b" containerName="payload-processing"
Nov 02 15:33:10.039155 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:33:10.023867    5304 memory_manager.go:365] "RemoveStaleState removing state" podUID="33fc6836-e163-4b43-8a3b-b04ce4e6f4bc" containerName="main"
Nov 02 15:33:10.039155 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:33:10.023879    5304 memory_manager.go:365] "RemoveStaleState removing state" podUID="e0cf2360-2112-4333-8fd9-e597685ea1cf" containerName="maas-api"
Nov 02 15:33:10.039155 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:33:10.023894    5304 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f8f4139-61bb-41be-828c-b1d6a7692ee2" containerName="payload-pre-processing"
Nov 02 15:33:10.039155 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:33:10.023904    5304 memory_manager.go:365] "RemoveStaleState removing state" podUID="c109588b-9544-4a37-bb13-d81572067546" containerName="istio-proxy"
Nov 02 15:33:10.039155 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:33:10.023917    5304 memory_manager.go:365] "RemoveStaleState removing state" podUID="c109588b-9544-4a37-bb13-d81572067546" containerName="istio-proxy"
Nov 02 15:33:10.039155 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:33:10.026958    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-501738/auth-test-model-501738-kserve-85cfb4c9cc-xkzq5"
Nov 02 15:33:10.068588 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:33:10.068553    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w4-501738/auth-test-model-501738-kserve-85cfb4c9cc-xkzq5"]
Nov 02 15:33:10.076754 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:33:10.076717    5304 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w4-501738\"/\"kube-root-ca.crt\""
Nov 02 15:33:10.076905 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:33:10.076787    5304 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w4-501738\"/\"openshift-service-ca.crt\""
Nov 02 15:33:10.076973 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:33:10.076901    5304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w4-501738\"/\"auth-test-model-501738-kserve-self-signed-certs\""
Nov 02 15:33:10.077039 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:33:10.076982    5304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w4-501738\"/\"default-dockercfg-cjptv\""
Nov 02 15:33:10.101800 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:33:10.101765    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c4240adb-0390-4339-a752-209bdd4b9b50-tmp-dir\") pod \"auth-test-model-501738-kserve-85cfb4c9cc-xkzq5\" (UID: \"c4240adb-0390-4339-a752-209bdd4b9b50\") " pod="ai-tenant-e2e-shared-a-w4-501738/auth-test-model-501738-kserve-85cfb4c9cc-xkzq5"
Nov 02 15:33:10.101956 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:33:10.101886    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c4240adb-0390-4339-a752-209bdd4b9b50-model-cache\") pod \"auth-test-model-501738-kserve-85cfb4c9cc-xkzq5\" (UID: \"c4240adb-0390-4339-a752-209bdd4b9b50\") " pod="ai-tenant-e2e-shared-a-w4-501738/auth-test-model-501738-kserve-85cfb4c9cc-xkzq5"
Nov 02 15:33:10.101956 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:33:10.101946    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c4240adb-0390-4339-a752-209bdd4b9b50-dshm\") pod \"auth-test-model-501738-kserve-85cfb4c9cc-xkzq5\" (UID: \"c4240adb-0390-4339-a752-209bdd4b9b50\") " pod="ai-tenant-e2e-shared-a-w4-501738/auth-test-model-501738-kserve-85cfb4c9cc-xkzq5"
Nov 02 15:33:10.102051 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:33:10.102037    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c4240adb-0390-4339-a752-209bdd4b9b50-home\") pod \"auth-test-model-501738-kserve-85cfb4c9cc-xkzq5\" (UID: \"c4240adb-0390-4339-a752-209bdd4b9b50\") " pod="ai-tenant-e2e-shared-a-w4-501738/auth-test-model-501738-kserve-85cfb4c9cc-xkzq5"
Nov 02 15:33:10.102092 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:33:10.102070    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c4240adb-0390-4339-a752-209bdd4b9b50-tls-certs\") pod \"auth-test-model-501738-kserve-85cfb4c9cc-xkzq5\" (UID: \"c4240adb-0390-4339-a752-209bdd4b9b50\") " pod="ai-tenant-e2e-shared-a-w4-501738/auth-test-model-501738-kserve-85cfb4c9cc-xkzq5"
Nov 02 15:33:10.102319 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:33:10.102297    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kwwpg\" (UniqueName: \"kubernetes.io/projected/c4240adb-0390-4339-a752-209bdd4b9b50-kube-api-access-kwwpg\") pod \"auth-test-model-501738-kserve-85cfb4c9cc-xkzq5\" (UID: \"c4240adb-0390-4339-a752-209bdd4b9b50\") " pod="ai-tenant-e2e-shared-a-w4-501738/auth-test-model-501738-kserve-85cfb4c9cc-xkzq5"
Nov 02 15:33:10.204064 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:33:10.204023    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c4240adb-0390-4339-a752-209bdd4b9b50-model-cache\") pod \"auth-test-model-501738-kserve-85cfb4c9cc-xkzq5\" (UID: \"c4240adb-0390-4339-a752-209bdd4b9b50\") " pod="ai-tenant-e2e-shared-a-w4-501738/auth-test-model-501738-kserve-85cfb4c9cc-xkzq5"
Nov 02 15:33:10.204272 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:33:10.204180    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c4240adb-0390-4339-a752-209bdd4b9b50-dshm\") pod \"auth-test-model-501738-kserve-85cfb4c9cc-xkzq5\" (UID: \"c4240adb-0390-4339-a752-209bdd4b9b50\") " pod="ai-tenant-e2e-shared-a-w4-501738/auth-test-model-501738-kserve-85cfb4c9cc-xkzq5"
Nov 02 15:33:10.204348 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:33:10.204276    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c4240adb-0390-4339-a752-209bdd4b9b50-home\") pod \"auth-test-model-501738-kserve-85cfb4c9cc-xkzq5\" (UID: \"c4240adb-0390-4339-a752-209bdd4b9b50\") " pod="ai-tenant-e2e-shared-a-w4-501738/auth-test-model-501738-kserve-85cfb4c9cc-xkzq5"
Nov 02 15:33:10.204348 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:33:10.204301    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c4240adb-0390-4339-a752-209bdd4b9b50-tls-certs\") pod \"auth-test-model-501738-kserve-85cfb4c9cc-xkzq5\" (UID: \"c4240adb-0390-4339-a752-209bdd4b9b50\") " pod="ai-tenant-e2e-shared-a-w4-501738/auth-test-model-501738-kserve-85cfb4c9cc-xkzq5"
Nov 02 15:33:10.204470 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:33:10.204448    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c4240adb-0390-4339-a752-209bdd4b9b50-model-cache\") pod \"auth-test-model-501738-kserve-85cfb4c9cc-xkzq5\" (UID: \"c4240adb-0390-4339-a752-209bdd4b9b50\") " pod="ai-tenant-e2e-shared-a-w4-501738/auth-test-model-501738-kserve-85cfb4c9cc-xkzq5"
Nov 02 15:33:10.204657 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:33:10.204629    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c4240adb-0390-4339-a752-209bdd4b9b50-home\") pod \"auth-test-model-501738-kserve-85cfb4c9cc-xkzq5\" (UID: \"c4240adb-0390-4339-a752-209bdd4b9b50\") " pod="ai-tenant-e2e-shared-a-w4-501738/auth-test-model-501738-kserve-85cfb4c9cc-xkzq5"
Nov 02 15:33:10.204771 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:33:10.204644    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kwwpg\" (UniqueName: \"kubernetes.io/projected/c4240adb-0390-4339-a752-209bdd4b9b50-kube-api-access-kwwpg\") pod \"auth-test-model-501738-kserve-85cfb4c9cc-xkzq5\" (UID: \"c4240adb-0390-4339-a752-209bdd4b9b50\") " pod="ai-tenant-e2e-shared-a-w4-501738/auth-test-model-501738-kserve-85cfb4c9cc-xkzq5"
Nov 02 15:33:10.204854 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:33:10.204837    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c4240adb-0390-4339-a752-209bdd4b9b50-tmp-dir\") pod \"auth-test-model-501738-kserve-85cfb4c9cc-xkzq5\" (UID: \"c4240adb-0390-4339-a752-209bdd4b9b50\") " pod="ai-tenant-e2e-shared-a-w4-501738/auth-test-model-501738-kserve-85cfb4c9cc-xkzq5"
Nov 02 15:33:10.205180 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:33:10.205161    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c4240adb-0390-4339-a752-209bdd4b9b50-tmp-dir\") pod \"auth-test-model-501738-kserve-85cfb4c9cc-xkzq5\" (UID: \"c4240adb-0390-4339-a752-209bdd4b9b50\") " pod="ai-tenant-e2e-shared-a-w4-501738/auth-test-model-501738-kserve-85cfb4c9cc-xkzq5"
Nov 02 15:33:10.206680 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:33:10.206657    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c4240adb-0390-4339-a752-209bdd4b9b50-dshm\") pod \"auth-test-model-501738-kserve-85cfb4c9cc-xkzq5\" (UID: \"c4240adb-0390-4339-a752-209bdd4b9b50\") " pod="ai-tenant-e2e-shared-a-w4-501738/auth-test-model-501738-kserve-85cfb4c9cc-xkzq5"
Nov 02 15:33:10.207050 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:33:10.207033    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c4240adb-0390-4339-a752-209bdd4b9b50-tls-certs\") pod \"auth-test-model-501738-kserve-85cfb4c9cc-xkzq5\" (UID: \"c4240adb-0390-4339-a752-209bdd4b9b50\") " pod="ai-tenant-e2e-shared-a-w4-501738/auth-test-model-501738-kserve-85cfb4c9cc-xkzq5"
Nov 02 15:33:10.257764 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:33:10.257724    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kwwpg\" (UniqueName: \"kubernetes.io/projected/c4240adb-0390-4339-a752-209bdd4b9b50-kube-api-access-kwwpg\") pod \"auth-test-model-501738-kserve-85cfb4c9cc-xkzq5\" (UID: \"c4240adb-0390-4339-a752-209bdd4b9b50\") " pod="ai-tenant-e2e-shared-a-w4-501738/auth-test-model-501738-kserve-85cfb4c9cc-xkzq5"
Nov 02 15:33:10.378440 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:33:10.378403    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-501738/auth-test-model-501738-kserve-85cfb4c9cc-xkzq5"
Nov 02 15:33:10.785885 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:33:10.785602    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w4-501738/auth-test-model-501738-kserve-85cfb4c9cc-xkzq5"]
Nov 02 15:33:10.787675 ip-10-0-135-138 kubenswrapper[5304]: W1102 15:33:10.787646    5304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc4240adb_0390_4339_a752_209bdd4b9b50.slice/crio-8d3b36d1aed724d6343a57d5c75026a6bb4dcf8a8c758b460ce586851acbf9c4 WatchSource:0}: Error finding container 8d3b36d1aed724d6343a57d5c75026a6bb4dcf8a8c758b460ce586851acbf9c4: Status 404 returned error can't find the container with id 8d3b36d1aed724d6343a57d5c75026a6bb4dcf8a8c758b460ce586851acbf9c4
Nov 02 15:33:11.297563 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:33:11.297527    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-501738/auth-test-model-501738-kserve-85cfb4c9cc-xkzq5" event={"ID":"c4240adb-0390-4339-a752-209bdd4b9b50","Type":"ContainerStarted","Data":"ddf8a0cc36dfc36aa248d0dc8b1655e9b489a379dd0261d665c76d1101b36ca7"}
Nov 02 15:33:11.297998 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:33:11.297572    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-501738/auth-test-model-501738-kserve-85cfb4c9cc-xkzq5" event={"ID":"c4240adb-0390-4339-a752-209bdd4b9b50","Type":"ContainerStarted","Data":"8d3b36d1aed724d6343a57d5c75026a6bb4dcf8a8c758b460ce586851acbf9c4"}
Nov 02 15:33:11.355150 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:33:11.355020    5304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w4-501738/auth-test-model-501738-kserve-85cfb4c9cc-xkzq5" podStartSLOduration=2.355000198 podStartE2EDuration="2.355000198s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:33:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:33:11.353553218 +0000 UTC m=+2191.347377630" watchObservedRunningTime="2026-11-02 15:33:11.355000198 +0000 UTC m=+2191.348824608"
Nov 02 15:33:11.379002 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:33:11.378966    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w4-501738/auth-test-model-501738-kserve-85cfb4c9cc-xkzq5"
Nov 02 15:33:13.566865 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:33:13.566837    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-2abeaa-openshift-default-ccdd5d949-6fmzt"
Nov 02 15:33:13.690699 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:33:13.690664    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa31b68f-cd14-41d9-92ca-2d1df613f263-istio-token\" (UniqueName: \"kubernetes.io/projected/aa31b68f-cd14-41d9-92ca-2d1df613f263-istio-token\") pod \"aa31b68f-cd14-41d9-92ca-2d1df613f263\" (UID: \"aa31b68f-cd14-41d9-92ca-2d1df613f263\") "
Nov 02 15:33:13.690879 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:33:13.690724    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa31b68f-cd14-41d9-92ca-2d1df613f263-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa31b68f-cd14-41d9-92ca-2d1df613f263-workload-certs\") pod \"aa31b68f-cd14-41d9-92ca-2d1df613f263\" (UID: \"aa31b68f-cd14-41d9-92ca-2d1df613f263\") "
Nov 02 15:33:13.690879 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:33:13.690787    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa31b68f-cd14-41d9-92ca-2d1df613f263-kube-api-access-gdscp\" (UniqueName: \"kubernetes.io/projected/aa31b68f-cd14-41d9-92ca-2d1df613f263-kube-api-access-gdscp\") pod \"aa31b68f-cd14-41d9-92ca-2d1df613f263\" (UID: \"aa31b68f-cd14-41d9-92ca-2d1df613f263\") "
Nov 02 15:33:13.690879 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:33:13.690858    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aa31b68f-cd14-41d9-92ca-2d1df613f263-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa31b68f-cd14-41d9-92ca-2d1df613f263-istiod-ca-cert\") pod \"aa31b68f-cd14-41d9-92ca-2d1df613f263\" (UID: \"aa31b68f-cd14-41d9-92ca-2d1df613f263\") "
Nov 02 15:33:13.691034 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:33:13.690894    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa31b68f-cd14-41d9-92ca-2d1df613f263-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa31b68f-cd14-41d9-92ca-2d1df613f263-istio-data\") pod \"aa31b68f-cd14-41d9-92ca-2d1df613f263\" (UID: \"aa31b68f-cd14-41d9-92ca-2d1df613f263\") "
Nov 02 15:33:13.691034 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:33:13.690954    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa31b68f-cd14-41d9-92ca-2d1df613f263-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa31b68f-cd14-41d9-92ca-2d1df613f263-credential-socket\") pod \"aa31b68f-cd14-41d9-92ca-2d1df613f263\" (UID: \"aa31b68f-cd14-41d9-92ca-2d1df613f263\") "
Nov 02 15:33:13.691189 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:33:13.691072    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa31b68f-cd14-41d9-92ca-2d1df613f263-workload-certs" pod "aa31b68f-cd14-41d9-92ca-2d1df613f263" (UID: "aa31b68f-cd14-41d9-92ca-2d1df613f263"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:33:13.691189 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:33:13.691152    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/aa31b68f-cd14-41d9-92ca-2d1df613f263-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa31b68f-cd14-41d9-92ca-2d1df613f263-istio-podinfo\") pod \"aa31b68f-cd14-41d9-92ca-2d1df613f263\" (UID: \"aa31b68f-cd14-41d9-92ca-2d1df613f263\") "
Nov 02 15:33:13.691189 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:33:13.691183    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa31b68f-cd14-41d9-92ca-2d1df613f263-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa31b68f-cd14-41d9-92ca-2d1df613f263-istio-envoy\") pod \"aa31b68f-cd14-41d9-92ca-2d1df613f263\" (UID: \"aa31b68f-cd14-41d9-92ca-2d1df613f263\") "
Nov 02 15:33:13.691362 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:33:13.691248    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa31b68f-cd14-41d9-92ca-2d1df613f263-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa31b68f-cd14-41d9-92ca-2d1df613f263-workload-socket\") pod \"aa31b68f-cd14-41d9-92ca-2d1df613f263\" (UID: \"aa31b68f-cd14-41d9-92ca-2d1df613f263\") "
Nov 02 15:33:13.691412 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:33:13.691352    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa31b68f-cd14-41d9-92ca-2d1df613f263-istiod-ca-cert" pod "aa31b68f-cd14-41d9-92ca-2d1df613f263" (UID: "aa31b68f-cd14-41d9-92ca-2d1df613f263"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:33:13.691686 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:33:13.691657    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa31b68f-cd14-41d9-92ca-2d1df613f263-istio-data" pod "aa31b68f-cd14-41d9-92ca-2d1df613f263" (UID: "aa31b68f-cd14-41d9-92ca-2d1df613f263"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:33:13.691788 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:33:13.691747    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa31b68f-cd14-41d9-92ca-2d1df613f263-credential-socket" pod "aa31b68f-cd14-41d9-92ca-2d1df613f263" (UID: "aa31b68f-cd14-41d9-92ca-2d1df613f263"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:33:13.691788 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:33:13.691763    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa31b68f-cd14-41d9-92ca-2d1df613f263-workload-socket" pod "aa31b68f-cd14-41d9-92ca-2d1df613f263" (UID: "aa31b68f-cd14-41d9-92ca-2d1df613f263"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:33:13.691904 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:33:13.691886    5304 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa31b68f-cd14-41d9-92ca-2d1df613f263-istiod-ca-cert\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:33:13.691959 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:33:13.691913    5304 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa31b68f-cd14-41d9-92ca-2d1df613f263-istio-data\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:33:13.691959 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:33:13.691927    5304 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa31b68f-cd14-41d9-92ca-2d1df613f263-workload-certs\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:33:13.693561 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:33:13.693534    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa31b68f-cd14-41d9-92ca-2d1df613f263-istio-token" pod "aa31b68f-cd14-41d9-92ca-2d1df613f263" (UID: "aa31b68f-cd14-41d9-92ca-2d1df613f263"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:33:13.693770 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:33:13.693747    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/aa31b68f-cd14-41d9-92ca-2d1df613f263-istio-podinfo" pod "aa31b68f-cd14-41d9-92ca-2d1df613f263" (UID: "aa31b68f-cd14-41d9-92ca-2d1df613f263"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 15:33:13.693975 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:33:13.693948    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa31b68f-cd14-41d9-92ca-2d1df613f263-kube-api-access-gdscp" pod "aa31b68f-cd14-41d9-92ca-2d1df613f263" (UID: "aa31b68f-cd14-41d9-92ca-2d1df613f263"). InnerVolumeSpecName "kube-api-access-gdscp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:33:13.694052 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:33:13.694031    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa31b68f-cd14-41d9-92ca-2d1df613f263-istio-envoy" pod "aa31b68f-cd14-41d9-92ca-2d1df613f263" (UID: "aa31b68f-cd14-41d9-92ca-2d1df613f263"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:33:13.793568 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:33:13.793495    5304 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa31b68f-cd14-41d9-92ca-2d1df613f263-credential-socket\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:33:13.793568 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:33:13.793521    5304 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa31b68f-cd14-41d9-92ca-2d1df613f263-istio-podinfo\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:33:13.793568 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:33:13.793530    5304 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa31b68f-cd14-41d9-92ca-2d1df613f263-istio-envoy\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:33:13.793568 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:33:13.793539    5304 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa31b68f-cd14-41d9-92ca-2d1df613f263-workload-socket\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:33:13.793568 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:33:13.793547    5304 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa31b68f-cd14-41d9-92ca-2d1df613f263-istio-token\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:33:13.793568 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:33:13.793556    5304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gdscp\" (UniqueName: \"kubernetes.io/projected/aa31b68f-cd14-41d9-92ca-2d1df613f263-kube-api-access-gdscp\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:33:14.319914 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:33:14.319872    5304 generic.go:356] "Generic (PLEG): container finished" podID="aa31b68f-cd14-41d9-92ca-2d1df613f263" containerID="cacbcf86b32c08d520cac3f78168b566738508a647c8224c74f5c07655bf3b7e" exitCode=0
Nov 02 15:33:14.320096 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:33:14.319933    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-2abeaa-openshift-default-ccdd5d949-6fmzt"
Nov 02 15:33:14.320096 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:33:14.319954    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-2abeaa-openshift-default-ccdd5d949-6fmzt" event={"ID":"aa31b68f-cd14-41d9-92ca-2d1df613f263","Type":"ContainerDied","Data":"cacbcf86b32c08d520cac3f78168b566738508a647c8224c74f5c07655bf3b7e"}
Nov 02 15:33:14.320096 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:33:14.320011    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-2abeaa-openshift-default-ccdd5d949-6fmzt" event={"ID":"aa31b68f-cd14-41d9-92ca-2d1df613f263","Type":"ContainerDied","Data":"d34aee5b205baeb0b7a7c30e51622fdd30adc7e9d6f88fca828b4a7b38041b2a"}
Nov 02 15:33:14.320096 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:33:14.320036    5304 scope.go:122] "RemoveContainer" containerID="cacbcf86b32c08d520cac3f78168b566738508a647c8224c74f5c07655bf3b7e"
Nov 02 15:33:14.332033 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:33:14.332006    5304 scope.go:122] "RemoveContainer" containerID="cacbcf86b32c08d520cac3f78168b566738508a647c8224c74f5c07655bf3b7e"
Nov 02 15:33:14.332398 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:33:14.332379    5304 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cacbcf86b32c08d520cac3f78168b566738508a647c8224c74f5c07655bf3b7e\": container with ID starting with cacbcf86b32c08d520cac3f78168b566738508a647c8224c74f5c07655bf3b7e not found: ID does not exist" containerID="cacbcf86b32c08d520cac3f78168b566738508a647c8224c74f5c07655bf3b7e"
Nov 02 15:33:14.332453 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:33:14.332408    5304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cacbcf86b32c08d520cac3f78168b566738508a647c8224c74f5c07655bf3b7e"} err="failed to get container status \"cacbcf86b32c08d520cac3f78168b566738508a647c8224c74f5c07655bf3b7e\": rpc error: code = NotFound desc = could not find container \"cacbcf86b32c08d520cac3f78168b566738508a647c8224c74f5c07655bf3b7e\": container with ID starting with cacbcf86b32c08d520cac3f78168b566738508a647c8224c74f5c07655bf3b7e not found: ID does not exist"
Nov 02 15:33:14.411732 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:33:14.411647    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-2abeaa-openshift-default-ccdd5d949-6fmzt"]
Nov 02 15:33:14.421846 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:33:14.421809    5304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-2abeaa-openshift-default-ccdd5d949-6fmzt"]
Nov 02 15:33:16.122337 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:33:16.122303    5304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aa31b68f-cd14-41d9-92ca-2d1df613f263" path="/var/lib/kubelet/pods/aa31b68f-cd14-41d9-92ca-2d1df613f263/volumes"
Nov 02 15:33:16.148307 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:33:16.148269    5304 patch_prober.go:29] interesting pod/e2e-shared-b-w6-b55da1-openshift-default-787ff9f87c-cmxhv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.147:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 15:33:16.148483 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:33:16.148328    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w6-b55da1-openshift-default-787ff9f87c-cmxhv" podUID="67da2b58-a0f0-44f6-9258-b418a21e73bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.147:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:33:17.345245 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:33:17.345212    5304 generic.go:356] "Generic (PLEG): container finished" podID="67da2b58-a0f0-44f6-9258-b418a21e73bc" containerID="aa7cb5d0093ee870a139ebc974db546ad5b647c091f54f48cddf10d8ddfc7646" exitCode=0
Nov 02 15:33:17.345700 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:33:17.345249    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-b55da1-openshift-default-787ff9f87c-cmxhv" event={"ID":"67da2b58-a0f0-44f6-9258-b418a21e73bc","Type":"ContainerDied","Data":"aa7cb5d0093ee870a139ebc974db546ad5b647c091f54f48cddf10d8ddfc7646"}
Nov 02 15:33:17.345700 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:33:17.345690    5304 scope.go:122] "RemoveContainer" containerID="aa7cb5d0093ee870a139ebc974db546ad5b647c091f54f48cddf10d8ddfc7646"
Nov 02 15:33:17.347635 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:33:17.347602    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:33:17.347728 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:33:17.347673    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:33:17.347728 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:33:17.347706    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:33:18.359224 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:33:18.359189    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-b55da1-openshift-default-787ff9f87c-cmxhv" event={"ID":"67da2b58-a0f0-44f6-9258-b418a21e73bc","Type":"ContainerStarted","Data":"efa3a256bfcda39d706219796dc57612df582f4045ec78771cb86c6cd80eb067"}
Nov 02 15:33:18.388853 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:33:18.388821    5304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w6-b55da1-openshift-default-787ff9f87c-cmxhv"
Nov 02 15:33:18.389075 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:33:18.388958    5304 patch_prober.go:29] interesting pod/e2e-shared-b-w6-b55da1-openshift-default-787ff9f87c-cmxhv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" start-of-body=
Nov 02 15:33:18.389075 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:33:18.389025    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-b55da1-openshift-default-787ff9f87c-cmxhv" podUID="67da2b58-a0f0-44f6-9258-b418a21e73bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused"
Nov 02 15:33:19.389105 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:33:19.389067    5304 patch_prober.go:29] interesting pod/e2e-shared-b-w6-b55da1-openshift-default-787ff9f87c-cmxhv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" start-of-body=
Nov 02 15:33:19.389509 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:33:19.389155    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-b55da1-openshift-default-787ff9f87c-cmxhv" podUID="67da2b58-a0f0-44f6-9258-b418a21e73bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused"
Nov 02 15:33:20.378945 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:33:20.378906    5304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w4-501738/auth-test-model-501738-kserve-85cfb4c9cc-xkzq5"
Nov 02 15:33:20.388167 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:33:20.388111    5304 patch_prober.go:29] interesting pod/e2e-shared-b-w6-b55da1-openshift-default-787ff9f87c-cmxhv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" start-of-body=
Nov 02 15:33:20.388332 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:33:20.388201    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-b55da1-openshift-default-787ff9f87c-cmxhv" podUID="67da2b58-a0f0-44f6-9258-b418a21e73bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused"
Nov 02 15:33:20.391974 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:33:20.391951    5304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w4-501738/auth-test-model-501738-kserve-85cfb4c9cc-xkzq5"
Nov 02 15:33:21.388228 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:33:21.388191    5304 patch_prober.go:29] interesting pod/e2e-shared-b-w6-b55da1-openshift-default-787ff9f87c-cmxhv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" start-of-body=
Nov 02 15:33:21.388440 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:33:21.388247    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-b55da1-openshift-default-787ff9f87c-cmxhv" podUID="67da2b58-a0f0-44f6-9258-b418a21e73bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused"
Nov 02 15:33:21.390699 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:33:21.390675    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w4-501738/auth-test-model-501738-kserve-85cfb4c9cc-xkzq5"
Nov 02 15:33:22.388933 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:33:22.388884    5304 patch_prober.go:29] interesting pod/e2e-shared-b-w6-b55da1-openshift-default-787ff9f87c-cmxhv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" start-of-body=
Nov 02 15:33:22.389373 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:33:22.388965    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-b55da1-openshift-default-787ff9f87c-cmxhv" podUID="67da2b58-a0f0-44f6-9258-b418a21e73bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused"
Nov 02 15:33:23.388857 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:33:23.388821    5304 patch_prober.go:29] interesting pod/e2e-shared-b-w6-b55da1-openshift-default-787ff9f87c-cmxhv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" start-of-body=
Nov 02 15:33:23.389097 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:33:23.388897    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-b55da1-openshift-default-787ff9f87c-cmxhv" podUID="67da2b58-a0f0-44f6-9258-b418a21e73bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused"
Nov 02 15:33:24.393478 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:33:24.393446    5304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w6-b55da1-openshift-default-787ff9f87c-cmxhv"
Nov 02 15:33:24.393973 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:33:24.393937    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w6-b55da1-openshift-default-787ff9f87c-cmxhv"
Nov 02 15:33:24.394681 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:33:24.394659    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w6-b55da1-openshift-default-787ff9f87c-cmxhv"
Nov 02 15:33:24.799572 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:33:24.799481    5304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k6fjk" containerName="extractor"
Nov 02 15:33:38.134169 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:33:38.134137    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-3daba5-openshift-default-59d44c74df-gjqj2"]
Nov 02 15:33:38.134628 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:33:38.134373    5304 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-3daba5-openshift-default-59d44c74df-gjqj2" podUID="ea04b74e-0059-4462-9b89-45069a0093cd" containerName="istio-proxy" containerID="cri-o://787771bdd1a185240d2b2b835d094eb6060e35a7a71bffd9517d61f8e4fa8c81" gracePeriod=30
Nov 02 15:33:43.410926 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:33:43.410863    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-3daba5-openshift-default-59d44c74df-gjqj2"
Nov 02 15:33:43.542441 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:33:43.542408    5304 generic.go:356] "Generic (PLEG): container finished" podID="ea04b74e-0059-4462-9b89-45069a0093cd" containerID="787771bdd1a185240d2b2b835d094eb6060e35a7a71bffd9517d61f8e4fa8c81" exitCode=0
Nov 02 15:33:43.542607 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:33:43.542457    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-3daba5-openshift-default-59d44c74df-gjqj2"
Nov 02 15:33:43.542607 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:33:43.542490    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-3daba5-openshift-default-59d44c74df-gjqj2" event={"ID":"ea04b74e-0059-4462-9b89-45069a0093cd","Type":"ContainerDied","Data":"787771bdd1a185240d2b2b835d094eb6060e35a7a71bffd9517d61f8e4fa8c81"}
Nov 02 15:33:43.542607 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:33:43.542547    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-3daba5-openshift-default-59d44c74df-gjqj2" event={"ID":"ea04b74e-0059-4462-9b89-45069a0093cd","Type":"ContainerDied","Data":"09f3b38ad06ace0567c1dde1b3b549309be38ed0ae51339dc960e1a7c653df90"}
Nov 02 15:33:43.542607 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:33:43.542571    5304 scope.go:122] "RemoveContainer" containerID="787771bdd1a185240d2b2b835d094eb6060e35a7a71bffd9517d61f8e4fa8c81"
Nov 02 15:33:43.552494 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:33:43.552467    5304 scope.go:122] "RemoveContainer" containerID="787771bdd1a185240d2b2b835d094eb6060e35a7a71bffd9517d61f8e4fa8c81"
Nov 02 15:33:43.552776 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:33:43.552757    5304 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"787771bdd1a185240d2b2b835d094eb6060e35a7a71bffd9517d61f8e4fa8c81\": container with ID starting with 787771bdd1a185240d2b2b835d094eb6060e35a7a71bffd9517d61f8e4fa8c81 not found: ID does not exist" containerID="787771bdd1a185240d2b2b835d094eb6060e35a7a71bffd9517d61f8e4fa8c81"
Nov 02 15:33:43.552836 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:33:43.552787    5304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"787771bdd1a185240d2b2b835d094eb6060e35a7a71bffd9517d61f8e4fa8c81"} err="failed to get container status \"787771bdd1a185240d2b2b835d094eb6060e35a7a71bffd9517d61f8e4fa8c81\": rpc error: code = NotFound desc = could not find container \"787771bdd1a185240d2b2b835d094eb6060e35a7a71bffd9517d61f8e4fa8c81\": container with ID starting with 787771bdd1a185240d2b2b835d094eb6060e35a7a71bffd9517d61f8e4fa8c81 not found: ID does not exist"
Nov 02 15:33:43.555317 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:33:43.555299    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea04b74e-0059-4462-9b89-45069a0093cd-kube-api-access-9jzrp\" (UniqueName: \"kubernetes.io/projected/ea04b74e-0059-4462-9b89-45069a0093cd-kube-api-access-9jzrp\") pod \"ea04b74e-0059-4462-9b89-45069a0093cd\" (UID: \"ea04b74e-0059-4462-9b89-45069a0093cd\") "
Nov 02 15:33:43.555396 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:33:43.555381    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea04b74e-0059-4462-9b89-45069a0093cd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea04b74e-0059-4462-9b89-45069a0093cd-workload-certs\") pod \"ea04b74e-0059-4462-9b89-45069a0093cd\" (UID: \"ea04b74e-0059-4462-9b89-45069a0093cd\") "
Nov 02 15:33:43.555450 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:33:43.555430    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea04b74e-0059-4462-9b89-45069a0093cd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea04b74e-0059-4462-9b89-45069a0093cd-credential-socket\") pod \"ea04b74e-0059-4462-9b89-45069a0093cd\" (UID: \"ea04b74e-0059-4462-9b89-45069a0093cd\") "
Nov 02 15:33:43.555549 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:33:43.555496    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ea04b74e-0059-4462-9b89-45069a0093cd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea04b74e-0059-4462-9b89-45069a0093cd-istio-podinfo\") pod \"ea04b74e-0059-4462-9b89-45069a0093cd\" (UID: \"ea04b74e-0059-4462-9b89-45069a0093cd\") "
Nov 02 15:33:43.555602 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:33:43.555546    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea04b74e-0059-4462-9b89-45069a0093cd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea04b74e-0059-4462-9b89-45069a0093cd-istio-data\") pod \"ea04b74e-0059-4462-9b89-45069a0093cd\" (UID: \"ea04b74e-0059-4462-9b89-45069a0093cd\") "
Nov 02 15:33:43.555602 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:33:43.555572    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ea04b74e-0059-4462-9b89-45069a0093cd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea04b74e-0059-4462-9b89-45069a0093cd-istiod-ca-cert\") pod \"ea04b74e-0059-4462-9b89-45069a0093cd\" (UID: \"ea04b74e-0059-4462-9b89-45069a0093cd\") "
Nov 02 15:33:43.555712 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:33:43.555659    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea04b74e-0059-4462-9b89-45069a0093cd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea04b74e-0059-4462-9b89-45069a0093cd-workload-socket\") pod \"ea04b74e-0059-4462-9b89-45069a0093cd\" (UID: \"ea04b74e-0059-4462-9b89-45069a0093cd\") "
Nov 02 15:33:43.555712 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:33:43.555678    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea04b74e-0059-4462-9b89-45069a0093cd-workload-certs" pod "ea04b74e-0059-4462-9b89-45069a0093cd" (UID: "ea04b74e-0059-4462-9b89-45069a0093cd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:33:43.555829 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:33:43.555723    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea04b74e-0059-4462-9b89-45069a0093cd-credential-socket" pod "ea04b74e-0059-4462-9b89-45069a0093cd" (UID: "ea04b74e-0059-4462-9b89-45069a0093cd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:33:43.555829 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:33:43.555773    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea04b74e-0059-4462-9b89-45069a0093cd-istio-token\" (UniqueName: \"kubernetes.io/projected/ea04b74e-0059-4462-9b89-45069a0093cd-istio-token\") pod \"ea04b74e-0059-4462-9b89-45069a0093cd\" (UID: \"ea04b74e-0059-4462-9b89-45069a0093cd\") "
Nov 02 15:33:43.555829 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:33:43.555806    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea04b74e-0059-4462-9b89-45069a0093cd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea04b74e-0059-4462-9b89-45069a0093cd-istio-envoy\") pod \"ea04b74e-0059-4462-9b89-45069a0093cd\" (UID: \"ea04b74e-0059-4462-9b89-45069a0093cd\") "
Nov 02 15:33:43.555987 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:33:43.555841    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea04b74e-0059-4462-9b89-45069a0093cd-istio-data" pod "ea04b74e-0059-4462-9b89-45069a0093cd" (UID: "ea04b74e-0059-4462-9b89-45069a0093cd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:33:43.555987 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:33:43.555953    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea04b74e-0059-4462-9b89-45069a0093cd-workload-socket" pod "ea04b74e-0059-4462-9b89-45069a0093cd" (UID: "ea04b74e-0059-4462-9b89-45069a0093cd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:33:43.556061 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:33:43.555999    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea04b74e-0059-4462-9b89-45069a0093cd-istiod-ca-cert" pod "ea04b74e-0059-4462-9b89-45069a0093cd" (UID: "ea04b74e-0059-4462-9b89-45069a0093cd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:33:43.556527 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:33:43.556494    5304 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea04b74e-0059-4462-9b89-45069a0093cd-workload-certs\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:33:43.556636 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:33:43.556531    5304 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea04b74e-0059-4462-9b89-45069a0093cd-credential-socket\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:33:43.556636 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:33:43.556546    5304 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea04b74e-0059-4462-9b89-45069a0093cd-istio-data\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:33:43.556636 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:33:43.556560    5304 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea04b74e-0059-4462-9b89-45069a0093cd-istiod-ca-cert\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:33:43.556636 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:33:43.556575    5304 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea04b74e-0059-4462-9b89-45069a0093cd-workload-socket\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:33:43.557646 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:33:43.557617    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea04b74e-0059-4462-9b89-45069a0093cd-kube-api-access-9jzrp" pod "ea04b74e-0059-4462-9b89-45069a0093cd" (UID: "ea04b74e-0059-4462-9b89-45069a0093cd"). InnerVolumeSpecName "kube-api-access-9jzrp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:33:43.557851 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:33:43.557834    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea04b74e-0059-4462-9b89-45069a0093cd-istio-envoy" pod "ea04b74e-0059-4462-9b89-45069a0093cd" (UID: "ea04b74e-0059-4462-9b89-45069a0093cd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:33:43.557998 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:33:43.557985    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea04b74e-0059-4462-9b89-45069a0093cd-istio-token" pod "ea04b74e-0059-4462-9b89-45069a0093cd" (UID: "ea04b74e-0059-4462-9b89-45069a0093cd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:33:43.558079 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:33:43.558067    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ea04b74e-0059-4462-9b89-45069a0093cd-istio-podinfo" pod "ea04b74e-0059-4462-9b89-45069a0093cd" (UID: "ea04b74e-0059-4462-9b89-45069a0093cd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 15:33:43.657910 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:33:43.657876    5304 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea04b74e-0059-4462-9b89-45069a0093cd-istio-podinfo\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:33:43.657910 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:33:43.657904    5304 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea04b74e-0059-4462-9b89-45069a0093cd-istio-token\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:33:43.657910 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:33:43.657913    5304 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea04b74e-0059-4462-9b89-45069a0093cd-istio-envoy\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:33:43.658170 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:33:43.657922    5304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9jzrp\" (UniqueName: \"kubernetes.io/projected/ea04b74e-0059-4462-9b89-45069a0093cd-kube-api-access-9jzrp\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:33:43.892039 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:33:43.892004    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-3daba5-openshift-default-59d44c74df-gjqj2"]
Nov 02 15:33:43.904648 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:33:43.904610    5304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-3daba5-openshift-default-59d44c74df-gjqj2"]
Nov 02 15:33:43.912749 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:33:43.912715    5304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w6-b955c5/auth-test-model-b955c5-kserve-7965dcb68f-75hkb"]
Nov 02 15:33:43.913407 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:33:43.913378    5304 cpu_manager.go:414] "containerMap: removing container" podUID="aa31b68f-cd14-41d9-92ca-2d1df613f263" containerName="istio-proxy"
Nov 02 15:33:43.913407 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:33:43.913398    5304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa31b68f-cd14-41d9-92ca-2d1df613f263" containerName="istio-proxy"
Nov 02 15:33:43.913596 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:33:43.913420    5304 cpu_manager.go:414] "containerMap: removing container" podUID="c109588b-9544-4a37-bb13-d81572067546" containerName="istio-proxy"
Nov 02 15:33:43.913596 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:33:43.913428    5304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c109588b-9544-4a37-bb13-d81572067546" containerName="istio-proxy"
Nov 02 15:33:43.913596 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:33:43.913460    5304 cpu_manager.go:414] "containerMap: removing container" podUID="ea04b74e-0059-4462-9b89-45069a0093cd" containerName="istio-proxy"
Nov 02 15:33:43.913596 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:33:43.913469    5304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea04b74e-0059-4462-9b89-45069a0093cd" containerName="istio-proxy"
Nov 02 15:33:43.913789 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:33:43.913642    5304 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa31b68f-cd14-41d9-92ca-2d1df613f263" containerName="istio-proxy"
Nov 02 15:33:43.913789 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:33:43.913659    5304 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea04b74e-0059-4462-9b89-45069a0093cd" containerName="istio-proxy"
Nov 02 15:33:43.918584 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:33:43.918561    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w6-b955c5/auth-test-model-b955c5-kserve-7965dcb68f-75hkb"
Nov 02 15:33:43.933589 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:33:43.933556    5304 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w6-b955c5\"/\"kube-root-ca.crt\""
Nov 02 15:33:43.933772 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:33:43.933754    5304 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w6-b955c5\"/\"openshift-service-ca.crt\""
Nov 02 15:33:43.934069 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:33:43.934029    5304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w6-b955c5\"/\"default-dockercfg-8pnkw\""
Nov 02 15:33:43.934069 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:33:43.934069    5304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w6-b955c5\"/\"auth-test-model-b955c5-kserve-self-signed-certs\""
Nov 02 15:33:43.938547 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:33:43.938520    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w6-b955c5/auth-test-model-b955c5-kserve-7965dcb68f-75hkb"]
Nov 02 15:33:44.063541 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:33:44.063505    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/61fd91e0-c9e9-4336-b47c-81f9e2799829-home\") pod \"auth-test-model-b955c5-kserve-7965dcb68f-75hkb\" (UID: \"61fd91e0-c9e9-4336-b47c-81f9e2799829\") " pod="ai-tenant-e2e-shared-a-w6-b955c5/auth-test-model-b955c5-kserve-7965dcb68f-75hkb"
Nov 02 15:33:44.063729 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:33:44.063570    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/61fd91e0-c9e9-4336-b47c-81f9e2799829-tls-certs\") pod \"auth-test-model-b955c5-kserve-7965dcb68f-75hkb\" (UID: \"61fd91e0-c9e9-4336-b47c-81f9e2799829\") " pod="ai-tenant-e2e-shared-a-w6-b955c5/auth-test-model-b955c5-kserve-7965dcb68f-75hkb"
Nov 02 15:33:44.063729 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:33:44.063702    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/61fd91e0-c9e9-4336-b47c-81f9e2799829-tmp-dir\") pod \"auth-test-model-b955c5-kserve-7965dcb68f-75hkb\" (UID: \"61fd91e0-c9e9-4336-b47c-81f9e2799829\") " pod="ai-tenant-e2e-shared-a-w6-b955c5/auth-test-model-b955c5-kserve-7965dcb68f-75hkb"
Nov 02 15:33:44.063817 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:33:44.063738    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/61fd91e0-c9e9-4336-b47c-81f9e2799829-model-cache\") pod \"auth-test-model-b955c5-kserve-7965dcb68f-75hkb\" (UID: \"61fd91e0-c9e9-4336-b47c-81f9e2799829\") " pod="ai-tenant-e2e-shared-a-w6-b955c5/auth-test-model-b955c5-kserve-7965dcb68f-75hkb"
Nov 02 15:33:44.063901 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:33:44.063886    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/61fd91e0-c9e9-4336-b47c-81f9e2799829-dshm\") pod \"auth-test-model-b955c5-kserve-7965dcb68f-75hkb\" (UID: \"61fd91e0-c9e9-4336-b47c-81f9e2799829\") " pod="ai-tenant-e2e-shared-a-w6-b955c5/auth-test-model-b955c5-kserve-7965dcb68f-75hkb"
Nov 02 15:33:44.063970 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:33:44.063953    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xhjwg\" (UniqueName: \"kubernetes.io/projected/61fd91e0-c9e9-4336-b47c-81f9e2799829-kube-api-access-xhjwg\") pod \"auth-test-model-b955c5-kserve-7965dcb68f-75hkb\" (UID: \"61fd91e0-c9e9-4336-b47c-81f9e2799829\") " pod="ai-tenant-e2e-shared-a-w6-b955c5/auth-test-model-b955c5-kserve-7965dcb68f-75hkb"
Nov 02 15:33:44.120799 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:33:44.120726    5304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ea04b74e-0059-4462-9b89-45069a0093cd" path="/var/lib/kubelet/pods/ea04b74e-0059-4462-9b89-45069a0093cd/volumes"
Nov 02 15:33:44.178147 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:33:44.176190    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xhjwg\" (UniqueName: \"kubernetes.io/projected/61fd91e0-c9e9-4336-b47c-81f9e2799829-kube-api-access-xhjwg\") pod \"auth-test-model-b955c5-kserve-7965dcb68f-75hkb\" (UID: \"61fd91e0-c9e9-4336-b47c-81f9e2799829\") " pod="ai-tenant-e2e-shared-a-w6-b955c5/auth-test-model-b955c5-kserve-7965dcb68f-75hkb"
Nov 02 15:33:44.178147 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:33:44.176272    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/61fd91e0-c9e9-4336-b47c-81f9e2799829-home\") pod \"auth-test-model-b955c5-kserve-7965dcb68f-75hkb\" (UID: \"61fd91e0-c9e9-4336-b47c-81f9e2799829\") " pod="ai-tenant-e2e-shared-a-w6-b955c5/auth-test-model-b955c5-kserve-7965dcb68f-75hkb"
Nov 02 15:33:44.178147 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:33:44.176361    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/61fd91e0-c9e9-4336-b47c-81f9e2799829-tls-certs\") pod \"auth-test-model-b955c5-kserve-7965dcb68f-75hkb\" (UID: \"61fd91e0-c9e9-4336-b47c-81f9e2799829\") " pod="ai-tenant-e2e-shared-a-w6-b955c5/auth-test-model-b955c5-kserve-7965dcb68f-75hkb"
Nov 02 15:33:44.178147 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:33:44.176471    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/61fd91e0-c9e9-4336-b47c-81f9e2799829-tmp-dir\") pod \"auth-test-model-b955c5-kserve-7965dcb68f-75hkb\" (UID: \"61fd91e0-c9e9-4336-b47c-81f9e2799829\") " pod="ai-tenant-e2e-shared-a-w6-b955c5/auth-test-model-b955c5-kserve-7965dcb68f-75hkb"
Nov 02 15:33:44.178147 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:33:44.176517    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/61fd91e0-c9e9-4336-b47c-81f9e2799829-model-cache\") pod \"auth-test-model-b955c5-kserve-7965dcb68f-75hkb\" (UID: \"61fd91e0-c9e9-4336-b47c-81f9e2799829\") " pod="ai-tenant-e2e-shared-a-w6-b955c5/auth-test-model-b955c5-kserve-7965dcb68f-75hkb"
Nov 02 15:33:44.178147 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:33:44.176648    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/61fd91e0-c9e9-4336-b47c-81f9e2799829-dshm\") pod \"auth-test-model-b955c5-kserve-7965dcb68f-75hkb\" (UID: \"61fd91e0-c9e9-4336-b47c-81f9e2799829\") " pod="ai-tenant-e2e-shared-a-w6-b955c5/auth-test-model-b955c5-kserve-7965dcb68f-75hkb"
Nov 02 15:33:44.180363 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:33:44.178845    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/61fd91e0-c9e9-4336-b47c-81f9e2799829-model-cache\") pod \"auth-test-model-b955c5-kserve-7965dcb68f-75hkb\" (UID: \"61fd91e0-c9e9-4336-b47c-81f9e2799829\") " pod="ai-tenant-e2e-shared-a-w6-b955c5/auth-test-model-b955c5-kserve-7965dcb68f-75hkb"
Nov 02 15:33:44.180363 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:33:44.179103    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/61fd91e0-c9e9-4336-b47c-81f9e2799829-home\") pod \"auth-test-model-b955c5-kserve-7965dcb68f-75hkb\" (UID: \"61fd91e0-c9e9-4336-b47c-81f9e2799829\") " pod="ai-tenant-e2e-shared-a-w6-b955c5/auth-test-model-b955c5-kserve-7965dcb68f-75hkb"
Nov 02 15:33:44.180363 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:33:44.179418    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/61fd91e0-c9e9-4336-b47c-81f9e2799829-tmp-dir\") pod \"auth-test-model-b955c5-kserve-7965dcb68f-75hkb\" (UID: \"61fd91e0-c9e9-4336-b47c-81f9e2799829\") " pod="ai-tenant-e2e-shared-a-w6-b955c5/auth-test-model-b955c5-kserve-7965dcb68f-75hkb"
Nov 02 15:33:44.195834 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:33:44.195796    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/61fd91e0-c9e9-4336-b47c-81f9e2799829-dshm\") pod \"auth-test-model-b955c5-kserve-7965dcb68f-75hkb\" (UID: \"61fd91e0-c9e9-4336-b47c-81f9e2799829\") " pod="ai-tenant-e2e-shared-a-w6-b955c5/auth-test-model-b955c5-kserve-7965dcb68f-75hkb"
Nov 02 15:33:44.213646 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:33:44.205205    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xhjwg\" (UniqueName: \"kubernetes.io/projected/61fd91e0-c9e9-4336-b47c-81f9e2799829-kube-api-access-xhjwg\") pod \"auth-test-model-b955c5-kserve-7965dcb68f-75hkb\" (UID: \"61fd91e0-c9e9-4336-b47c-81f9e2799829\") " pod="ai-tenant-e2e-shared-a-w6-b955c5/auth-test-model-b955c5-kserve-7965dcb68f-75hkb"
Nov 02 15:33:44.222145 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:33:44.221835    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/61fd91e0-c9e9-4336-b47c-81f9e2799829-tls-certs\") pod \"auth-test-model-b955c5-kserve-7965dcb68f-75hkb\" (UID: \"61fd91e0-c9e9-4336-b47c-81f9e2799829\") " pod="ai-tenant-e2e-shared-a-w6-b955c5/auth-test-model-b955c5-kserve-7965dcb68f-75hkb"
Nov 02 15:33:44.235601 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:33:44.235564    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w6-b955c5/auth-test-model-b955c5-kserve-7965dcb68f-75hkb"
Nov 02 15:33:44.552973 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:33:44.552882    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w6-b955c5/auth-test-model-b955c5-kserve-7965dcb68f-75hkb"]
Nov 02 15:33:44.556245 ip-10-0-135-138 kubenswrapper[5304]: W1102 15:33:44.556214    5304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod61fd91e0_c9e9_4336_b47c_81f9e2799829.slice/crio-cc0da4b8b6b71f7e6e9dd4313cf201a74b3688e300789d8c2564705fe9ef4814 WatchSource:0}: Error finding container cc0da4b8b6b71f7e6e9dd4313cf201a74b3688e300789d8c2564705fe9ef4814: Status 404 returned error can't find the container with id cc0da4b8b6b71f7e6e9dd4313cf201a74b3688e300789d8c2564705fe9ef4814
Nov 02 15:33:45.560811 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:33:45.560775    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w6-b955c5/auth-test-model-b955c5-kserve-7965dcb68f-75hkb" event={"ID":"61fd91e0-c9e9-4336-b47c-81f9e2799829","Type":"ContainerStarted","Data":"c49f2acccdf4a9e0984df2047dd2c9c747bf749d124acdc5570d94cf4b56949b"}
Nov 02 15:33:45.560811 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:33:45.560818    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w6-b955c5/auth-test-model-b955c5-kserve-7965dcb68f-75hkb" event={"ID":"61fd91e0-c9e9-4336-b47c-81f9e2799829","Type":"ContainerStarted","Data":"cc0da4b8b6b71f7e6e9dd4313cf201a74b3688e300789d8c2564705fe9ef4814"}
Nov 02 15:33:45.603463 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:33:45.603409    5304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w6-b955c5/auth-test-model-b955c5-kserve-7965dcb68f-75hkb" podStartSLOduration=2.603392278 podStartE2EDuration="2.603392278s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:33:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:33:45.601989946 +0000 UTC m=+2225.595814355" watchObservedRunningTime="2026-11-02 15:33:45.603392278 +0000 UTC m=+2225.597216689"
Nov 02 15:33:46.236281 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:33:46.236252    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w6-b955c5/auth-test-model-b955c5-kserve-7965dcb68f-75hkb"
Nov 02 15:33:46.402537 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:33:46.402501    5304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w1-fda646/auth-test-model-fda646-kserve-568f9ddd9-t6lxf"]
Nov 02 15:33:46.406636 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:33:46.406611    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-fda646/auth-test-model-fda646-kserve-568f9ddd9-t6lxf"
Nov 02 15:33:46.427194 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:33:46.427161    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w1-fda646/auth-test-model-fda646-kserve-568f9ddd9-t6lxf"]
Nov 02 15:33:46.435624 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:33:46.435594    5304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w1-fda646\"/\"default-dockercfg-sn8s6\""
Nov 02 15:33:46.435805 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:33:46.435711    5304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w1-fda646\"/\"auth-test-model-fda646-kserve-self-signed-certs\""
Nov 02 15:33:46.444522 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:33:46.444480    5304 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w1-fda646\"/\"openshift-service-ca.crt\""
Nov 02 15:33:46.444834 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:33:46.444816    5304 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w1-fda646\"/\"kube-root-ca.crt\""
Nov 02 15:33:46.509221 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:33:46.509103    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7e84c4e2-b5e2-4ba8-a6e8-8090a06ac68c-home\") pod \"auth-test-model-fda646-kserve-568f9ddd9-t6lxf\" (UID: \"7e84c4e2-b5e2-4ba8-a6e8-8090a06ac68c\") " pod="ai-tenant-e2e-shared-a-w1-fda646/auth-test-model-fda646-kserve-568f9ddd9-t6lxf"
Nov 02 15:33:46.509355 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:33:46.509317    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7e84c4e2-b5e2-4ba8-a6e8-8090a06ac68c-model-cache\") pod \"auth-test-model-fda646-kserve-568f9ddd9-t6lxf\" (UID: \"7e84c4e2-b5e2-4ba8-a6e8-8090a06ac68c\") " pod="ai-tenant-e2e-shared-a-w1-fda646/auth-test-model-fda646-kserve-568f9ddd9-t6lxf"
Nov 02 15:33:46.509427 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:33:46.509412    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xh978\" (UniqueName: \"kubernetes.io/projected/7e84c4e2-b5e2-4ba8-a6e8-8090a06ac68c-kube-api-access-xh978\") pod \"auth-test-model-fda646-kserve-568f9ddd9-t6lxf\" (UID: \"7e84c4e2-b5e2-4ba8-a6e8-8090a06ac68c\") " pod="ai-tenant-e2e-shared-a-w1-fda646/auth-test-model-fda646-kserve-568f9ddd9-t6lxf"
Nov 02 15:33:46.509463 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:33:46.509443    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e84c4e2-b5e2-4ba8-a6e8-8090a06ac68c-tmp-dir\") pod \"auth-test-model-fda646-kserve-568f9ddd9-t6lxf\" (UID: \"7e84c4e2-b5e2-4ba8-a6e8-8090a06ac68c\") " pod="ai-tenant-e2e-shared-a-w1-fda646/auth-test-model-fda646-kserve-568f9ddd9-t6lxf"
Nov 02 15:33:46.509557 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:33:46.509546    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7e84c4e2-b5e2-4ba8-a6e8-8090a06ac68c-dshm\") pod \"auth-test-model-fda646-kserve-568f9ddd9-t6lxf\" (UID: \"7e84c4e2-b5e2-4ba8-a6e8-8090a06ac68c\") " pod="ai-tenant-e2e-shared-a-w1-fda646/auth-test-model-fda646-kserve-568f9ddd9-t6lxf"
Nov 02 15:33:46.509595 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:33:46.509568    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e84c4e2-b5e2-4ba8-a6e8-8090a06ac68c-tls-certs\") pod \"auth-test-model-fda646-kserve-568f9ddd9-t6lxf\" (UID: \"7e84c4e2-b5e2-4ba8-a6e8-8090a06ac68c\") " pod="ai-tenant-e2e-shared-a-w1-fda646/auth-test-model-fda646-kserve-568f9ddd9-t6lxf"
Nov 02 15:33:46.610500 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:33:46.610465    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7e84c4e2-b5e2-4ba8-a6e8-8090a06ac68c-dshm\") pod \"auth-test-model-fda646-kserve-568f9ddd9-t6lxf\" (UID: \"7e84c4e2-b5e2-4ba8-a6e8-8090a06ac68c\") " pod="ai-tenant-e2e-shared-a-w1-fda646/auth-test-model-fda646-kserve-568f9ddd9-t6lxf"
Nov 02 15:33:46.610500 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:33:46.610504    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e84c4e2-b5e2-4ba8-a6e8-8090a06ac68c-tls-certs\") pod \"auth-test-model-fda646-kserve-568f9ddd9-t6lxf\" (UID: \"7e84c4e2-b5e2-4ba8-a6e8-8090a06ac68c\") " pod="ai-tenant-e2e-shared-a-w1-fda646/auth-test-model-fda646-kserve-568f9ddd9-t6lxf"
Nov 02 15:33:46.611044 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:33:46.610536    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7e84c4e2-b5e2-4ba8-a6e8-8090a06ac68c-home\") pod \"auth-test-model-fda646-kserve-568f9ddd9-t6lxf\" (UID: \"7e84c4e2-b5e2-4ba8-a6e8-8090a06ac68c\") " pod="ai-tenant-e2e-shared-a-w1-fda646/auth-test-model-fda646-kserve-568f9ddd9-t6lxf"
Nov 02 15:33:46.611044 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:33:46.610634    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7e84c4e2-b5e2-4ba8-a6e8-8090a06ac68c-model-cache\") pod \"auth-test-model-fda646-kserve-568f9ddd9-t6lxf\" (UID: \"7e84c4e2-b5e2-4ba8-a6e8-8090a06ac68c\") " pod="ai-tenant-e2e-shared-a-w1-fda646/auth-test-model-fda646-kserve-568f9ddd9-t6lxf"
Nov 02 15:33:46.611044 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:33:46.610686    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xh978\" (UniqueName: \"kubernetes.io/projected/7e84c4e2-b5e2-4ba8-a6e8-8090a06ac68c-kube-api-access-xh978\") pod \"auth-test-model-fda646-kserve-568f9ddd9-t6lxf\" (UID: \"7e84c4e2-b5e2-4ba8-a6e8-8090a06ac68c\") " pod="ai-tenant-e2e-shared-a-w1-fda646/auth-test-model-fda646-kserve-568f9ddd9-t6lxf"
Nov 02 15:33:46.611044 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:33:46.610715    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e84c4e2-b5e2-4ba8-a6e8-8090a06ac68c-tmp-dir\") pod \"auth-test-model-fda646-kserve-568f9ddd9-t6lxf\" (UID: \"7e84c4e2-b5e2-4ba8-a6e8-8090a06ac68c\") " pod="ai-tenant-e2e-shared-a-w1-fda646/auth-test-model-fda646-kserve-568f9ddd9-t6lxf"
Nov 02 15:33:46.611044 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:33:46.611024    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7e84c4e2-b5e2-4ba8-a6e8-8090a06ac68c-model-cache\") pod \"auth-test-model-fda646-kserve-568f9ddd9-t6lxf\" (UID: \"7e84c4e2-b5e2-4ba8-a6e8-8090a06ac68c\") " pod="ai-tenant-e2e-shared-a-w1-fda646/auth-test-model-fda646-kserve-568f9ddd9-t6lxf"
Nov 02 15:33:46.611283 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:33:46.611045    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7e84c4e2-b5e2-4ba8-a6e8-8090a06ac68c-home\") pod \"auth-test-model-fda646-kserve-568f9ddd9-t6lxf\" (UID: \"7e84c4e2-b5e2-4ba8-a6e8-8090a06ac68c\") " pod="ai-tenant-e2e-shared-a-w1-fda646/auth-test-model-fda646-kserve-568f9ddd9-t6lxf"
Nov 02 15:33:46.611283 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:33:46.611093    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e84c4e2-b5e2-4ba8-a6e8-8090a06ac68c-tmp-dir\") pod \"auth-test-model-fda646-kserve-568f9ddd9-t6lxf\" (UID: \"7e84c4e2-b5e2-4ba8-a6e8-8090a06ac68c\") " pod="ai-tenant-e2e-shared-a-w1-fda646/auth-test-model-fda646-kserve-568f9ddd9-t6lxf"
Nov 02 15:33:46.612926 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:33:46.612903    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7e84c4e2-b5e2-4ba8-a6e8-8090a06ac68c-dshm\") pod \"auth-test-model-fda646-kserve-568f9ddd9-t6lxf\" (UID: \"7e84c4e2-b5e2-4ba8-a6e8-8090a06ac68c\") " pod="ai-tenant-e2e-shared-a-w1-fda646/auth-test-model-fda646-kserve-568f9ddd9-t6lxf"
Nov 02 15:33:46.613085 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:33:46.613065    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e84c4e2-b5e2-4ba8-a6e8-8090a06ac68c-tls-certs\") pod \"auth-test-model-fda646-kserve-568f9ddd9-t6lxf\" (UID: \"7e84c4e2-b5e2-4ba8-a6e8-8090a06ac68c\") " pod="ai-tenant-e2e-shared-a-w1-fda646/auth-test-model-fda646-kserve-568f9ddd9-t6lxf"
Nov 02 15:33:46.644856 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:33:46.644821    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xh978\" (UniqueName: \"kubernetes.io/projected/7e84c4e2-b5e2-4ba8-a6e8-8090a06ac68c-kube-api-access-xh978\") pod \"auth-test-model-fda646-kserve-568f9ddd9-t6lxf\" (UID: \"7e84c4e2-b5e2-4ba8-a6e8-8090a06ac68c\") " pod="ai-tenant-e2e-shared-a-w1-fda646/auth-test-model-fda646-kserve-568f9ddd9-t6lxf"
Nov 02 15:33:46.724199 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:33:46.724170    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-fda646/auth-test-model-fda646-kserve-568f9ddd9-t6lxf"
Nov 02 15:33:47.095946 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:33:47.095918    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w1-fda646/auth-test-model-fda646-kserve-568f9ddd9-t6lxf"]
Nov 02 15:33:47.098649 ip-10-0-135-138 kubenswrapper[5304]: W1102 15:33:47.098619    5304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7e84c4e2_b5e2_4ba8_a6e8_8090a06ac68c.slice/crio-0220bd62c1c608cb1f9736b7f9ab56d1fd1eded19095e11fae4b99943e64a9bb WatchSource:0}: Error finding container 0220bd62c1c608cb1f9736b7f9ab56d1fd1eded19095e11fae4b99943e64a9bb: Status 404 returned error can't find the container with id 0220bd62c1c608cb1f9736b7f9ab56d1fd1eded19095e11fae4b99943e64a9bb
Nov 02 15:33:47.576413 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:33:47.576316    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-fda646/auth-test-model-fda646-kserve-568f9ddd9-t6lxf" event={"ID":"7e84c4e2-b5e2-4ba8-a6e8-8090a06ac68c","Type":"ContainerStarted","Data":"3aaad7829e41605fe646a6dd017ea705ffd0d8207c90df5eea30c87ee418c27f"}
Nov 02 15:33:47.576413 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:33:47.576367    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-fda646/auth-test-model-fda646-kserve-568f9ddd9-t6lxf" event={"ID":"7e84c4e2-b5e2-4ba8-a6e8-8090a06ac68c","Type":"ContainerStarted","Data":"0220bd62c1c608cb1f9736b7f9ab56d1fd1eded19095e11fae4b99943e64a9bb"}
Nov 02 15:33:47.618584 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:33:47.618529    5304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w1-fda646/auth-test-model-fda646-kserve-568f9ddd9-t6lxf" podStartSLOduration=1.6185151119999999 podStartE2EDuration="1.618515112s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:33:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:33:47.617250668 +0000 UTC m=+2227.611075106" watchObservedRunningTime="2026-11-02 15:33:47.618515112 +0000 UTC m=+2227.612339520"
Nov 02 15:33:47.724377 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:33:47.724345    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w1-fda646/auth-test-model-fda646-kserve-568f9ddd9-t6lxf"
Nov 02 15:33:54.236348 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:33:54.236315    5304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w6-b955c5/auth-test-model-b955c5-kserve-7965dcb68f-75hkb"
Nov 02 15:33:54.250031 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:33:54.249997    5304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w6-b955c5/auth-test-model-b955c5-kserve-7965dcb68f-75hkb"
Nov 02 15:33:54.670907 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:33:54.670877    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w6-b955c5/auth-test-model-b955c5-kserve-7965dcb68f-75hkb"
Nov 02 15:33:54.799837 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:33:54.799803    5304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k6fjk" containerName="extractor"
Nov 02 15:33:56.724895 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:33:56.724858    5304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w1-fda646/auth-test-model-fda646-kserve-568f9ddd9-t6lxf"
Nov 02 15:33:56.738633 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:33:56.738604    5304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w1-fda646/auth-test-model-fda646-kserve-568f9ddd9-t6lxf"
Nov 02 15:33:57.692883 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:33:57.692852    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w1-fda646/auth-test-model-fda646-kserve-568f9ddd9-t6lxf"
Nov 02 15:34:24.799726 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:34:24.799643    5304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k6fjk" containerName="extractor"
Nov 02 15:34:31.883738 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:34:31.883702    5304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w6-b55da1/auth-test-model-b55da1-kserve-9f6cd75b5-8lljz"]
Nov 02 15:34:31.890085 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:34:31.890061    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w6-b55da1/auth-test-model-b55da1-kserve-9f6cd75b5-8lljz"
Nov 02 15:34:31.908823 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:34:31.908796    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w6-b55da1/auth-test-model-b55da1-kserve-9f6cd75b5-8lljz"]
Nov 02 15:34:31.928507 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:34:31.928467    5304 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w6-b55da1\"/\"kube-root-ca.crt\""
Nov 02 15:34:31.928664 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:34:31.928616    5304 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w6-b55da1\"/\"openshift-service-ca.crt\""
Nov 02 15:34:31.929414 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:34:31.929393    5304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w6-b55da1\"/\"default-dockercfg-qnfws\""
Nov 02 15:34:31.932098 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:34:31.932068    5304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w6-b55da1\"/\"auth-test-model-b55da1-kserve-self-signed-certs\""
Nov 02 15:34:31.964194 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:34:31.963359    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/860edf42-9efd-4d01-aaf3-78bbd87cc2f9-dshm\") pod \"auth-test-model-b55da1-kserve-9f6cd75b5-8lljz\" (UID: \"860edf42-9efd-4d01-aaf3-78bbd87cc2f9\") " pod="ai-tenant-e2e-shared-b-w6-b55da1/auth-test-model-b55da1-kserve-9f6cd75b5-8lljz"
Nov 02 15:34:31.964194 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:34:31.963430    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/860edf42-9efd-4d01-aaf3-78bbd87cc2f9-home\") pod \"auth-test-model-b55da1-kserve-9f6cd75b5-8lljz\" (UID: \"860edf42-9efd-4d01-aaf3-78bbd87cc2f9\") " pod="ai-tenant-e2e-shared-b-w6-b55da1/auth-test-model-b55da1-kserve-9f6cd75b5-8lljz"
Nov 02 15:34:31.964194 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:34:31.963552    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-84zg5\" (UniqueName: \"kubernetes.io/projected/860edf42-9efd-4d01-aaf3-78bbd87cc2f9-kube-api-access-84zg5\") pod \"auth-test-model-b55da1-kserve-9f6cd75b5-8lljz\" (UID: \"860edf42-9efd-4d01-aaf3-78bbd87cc2f9\") " pod="ai-tenant-e2e-shared-b-w6-b55da1/auth-test-model-b55da1-kserve-9f6cd75b5-8lljz"
Nov 02 15:34:31.964194 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:34:31.963649    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/860edf42-9efd-4d01-aaf3-78bbd87cc2f9-model-cache\") pod \"auth-test-model-b55da1-kserve-9f6cd75b5-8lljz\" (UID: \"860edf42-9efd-4d01-aaf3-78bbd87cc2f9\") " pod="ai-tenant-e2e-shared-b-w6-b55da1/auth-test-model-b55da1-kserve-9f6cd75b5-8lljz"
Nov 02 15:34:31.964194 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:34:31.963810    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/860edf42-9efd-4d01-aaf3-78bbd87cc2f9-tmp-dir\") pod \"auth-test-model-b55da1-kserve-9f6cd75b5-8lljz\" (UID: \"860edf42-9efd-4d01-aaf3-78bbd87cc2f9\") " pod="ai-tenant-e2e-shared-b-w6-b55da1/auth-test-model-b55da1-kserve-9f6cd75b5-8lljz"
Nov 02 15:34:31.964194 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:34:31.963861    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/860edf42-9efd-4d01-aaf3-78bbd87cc2f9-tls-certs\") pod \"auth-test-model-b55da1-kserve-9f6cd75b5-8lljz\" (UID: \"860edf42-9efd-4d01-aaf3-78bbd87cc2f9\") " pod="ai-tenant-e2e-shared-b-w6-b55da1/auth-test-model-b55da1-kserve-9f6cd75b5-8lljz"
Nov 02 15:34:32.065759 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:34:32.065722    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/860edf42-9efd-4d01-aaf3-78bbd87cc2f9-tls-certs\") pod \"auth-test-model-b55da1-kserve-9f6cd75b5-8lljz\" (UID: \"860edf42-9efd-4d01-aaf3-78bbd87cc2f9\") " pod="ai-tenant-e2e-shared-b-w6-b55da1/auth-test-model-b55da1-kserve-9f6cd75b5-8lljz"
Nov 02 15:34:32.065938 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:34:32.065904    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/860edf42-9efd-4d01-aaf3-78bbd87cc2f9-dshm\") pod \"auth-test-model-b55da1-kserve-9f6cd75b5-8lljz\" (UID: \"860edf42-9efd-4d01-aaf3-78bbd87cc2f9\") " pod="ai-tenant-e2e-shared-b-w6-b55da1/auth-test-model-b55da1-kserve-9f6cd75b5-8lljz"
Nov 02 15:34:32.065938 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:34:32.065932    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/860edf42-9efd-4d01-aaf3-78bbd87cc2f9-home\") pod \"auth-test-model-b55da1-kserve-9f6cd75b5-8lljz\" (UID: \"860edf42-9efd-4d01-aaf3-78bbd87cc2f9\") " pod="ai-tenant-e2e-shared-b-w6-b55da1/auth-test-model-b55da1-kserve-9f6cd75b5-8lljz"
Nov 02 15:34:32.066063 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:34:32.065972    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-84zg5\" (UniqueName: \"kubernetes.io/projected/860edf42-9efd-4d01-aaf3-78bbd87cc2f9-kube-api-access-84zg5\") pod \"auth-test-model-b55da1-kserve-9f6cd75b5-8lljz\" (UID: \"860edf42-9efd-4d01-aaf3-78bbd87cc2f9\") " pod="ai-tenant-e2e-shared-b-w6-b55da1/auth-test-model-b55da1-kserve-9f6cd75b5-8lljz"
Nov 02 15:34:32.066063 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:34:32.066000    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/860edf42-9efd-4d01-aaf3-78bbd87cc2f9-model-cache\") pod \"auth-test-model-b55da1-kserve-9f6cd75b5-8lljz\" (UID: \"860edf42-9efd-4d01-aaf3-78bbd87cc2f9\") " pod="ai-tenant-e2e-shared-b-w6-b55da1/auth-test-model-b55da1-kserve-9f6cd75b5-8lljz"
Nov 02 15:34:32.066192 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:34:32.066104    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/860edf42-9efd-4d01-aaf3-78bbd87cc2f9-tmp-dir\") pod \"auth-test-model-b55da1-kserve-9f6cd75b5-8lljz\" (UID: \"860edf42-9efd-4d01-aaf3-78bbd87cc2f9\") " pod="ai-tenant-e2e-shared-b-w6-b55da1/auth-test-model-b55da1-kserve-9f6cd75b5-8lljz"
Nov 02 15:34:32.066377 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:34:32.066345    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/860edf42-9efd-4d01-aaf3-78bbd87cc2f9-home\") pod \"auth-test-model-b55da1-kserve-9f6cd75b5-8lljz\" (UID: \"860edf42-9efd-4d01-aaf3-78bbd87cc2f9\") " pod="ai-tenant-e2e-shared-b-w6-b55da1/auth-test-model-b55da1-kserve-9f6cd75b5-8lljz"
Nov 02 15:34:32.066499 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:34:32.066477    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/860edf42-9efd-4d01-aaf3-78bbd87cc2f9-model-cache\") pod \"auth-test-model-b55da1-kserve-9f6cd75b5-8lljz\" (UID: \"860edf42-9efd-4d01-aaf3-78bbd87cc2f9\") " pod="ai-tenant-e2e-shared-b-w6-b55da1/auth-test-model-b55da1-kserve-9f6cd75b5-8lljz"
Nov 02 15:34:32.066591 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:34:32.066493    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/860edf42-9efd-4d01-aaf3-78bbd87cc2f9-tmp-dir\") pod \"auth-test-model-b55da1-kserve-9f6cd75b5-8lljz\" (UID: \"860edf42-9efd-4d01-aaf3-78bbd87cc2f9\") " pod="ai-tenant-e2e-shared-b-w6-b55da1/auth-test-model-b55da1-kserve-9f6cd75b5-8lljz"
Nov 02 15:34:32.068181 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:34:32.068151    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/860edf42-9efd-4d01-aaf3-78bbd87cc2f9-dshm\") pod \"auth-test-model-b55da1-kserve-9f6cd75b5-8lljz\" (UID: \"860edf42-9efd-4d01-aaf3-78bbd87cc2f9\") " pod="ai-tenant-e2e-shared-b-w6-b55da1/auth-test-model-b55da1-kserve-9f6cd75b5-8lljz"
Nov 02 15:34:32.068427 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:34:32.068405    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/860edf42-9efd-4d01-aaf3-78bbd87cc2f9-tls-certs\") pod \"auth-test-model-b55da1-kserve-9f6cd75b5-8lljz\" (UID: \"860edf42-9efd-4d01-aaf3-78bbd87cc2f9\") " pod="ai-tenant-e2e-shared-b-w6-b55da1/auth-test-model-b55da1-kserve-9f6cd75b5-8lljz"
Nov 02 15:34:32.089223 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:34:32.089199    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-84zg5\" (UniqueName: \"kubernetes.io/projected/860edf42-9efd-4d01-aaf3-78bbd87cc2f9-kube-api-access-84zg5\") pod \"auth-test-model-b55da1-kserve-9f6cd75b5-8lljz\" (UID: \"860edf42-9efd-4d01-aaf3-78bbd87cc2f9\") " pod="ai-tenant-e2e-shared-b-w6-b55da1/auth-test-model-b55da1-kserve-9f6cd75b5-8lljz"
Nov 02 15:34:32.203200 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:34:32.203063    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w6-b55da1/auth-test-model-b55da1-kserve-9f6cd75b5-8lljz"
Nov 02 15:34:32.534665 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:34:32.534580    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w6-b55da1/auth-test-model-b55da1-kserve-9f6cd75b5-8lljz"]
Nov 02 15:34:32.538824 ip-10-0-135-138 kubenswrapper[5304]: W1102 15:34:32.538787    5304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod860edf42_9efd_4d01_aaf3_78bbd87cc2f9.slice/crio-e77c8a42554b98a42ca05f8411c0d2cc850f1a36c5baa7d341b6b3449975868c WatchSource:0}: Error finding container e77c8a42554b98a42ca05f8411c0d2cc850f1a36c5baa7d341b6b3449975868c: Status 404 returned error can't find the container with id e77c8a42554b98a42ca05f8411c0d2cc850f1a36c5baa7d341b6b3449975868c
Nov 02 15:34:32.974200 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:34:32.974151    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w6-b55da1/auth-test-model-b55da1-kserve-9f6cd75b5-8lljz" event={"ID":"860edf42-9efd-4d01-aaf3-78bbd87cc2f9","Type":"ContainerStarted","Data":"3dad92941bbe1f68a7b5b2e5994fae11ae109b4e86f6d10472a30fe77e5e1ff2"}
Nov 02 15:34:32.974662 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:34:32.974215    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w6-b55da1/auth-test-model-b55da1-kserve-9f6cd75b5-8lljz" event={"ID":"860edf42-9efd-4d01-aaf3-78bbd87cc2f9","Type":"ContainerStarted","Data":"e77c8a42554b98a42ca05f8411c0d2cc850f1a36c5baa7d341b6b3449975868c"}
Nov 02 15:34:33.045442 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:34:33.045384    5304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w6-b55da1/auth-test-model-b55da1-kserve-9f6cd75b5-8lljz" podStartSLOduration=2.045368715 podStartE2EDuration="2.045368715s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:34:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:34:33.042268062 +0000 UTC m=+2273.036092483" watchObservedRunningTime="2026-11-02 15:34:33.045368715 +0000 UTC m=+2273.039193127"
Nov 02 15:34:33.203779 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:34:33.203748    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w6-b55da1/auth-test-model-b55da1-kserve-9f6cd75b5-8lljz"
Nov 02 15:34:42.203883 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:34:42.203800    5304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w6-b55da1/auth-test-model-b55da1-kserve-9f6cd75b5-8lljz"
Nov 02 15:34:42.217180 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:34:42.217155    5304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w6-b55da1/auth-test-model-b55da1-kserve-9f6cd75b5-8lljz"
Nov 02 15:34:43.068408 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:34:43.068377    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w6-b55da1/auth-test-model-b55da1-kserve-9f6cd75b5-8lljz"
Nov 02 15:34:54.803322 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:34:54.803284    5304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k6fjk" containerName="extractor"
Nov 02 15:34:59.189234 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:34:59.189199    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-nrs9j"]
Nov 02 15:34:59.189791 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:34:59.189401    5304 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-nrs9j" podUID="c1051c4a-02ca-40b8-b8de-180bd2ca598b" containerName="istio-proxy" containerID="cri-o://47952bf72c2463b71b7b3c41deda3dfc9230b87048e3357cd2c0eae2fd88a945" gracePeriod=30
Nov 02 15:35:04.468266 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:35:04.468201    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-nrs9j"
Nov 02 15:35:04.566333 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:35:04.566292    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1051c4a-02ca-40b8-b8de-180bd2ca598b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1051c4a-02ca-40b8-b8de-180bd2ca598b-istio-envoy\") pod \"c1051c4a-02ca-40b8-b8de-180bd2ca598b\" (UID: \"c1051c4a-02ca-40b8-b8de-180bd2ca598b\") "
Nov 02 15:35:04.566546 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:35:04.566361    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1051c4a-02ca-40b8-b8de-180bd2ca598b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1051c4a-02ca-40b8-b8de-180bd2ca598b-workload-certs\") pod \"c1051c4a-02ca-40b8-b8de-180bd2ca598b\" (UID: \"c1051c4a-02ca-40b8-b8de-180bd2ca598b\") "
Nov 02 15:35:04.566546 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:35:04.566412    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1051c4a-02ca-40b8-b8de-180bd2ca598b-istio-token\" (UniqueName: \"kubernetes.io/projected/c1051c4a-02ca-40b8-b8de-180bd2ca598b-istio-token\") pod \"c1051c4a-02ca-40b8-b8de-180bd2ca598b\" (UID: \"c1051c4a-02ca-40b8-b8de-180bd2ca598b\") "
Nov 02 15:35:04.566546 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:35:04.566438    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1051c4a-02ca-40b8-b8de-180bd2ca598b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1051c4a-02ca-40b8-b8de-180bd2ca598b-istio-data\") pod \"c1051c4a-02ca-40b8-b8de-180bd2ca598b\" (UID: \"c1051c4a-02ca-40b8-b8de-180bd2ca598b\") "
Nov 02 15:35:04.566546 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:35:04.566521    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1051c4a-02ca-40b8-b8de-180bd2ca598b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1051c4a-02ca-40b8-b8de-180bd2ca598b-workload-socket\") pod \"c1051c4a-02ca-40b8-b8de-180bd2ca598b\" (UID: \"c1051c4a-02ca-40b8-b8de-180bd2ca598b\") "
Nov 02 15:35:04.566795 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:35:04.566550    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1051c4a-02ca-40b8-b8de-180bd2ca598b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1051c4a-02ca-40b8-b8de-180bd2ca598b-credential-socket\") pod \"c1051c4a-02ca-40b8-b8de-180bd2ca598b\" (UID: \"c1051c4a-02ca-40b8-b8de-180bd2ca598b\") "
Nov 02 15:35:04.566795 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:35:04.566666    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c1051c4a-02ca-40b8-b8de-180bd2ca598b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1051c4a-02ca-40b8-b8de-180bd2ca598b-istio-podinfo\") pod \"c1051c4a-02ca-40b8-b8de-180bd2ca598b\" (UID: \"c1051c4a-02ca-40b8-b8de-180bd2ca598b\") "
Nov 02 15:35:04.566795 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:35:04.566776    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1051c4a-02ca-40b8-b8de-180bd2ca598b-kube-api-access-h46cm\" (UniqueName: \"kubernetes.io/projected/c1051c4a-02ca-40b8-b8de-180bd2ca598b-kube-api-access-h46cm\") pod \"c1051c4a-02ca-40b8-b8de-180bd2ca598b\" (UID: \"c1051c4a-02ca-40b8-b8de-180bd2ca598b\") "
Nov 02 15:35:04.566950 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:35:04.566855    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c1051c4a-02ca-40b8-b8de-180bd2ca598b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1051c4a-02ca-40b8-b8de-180bd2ca598b-istiod-ca-cert\") pod \"c1051c4a-02ca-40b8-b8de-180bd2ca598b\" (UID: \"c1051c4a-02ca-40b8-b8de-180bd2ca598b\") "
Nov 02 15:35:04.567036 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:35:04.567003    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1051c4a-02ca-40b8-b8de-180bd2ca598b-workload-socket" pod "c1051c4a-02ca-40b8-b8de-180bd2ca598b" (UID: "c1051c4a-02ca-40b8-b8de-180bd2ca598b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:35:04.567239 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:35:04.567193    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1051c4a-02ca-40b8-b8de-180bd2ca598b-workload-certs" pod "c1051c4a-02ca-40b8-b8de-180bd2ca598b" (UID: "c1051c4a-02ca-40b8-b8de-180bd2ca598b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:35:04.567634 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:35:04.567499    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1051c4a-02ca-40b8-b8de-180bd2ca598b-istiod-ca-cert" pod "c1051c4a-02ca-40b8-b8de-180bd2ca598b" (UID: "c1051c4a-02ca-40b8-b8de-180bd2ca598b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:35:04.567634 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:35:04.567574    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1051c4a-02ca-40b8-b8de-180bd2ca598b-istio-data" pod "c1051c4a-02ca-40b8-b8de-180bd2ca598b" (UID: "c1051c4a-02ca-40b8-b8de-180bd2ca598b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:35:04.567841 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:35:04.567820    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1051c4a-02ca-40b8-b8de-180bd2ca598b-credential-socket" pod "c1051c4a-02ca-40b8-b8de-180bd2ca598b" (UID: "c1051c4a-02ca-40b8-b8de-180bd2ca598b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:35:04.568046 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:35:04.568019    5304 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1051c4a-02ca-40b8-b8de-180bd2ca598b-istiod-ca-cert\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:35:04.568173 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:35:04.568161    5304 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1051c4a-02ca-40b8-b8de-180bd2ca598b-workload-certs\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:35:04.568314 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:35:04.568303    5304 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1051c4a-02ca-40b8-b8de-180bd2ca598b-istio-data\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:35:04.568399 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:35:04.568388    5304 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1051c4a-02ca-40b8-b8de-180bd2ca598b-workload-socket\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:35:04.569254 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:35:04.569217    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1051c4a-02ca-40b8-b8de-180bd2ca598b-istio-envoy" pod "c1051c4a-02ca-40b8-b8de-180bd2ca598b" (UID: "c1051c4a-02ca-40b8-b8de-180bd2ca598b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:35:04.570229 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:35:04.570193    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1051c4a-02ca-40b8-b8de-180bd2ca598b-istio-token" pod "c1051c4a-02ca-40b8-b8de-180bd2ca598b" (UID: "c1051c4a-02ca-40b8-b8de-180bd2ca598b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:35:04.570512 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:35:04.570484    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c1051c4a-02ca-40b8-b8de-180bd2ca598b-istio-podinfo" pod "c1051c4a-02ca-40b8-b8de-180bd2ca598b" (UID: "c1051c4a-02ca-40b8-b8de-180bd2ca598b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 15:35:04.570788 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:35:04.570763    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1051c4a-02ca-40b8-b8de-180bd2ca598b-kube-api-access-h46cm" pod "c1051c4a-02ca-40b8-b8de-180bd2ca598b" (UID: "c1051c4a-02ca-40b8-b8de-180bd2ca598b"). InnerVolumeSpecName "kube-api-access-h46cm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:35:04.670400 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:35:04.670363    5304 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1051c4a-02ca-40b8-b8de-180bd2ca598b-istio-podinfo\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:35:04.670400 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:35:04.670397    5304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h46cm\" (UniqueName: \"kubernetes.io/projected/c1051c4a-02ca-40b8-b8de-180bd2ca598b-kube-api-access-h46cm\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:35:04.670400 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:35:04.670409    5304 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1051c4a-02ca-40b8-b8de-180bd2ca598b-istio-envoy\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:35:04.670661 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:35:04.670418    5304 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1051c4a-02ca-40b8-b8de-180bd2ca598b-istio-token\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:35:04.670661 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:35:04.670427    5304 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1051c4a-02ca-40b8-b8de-180bd2ca598b-credential-socket\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:35:05.204603 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:35:05.204566    5304 generic.go:356] "Generic (PLEG): container finished" podID="c1051c4a-02ca-40b8-b8de-180bd2ca598b" containerID="47952bf72c2463b71b7b3c41deda3dfc9230b87048e3357cd2c0eae2fd88a945" exitCode=0
Nov 02 15:35:05.205315 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:35:05.204629    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-nrs9j" event={"ID":"c1051c4a-02ca-40b8-b8de-180bd2ca598b","Type":"ContainerDied","Data":"47952bf72c2463b71b7b3c41deda3dfc9230b87048e3357cd2c0eae2fd88a945"}
Nov 02 15:35:05.205315 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:35:05.204695    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-nrs9j"
Nov 02 15:35:05.205315 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:35:05.205097    5304 scope.go:122] "RemoveContainer" containerID="47952bf72c2463b71b7b3c41deda3dfc9230b87048e3357cd2c0eae2fd88a945"
Nov 02 15:35:05.205315 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:35:05.205077    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-nrs9j" event={"ID":"c1051c4a-02ca-40b8-b8de-180bd2ca598b","Type":"ContainerDied","Data":"d36f3b581eeaaa3d3dd863e265ff6ed4a3920d387341834892edded20fdd5f99"}
Nov 02 15:35:05.216462 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:35:05.216440    5304 scope.go:122] "RemoveContainer" containerID="9fafab017e9dff6e5bce71d3792068134589f9b46bfe53dbbefbeaefabbdb83c"
Nov 02 15:35:05.226441 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:35:05.226420    5304 scope.go:122] "RemoveContainer" containerID="47952bf72c2463b71b7b3c41deda3dfc9230b87048e3357cd2c0eae2fd88a945"
Nov 02 15:35:05.226774 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:35:05.226753    5304 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"47952bf72c2463b71b7b3c41deda3dfc9230b87048e3357cd2c0eae2fd88a945\": container with ID starting with 47952bf72c2463b71b7b3c41deda3dfc9230b87048e3357cd2c0eae2fd88a945 not found: ID does not exist" containerID="47952bf72c2463b71b7b3c41deda3dfc9230b87048e3357cd2c0eae2fd88a945"
Nov 02 15:35:05.226850 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:35:05.226782    5304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47952bf72c2463b71b7b3c41deda3dfc9230b87048e3357cd2c0eae2fd88a945"} err="failed to get container status \"47952bf72c2463b71b7b3c41deda3dfc9230b87048e3357cd2c0eae2fd88a945\": rpc error: code = NotFound desc = could not find container \"47952bf72c2463b71b7b3c41deda3dfc9230b87048e3357cd2c0eae2fd88a945\": container with ID starting with 47952bf72c2463b71b7b3c41deda3dfc9230b87048e3357cd2c0eae2fd88a945 not found: ID does not exist"
Nov 02 15:35:05.226850 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:35:05.226802    5304 scope.go:122] "RemoveContainer" containerID="9fafab017e9dff6e5bce71d3792068134589f9b46bfe53dbbefbeaefabbdb83c"
Nov 02 15:35:05.227046 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:35:05.227029    5304 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9fafab017e9dff6e5bce71d3792068134589f9b46bfe53dbbefbeaefabbdb83c\": container with ID starting with 9fafab017e9dff6e5bce71d3792068134589f9b46bfe53dbbefbeaefabbdb83c not found: ID does not exist" containerID="9fafab017e9dff6e5bce71d3792068134589f9b46bfe53dbbefbeaefabbdb83c"
Nov 02 15:35:05.227094 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:35:05.227052    5304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9fafab017e9dff6e5bce71d3792068134589f9b46bfe53dbbefbeaefabbdb83c"} err="failed to get container status \"9fafab017e9dff6e5bce71d3792068134589f9b46bfe53dbbefbeaefabbdb83c\": rpc error: code = NotFound desc = could not find container \"9fafab017e9dff6e5bce71d3792068134589f9b46bfe53dbbefbeaefabbdb83c\": container with ID starting with 9fafab017e9dff6e5bce71d3792068134589f9b46bfe53dbbefbeaefabbdb83c not found: ID does not exist"
Nov 02 15:35:05.273851 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:35:05.273820    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-nrs9j"]
Nov 02 15:35:05.285649 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:35:05.285619    5304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-nrs9j"]
Nov 02 15:35:06.119716 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:35:06.119682    5304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c1051c4a-02ca-40b8-b8de-180bd2ca598b" path="/var/lib/kubelet/pods/c1051c4a-02ca-40b8-b8de-180bd2ca598b/volumes"
Nov 02 15:35:24.802420 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:35:24.802385    5304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k6fjk" containerName="extractor"
Nov 02 15:35:54.809244 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:35:54.809116    5304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k6fjk" containerName="extractor"
Nov 02 15:35:58.512635 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:35:58.512582    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-25tqv"]
Nov 02 15:35:58.513144 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:35:58.512806    5304 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-25tqv" podUID="3ed68512-6850-45d4-af0d-a941c2bf6ec9" containerName="istio-proxy" containerID="cri-o://1442a2c5a420e08de2915db274b4fc9b201da678242c7a4d4e215143733d6a93" gracePeriod=30
Nov 02 15:35:59.119646 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:35:59.119600    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-g8tjj"]
Nov 02 15:35:59.119972 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:35:59.119875    5304 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-g8tjj" podUID="0b48fb40-a77c-4345-a9d4-776bcdabb078" containerName="istio-proxy" containerID="cri-o://8dae976028ac237c8abcbbaf74aab1520943f99f8ed0e7c8875888f531c144c2" gracePeriod=30
Nov 02 15:35:59.148756 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:35:59.148724    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-28f621-dd989bdc-9x66m"]
Nov 02 15:35:59.165505 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:35:59.165477    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-pshqj"]
Nov 02 15:35:59.165702 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:35:59.165682    5304 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-pshqj" podUID="89710086-0e78-4395-b8f9-e47cce4f7dfe" containerName="istio-proxy" containerID="cri-o://7141638a35a6e8d0eeea3c7ac8e21defc3c8db61481fb5e2a002c417397cc317" gracePeriod=30
Nov 02 15:36:01.071298 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:36:01.071260    5304 patch_prober.go:29] interesting pod/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-25tqv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 15:36:01.071704 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:36:01.071329    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-25tqv" podUID="3ed68512-6850-45d4-af0d-a941c2bf6ec9" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 15:36:03.812428 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:36:03.812396    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-25tqv"
Nov 02 15:36:03.901725 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:36:03.901688    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ed68512-6850-45d4-af0d-a941c2bf6ec9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ed68512-6850-45d4-af0d-a941c2bf6ec9-istio-envoy\") pod \"3ed68512-6850-45d4-af0d-a941c2bf6ec9\" (UID: \"3ed68512-6850-45d4-af0d-a941c2bf6ec9\") "
Nov 02 15:36:03.901916 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:36:03.901786    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ed68512-6850-45d4-af0d-a941c2bf6ec9-kube-api-access-rwf6k\" (UniqueName: \"kubernetes.io/projected/3ed68512-6850-45d4-af0d-a941c2bf6ec9-kube-api-access-rwf6k\") pod \"3ed68512-6850-45d4-af0d-a941c2bf6ec9\" (UID: \"3ed68512-6850-45d4-af0d-a941c2bf6ec9\") "
Nov 02 15:36:03.901916 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:36:03.901835    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ed68512-6850-45d4-af0d-a941c2bf6ec9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ed68512-6850-45d4-af0d-a941c2bf6ec9-workload-certs\") pod \"3ed68512-6850-45d4-af0d-a941c2bf6ec9\" (UID: \"3ed68512-6850-45d4-af0d-a941c2bf6ec9\") "
Nov 02 15:36:03.901916 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:36:03.901879    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ed68512-6850-45d4-af0d-a941c2bf6ec9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ed68512-6850-45d4-af0d-a941c2bf6ec9-credential-socket\") pod \"3ed68512-6850-45d4-af0d-a941c2bf6ec9\" (UID: \"3ed68512-6850-45d4-af0d-a941c2bf6ec9\") "
Nov 02 15:36:03.902879 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:36:03.902190    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ed68512-6850-45d4-af0d-a941c2bf6ec9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ed68512-6850-45d4-af0d-a941c2bf6ec9-workload-socket\") pod \"3ed68512-6850-45d4-af0d-a941c2bf6ec9\" (UID: \"3ed68512-6850-45d4-af0d-a941c2bf6ec9\") "
Nov 02 15:36:03.902879 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:36:03.902195    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ed68512-6850-45d4-af0d-a941c2bf6ec9-credential-socket" pod "3ed68512-6850-45d4-af0d-a941c2bf6ec9" (UID: "3ed68512-6850-45d4-af0d-a941c2bf6ec9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:36:03.902879 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:36:03.902272    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ed68512-6850-45d4-af0d-a941c2bf6ec9-workload-certs" pod "3ed68512-6850-45d4-af0d-a941c2bf6ec9" (UID: "3ed68512-6850-45d4-af0d-a941c2bf6ec9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:36:03.902879 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:36:03.902303    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3ed68512-6850-45d4-af0d-a941c2bf6ec9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ed68512-6850-45d4-af0d-a941c2bf6ec9-istio-podinfo\") pod \"3ed68512-6850-45d4-af0d-a941c2bf6ec9\" (UID: \"3ed68512-6850-45d4-af0d-a941c2bf6ec9\") "
Nov 02 15:36:03.902879 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:36:03.902345    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3ed68512-6850-45d4-af0d-a941c2bf6ec9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ed68512-6850-45d4-af0d-a941c2bf6ec9-istiod-ca-cert\") pod \"3ed68512-6850-45d4-af0d-a941c2bf6ec9\" (UID: \"3ed68512-6850-45d4-af0d-a941c2bf6ec9\") "
Nov 02 15:36:03.902879 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:36:03.902421    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ed68512-6850-45d4-af0d-a941c2bf6ec9-workload-socket" pod "3ed68512-6850-45d4-af0d-a941c2bf6ec9" (UID: "3ed68512-6850-45d4-af0d-a941c2bf6ec9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:36:03.902879 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:36:03.902495    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ed68512-6850-45d4-af0d-a941c2bf6ec9-istio-token\" (UniqueName: \"kubernetes.io/projected/3ed68512-6850-45d4-af0d-a941c2bf6ec9-istio-token\") pod \"3ed68512-6850-45d4-af0d-a941c2bf6ec9\" (UID: \"3ed68512-6850-45d4-af0d-a941c2bf6ec9\") "
Nov 02 15:36:03.902879 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:36:03.902572    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ed68512-6850-45d4-af0d-a941c2bf6ec9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ed68512-6850-45d4-af0d-a941c2bf6ec9-istio-data\") pod \"3ed68512-6850-45d4-af0d-a941c2bf6ec9\" (UID: \"3ed68512-6850-45d4-af0d-a941c2bf6ec9\") "
Nov 02 15:36:03.902879 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:36:03.902839    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ed68512-6850-45d4-af0d-a941c2bf6ec9-istiod-ca-cert" pod "3ed68512-6850-45d4-af0d-a941c2bf6ec9" (UID: "3ed68512-6850-45d4-af0d-a941c2bf6ec9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:36:03.903489 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:36:03.902929    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ed68512-6850-45d4-af0d-a941c2bf6ec9-istio-data" pod "3ed68512-6850-45d4-af0d-a941c2bf6ec9" (UID: "3ed68512-6850-45d4-af0d-a941c2bf6ec9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:36:03.903489 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:36:03.903389    5304 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ed68512-6850-45d4-af0d-a941c2bf6ec9-workload-certs\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:36:03.903489 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:36:03.903412    5304 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ed68512-6850-45d4-af0d-a941c2bf6ec9-credential-socket\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:36:03.903489 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:36:03.903425    5304 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ed68512-6850-45d4-af0d-a941c2bf6ec9-workload-socket\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:36:03.903489 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:36:03.903438    5304 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ed68512-6850-45d4-af0d-a941c2bf6ec9-istiod-ca-cert\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:36:03.903489 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:36:03.903452    5304 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ed68512-6850-45d4-af0d-a941c2bf6ec9-istio-data\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:36:03.904596 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:36:03.904564    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ed68512-6850-45d4-af0d-a941c2bf6ec9-kube-api-access-rwf6k" pod "3ed68512-6850-45d4-af0d-a941c2bf6ec9" (UID: "3ed68512-6850-45d4-af0d-a941c2bf6ec9"). InnerVolumeSpecName "kube-api-access-rwf6k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:36:03.904723 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:36:03.904643    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ed68512-6850-45d4-af0d-a941c2bf6ec9-istio-envoy" pod "3ed68512-6850-45d4-af0d-a941c2bf6ec9" (UID: "3ed68512-6850-45d4-af0d-a941c2bf6ec9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:36:03.905574 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:36:03.905545    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ed68512-6850-45d4-af0d-a941c2bf6ec9-istio-token" pod "3ed68512-6850-45d4-af0d-a941c2bf6ec9" (UID: "3ed68512-6850-45d4-af0d-a941c2bf6ec9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:36:03.906368 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:36:03.906344    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3ed68512-6850-45d4-af0d-a941c2bf6ec9-istio-podinfo" pod "3ed68512-6850-45d4-af0d-a941c2bf6ec9" (UID: "3ed68512-6850-45d4-af0d-a941c2bf6ec9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 15:36:04.004853 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:36:04.004820    5304 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ed68512-6850-45d4-af0d-a941c2bf6ec9-istio-token\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:36:04.004853 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:36:04.004845    5304 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ed68512-6850-45d4-af0d-a941c2bf6ec9-istio-envoy\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:36:04.005053 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:36:04.004855    5304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rwf6k\" (UniqueName: \"kubernetes.io/projected/3ed68512-6850-45d4-af0d-a941c2bf6ec9-kube-api-access-rwf6k\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:36:04.005053 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:36:04.004873    5304 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ed68512-6850-45d4-af0d-a941c2bf6ec9-istio-podinfo\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:36:04.149585 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:36:04.149537    5304 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-28f621-dd989bdc-9x66m" podUID="dc422942-41f9-4cee-938f-e9df1b05d0c7" containerName="payload-processing" containerID="cri-o://6cf882f73d0856520ef43074ea73feb0d1d6276d15f6807cdc1c4dde65dbe01a" gracePeriod=25
Nov 02 15:36:04.419277 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:36:04.419227    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-g8tjj"
Nov 02 15:36:04.478781 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:36:04.478095    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-pshqj"
Nov 02 15:36:04.510346 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:36:04.510313    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b48fb40-a77c-4345-a9d4-776bcdabb078-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b48fb40-a77c-4345-a9d4-776bcdabb078-workload-certs\") pod \"0b48fb40-a77c-4345-a9d4-776bcdabb078\" (UID: \"0b48fb40-a77c-4345-a9d4-776bcdabb078\") "
Nov 02 15:36:04.510557 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:36:04.510397    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b48fb40-a77c-4345-a9d4-776bcdabb078-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b48fb40-a77c-4345-a9d4-776bcdabb078-istio-data\") pod \"0b48fb40-a77c-4345-a9d4-776bcdabb078\" (UID: \"0b48fb40-a77c-4345-a9d4-776bcdabb078\") "
Nov 02 15:36:04.510557 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:36:04.510431    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b48fb40-a77c-4345-a9d4-776bcdabb078-kube-api-access-xmffj\" (UniqueName: \"kubernetes.io/projected/0b48fb40-a77c-4345-a9d4-776bcdabb078-kube-api-access-xmffj\") pod \"0b48fb40-a77c-4345-a9d4-776bcdabb078\" (UID: \"0b48fb40-a77c-4345-a9d4-776bcdabb078\") "
Nov 02 15:36:04.510557 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:36:04.510464    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89710086-0e78-4395-b8f9-e47cce4f7dfe-istio-token\" (UniqueName: \"kubernetes.io/projected/89710086-0e78-4395-b8f9-e47cce4f7dfe-istio-token\") pod \"89710086-0e78-4395-b8f9-e47cce4f7dfe\" (UID: \"89710086-0e78-4395-b8f9-e47cce4f7dfe\") "
Nov 02 15:36:04.510557 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:36:04.510519    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89710086-0e78-4395-b8f9-e47cce4f7dfe-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89710086-0e78-4395-b8f9-e47cce4f7dfe-istio-data\") pod \"89710086-0e78-4395-b8f9-e47cce4f7dfe\" (UID: \"89710086-0e78-4395-b8f9-e47cce4f7dfe\") "
Nov 02 15:36:04.510557 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:36:04.510547    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89710086-0e78-4395-b8f9-e47cce4f7dfe-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89710086-0e78-4395-b8f9-e47cce4f7dfe-credential-socket\") pod \"89710086-0e78-4395-b8f9-e47cce4f7dfe\" (UID: \"89710086-0e78-4395-b8f9-e47cce4f7dfe\") "
Nov 02 15:36:04.510852 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:36:04.510582    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0b48fb40-a77c-4345-a9d4-776bcdabb078-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b48fb40-a77c-4345-a9d4-776bcdabb078-istiod-ca-cert\") pod \"0b48fb40-a77c-4345-a9d4-776bcdabb078\" (UID: \"0b48fb40-a77c-4345-a9d4-776bcdabb078\") "
Nov 02 15:36:04.510852 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:36:04.510626    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b48fb40-a77c-4345-a9d4-776bcdabb078-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b48fb40-a77c-4345-a9d4-776bcdabb078-credential-socket\") pod \"0b48fb40-a77c-4345-a9d4-776bcdabb078\" (UID: \"0b48fb40-a77c-4345-a9d4-776bcdabb078\") "
Nov 02 15:36:04.510852 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:36:04.510636    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b48fb40-a77c-4345-a9d4-776bcdabb078-workload-certs" pod "0b48fb40-a77c-4345-a9d4-776bcdabb078" (UID: "0b48fb40-a77c-4345-a9d4-776bcdabb078"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:36:04.510852 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:36:04.510662    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/89710086-0e78-4395-b8f9-e47cce4f7dfe-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89710086-0e78-4395-b8f9-e47cce4f7dfe-istio-podinfo\") pod \"89710086-0e78-4395-b8f9-e47cce4f7dfe\" (UID: \"89710086-0e78-4395-b8f9-e47cce4f7dfe\") "
Nov 02 15:36:04.510852 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:36:04.510728    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b48fb40-a77c-4345-a9d4-776bcdabb078-istio-data" pod "0b48fb40-a77c-4345-a9d4-776bcdabb078" (UID: "0b48fb40-a77c-4345-a9d4-776bcdabb078"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:36:04.510852 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:36:04.510808    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b48fb40-a77c-4345-a9d4-776bcdabb078-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b48fb40-a77c-4345-a9d4-776bcdabb078-workload-socket\") pod \"0b48fb40-a77c-4345-a9d4-776bcdabb078\" (UID: \"0b48fb40-a77c-4345-a9d4-776bcdabb078\") "
Nov 02 15:36:04.511198 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:36:04.510883    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89710086-0e78-4395-b8f9-e47cce4f7dfe-kube-api-access-mfhk7\" (UniqueName: \"kubernetes.io/projected/89710086-0e78-4395-b8f9-e47cce4f7dfe-kube-api-access-mfhk7\") pod \"89710086-0e78-4395-b8f9-e47cce4f7dfe\" (UID: \"89710086-0e78-4395-b8f9-e47cce4f7dfe\") "
Nov 02 15:36:04.511198 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:36:04.510897    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89710086-0e78-4395-b8f9-e47cce4f7dfe-credential-socket" pod "89710086-0e78-4395-b8f9-e47cce4f7dfe" (UID: "89710086-0e78-4395-b8f9-e47cce4f7dfe"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:36:04.511198 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:36:04.510953    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89710086-0e78-4395-b8f9-e47cce4f7dfe-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89710086-0e78-4395-b8f9-e47cce4f7dfe-workload-certs\") pod \"89710086-0e78-4395-b8f9-e47cce4f7dfe\" (UID: \"89710086-0e78-4395-b8f9-e47cce4f7dfe\") "
Nov 02 15:36:04.511198 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:36:04.510982    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0b48fb40-a77c-4345-a9d4-776bcdabb078-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b48fb40-a77c-4345-a9d4-776bcdabb078-istio-podinfo\") pod \"0b48fb40-a77c-4345-a9d4-776bcdabb078\" (UID: \"0b48fb40-a77c-4345-a9d4-776bcdabb078\") "
Nov 02 15:36:04.511198 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:36:04.511052    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89710086-0e78-4395-b8f9-e47cce4f7dfe-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89710086-0e78-4395-b8f9-e47cce4f7dfe-istio-envoy\") pod \"89710086-0e78-4395-b8f9-e47cce4f7dfe\" (UID: \"89710086-0e78-4395-b8f9-e47cce4f7dfe\") "
Nov 02 15:36:04.511198 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:36:04.511091    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89710086-0e78-4395-b8f9-e47cce4f7dfe-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89710086-0e78-4395-b8f9-e47cce4f7dfe-workload-socket\") pod \"89710086-0e78-4395-b8f9-e47cce4f7dfe\" (UID: \"89710086-0e78-4395-b8f9-e47cce4f7dfe\") "
Nov 02 15:36:04.511198 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:36:04.511106    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b48fb40-a77c-4345-a9d4-776bcdabb078-workload-socket" pod "0b48fb40-a77c-4345-a9d4-776bcdabb078" (UID: "0b48fb40-a77c-4345-a9d4-776bcdabb078"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:36:04.511198 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:36:04.511111    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89710086-0e78-4395-b8f9-e47cce4f7dfe-istio-data" pod "89710086-0e78-4395-b8f9-e47cce4f7dfe" (UID: "89710086-0e78-4395-b8f9-e47cce4f7dfe"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:36:04.511198 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:36:04.511153    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b48fb40-a77c-4345-a9d4-776bcdabb078-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b48fb40-a77c-4345-a9d4-776bcdabb078-istio-envoy\") pod \"0b48fb40-a77c-4345-a9d4-776bcdabb078\" (UID: \"0b48fb40-a77c-4345-a9d4-776bcdabb078\") "
Nov 02 15:36:04.511678 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:36:04.511210    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/89710086-0e78-4395-b8f9-e47cce4f7dfe-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89710086-0e78-4395-b8f9-e47cce4f7dfe-istiod-ca-cert\") pod \"89710086-0e78-4395-b8f9-e47cce4f7dfe\" (UID: \"89710086-0e78-4395-b8f9-e47cce4f7dfe\") "
Nov 02 15:36:04.511678 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:36:04.511258    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b48fb40-a77c-4345-a9d4-776bcdabb078-istio-token\" (UniqueName: \"kubernetes.io/projected/0b48fb40-a77c-4345-a9d4-776bcdabb078-istio-token\") pod \"0b48fb40-a77c-4345-a9d4-776bcdabb078\" (UID: \"0b48fb40-a77c-4345-a9d4-776bcdabb078\") "
Nov 02 15:36:04.511678 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:36:04.511384    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b48fb40-a77c-4345-a9d4-776bcdabb078-credential-socket" pod "0b48fb40-a77c-4345-a9d4-776bcdabb078" (UID: "0b48fb40-a77c-4345-a9d4-776bcdabb078"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:36:04.511678 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:36:04.511468    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89710086-0e78-4395-b8f9-e47cce4f7dfe-workload-socket" pod "89710086-0e78-4395-b8f9-e47cce4f7dfe" (UID: "89710086-0e78-4395-b8f9-e47cce4f7dfe"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:36:04.511678 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:36:04.511668    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b48fb40-a77c-4345-a9d4-776bcdabb078-istiod-ca-cert" pod "0b48fb40-a77c-4345-a9d4-776bcdabb078" (UID: "0b48fb40-a77c-4345-a9d4-776bcdabb078"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:36:04.511953 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:36:04.511911    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89710086-0e78-4395-b8f9-e47cce4f7dfe-workload-certs" pod "89710086-0e78-4395-b8f9-e47cce4f7dfe" (UID: "89710086-0e78-4395-b8f9-e47cce4f7dfe"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:36:04.512018 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:36:04.511973    5304 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b48fb40-a77c-4345-a9d4-776bcdabb078-istio-data\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:36:04.512018 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:36:04.511994    5304 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89710086-0e78-4395-b8f9-e47cce4f7dfe-istio-data\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:36:04.512018 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:36:04.512008    5304 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89710086-0e78-4395-b8f9-e47cce4f7dfe-credential-socket\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:36:04.512203 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:36:04.512024    5304 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b48fb40-a77c-4345-a9d4-776bcdabb078-istiod-ca-cert\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:36:04.512203 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:36:04.512037    5304 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b48fb40-a77c-4345-a9d4-776bcdabb078-credential-socket\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:36:04.512203 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:36:04.512049    5304 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b48fb40-a77c-4345-a9d4-776bcdabb078-workload-socket\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:36:04.512203 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:36:04.512063    5304 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89710086-0e78-4395-b8f9-e47cce4f7dfe-workload-socket\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:36:04.512203 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:36:04.512079    5304 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b48fb40-a77c-4345-a9d4-776bcdabb078-workload-certs\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:36:04.512203 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:36:04.512187    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89710086-0e78-4395-b8f9-e47cce4f7dfe-istiod-ca-cert" pod "89710086-0e78-4395-b8f9-e47cce4f7dfe" (UID: "89710086-0e78-4395-b8f9-e47cce4f7dfe"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:36:04.513461 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:36:04.513423    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0b48fb40-a77c-4345-a9d4-776bcdabb078-istio-podinfo" pod "0b48fb40-a77c-4345-a9d4-776bcdabb078" (UID: "0b48fb40-a77c-4345-a9d4-776bcdabb078"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 15:36:04.513703 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:36:04.513664    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/89710086-0e78-4395-b8f9-e47cce4f7dfe-istio-podinfo" pod "89710086-0e78-4395-b8f9-e47cce4f7dfe" (UID: "89710086-0e78-4395-b8f9-e47cce4f7dfe"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 15:36:04.514777 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:36:04.514749    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89710086-0e78-4395-b8f9-e47cce4f7dfe-istio-envoy" pod "89710086-0e78-4395-b8f9-e47cce4f7dfe" (UID: "89710086-0e78-4395-b8f9-e47cce4f7dfe"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:36:04.515357 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:36:04.515330    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b48fb40-a77c-4345-a9d4-776bcdabb078-kube-api-access-xmffj" pod "0b48fb40-a77c-4345-a9d4-776bcdabb078" (UID: "0b48fb40-a77c-4345-a9d4-776bcdabb078"). InnerVolumeSpecName "kube-api-access-xmffj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:36:04.515462 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:36:04.515365    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b48fb40-a77c-4345-a9d4-776bcdabb078-istio-envoy" pod "0b48fb40-a77c-4345-a9d4-776bcdabb078" (UID: "0b48fb40-a77c-4345-a9d4-776bcdabb078"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:36:04.515779 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:36:04.515752    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89710086-0e78-4395-b8f9-e47cce4f7dfe-istio-token" pod "89710086-0e78-4395-b8f9-e47cce4f7dfe" (UID: "89710086-0e78-4395-b8f9-e47cce4f7dfe"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:36:04.516239 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:36:04.516213    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89710086-0e78-4395-b8f9-e47cce4f7dfe-kube-api-access-mfhk7" pod "89710086-0e78-4395-b8f9-e47cce4f7dfe" (UID: "89710086-0e78-4395-b8f9-e47cce4f7dfe"). InnerVolumeSpecName "kube-api-access-mfhk7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:36:04.516748 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:36:04.516723    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b48fb40-a77c-4345-a9d4-776bcdabb078-istio-token" pod "0b48fb40-a77c-4345-a9d4-776bcdabb078" (UID: "0b48fb40-a77c-4345-a9d4-776bcdabb078"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:36:04.596635 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:36:04.596597    5304 generic.go:356] "Generic (PLEG): container finished" podID="0b48fb40-a77c-4345-a9d4-776bcdabb078" containerID="8dae976028ac237c8abcbbaf74aab1520943f99f8ed0e7c8875888f531c144c2" exitCode=0
Nov 02 15:36:04.596819 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:36:04.596661    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-g8tjj"
Nov 02 15:36:04.596819 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:36:04.596687    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-g8tjj" event={"ID":"0b48fb40-a77c-4345-a9d4-776bcdabb078","Type":"ContainerDied","Data":"8dae976028ac237c8abcbbaf74aab1520943f99f8ed0e7c8875888f531c144c2"}
Nov 02 15:36:04.596819 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:36:04.596768    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-g8tjj" event={"ID":"0b48fb40-a77c-4345-a9d4-776bcdabb078","Type":"ContainerDied","Data":"a5cadc32746496ae272409ec178c19410b6bbf9db2d7a5753600207f243cf8c2"}
Nov 02 15:36:04.596819 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:36:04.596801    5304 scope.go:122] "RemoveContainer" containerID="8dae976028ac237c8abcbbaf74aab1520943f99f8ed0e7c8875888f531c144c2"
Nov 02 15:36:04.598673 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:36:04.598646    5304 generic.go:356] "Generic (PLEG): container finished" podID="3ed68512-6850-45d4-af0d-a941c2bf6ec9" containerID="1442a2c5a420e08de2915db274b4fc9b201da678242c7a4d4e215143733d6a93" exitCode=0
Nov 02 15:36:04.598814 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:36:04.598703    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-25tqv"
Nov 02 15:36:04.598814 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:36:04.598735    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-25tqv" event={"ID":"3ed68512-6850-45d4-af0d-a941c2bf6ec9","Type":"ContainerDied","Data":"1442a2c5a420e08de2915db274b4fc9b201da678242c7a4d4e215143733d6a93"}
Nov 02 15:36:04.598814 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:36:04.598785    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-25tqv" event={"ID":"3ed68512-6850-45d4-af0d-a941c2bf6ec9","Type":"ContainerDied","Data":"ed3cfb3ddf9d5584473a5939c1c4f17ae2ee1220e947a2364f1106d9120fcf13"}
Nov 02 15:36:04.601270 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:36:04.601246    5304 generic.go:356] "Generic (PLEG): container finished" podID="89710086-0e78-4395-b8f9-e47cce4f7dfe" containerID="7141638a35a6e8d0eeea3c7ac8e21defc3c8db61481fb5e2a002c417397cc317" exitCode=0
Nov 02 15:36:04.601539 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:36:04.601338    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-pshqj" event={"ID":"89710086-0e78-4395-b8f9-e47cce4f7dfe","Type":"ContainerDied","Data":"7141638a35a6e8d0eeea3c7ac8e21defc3c8db61481fb5e2a002c417397cc317"}
Nov 02 15:36:04.601539 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:36:04.601389    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-pshqj" event={"ID":"89710086-0e78-4395-b8f9-e47cce4f7dfe","Type":"ContainerDied","Data":"b22b8b1d23aa8eac8a94fe384a915c78536066ad85d43a9e083e3c5d3d36208d"}
Nov 02 15:36:04.601539 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:36:04.601439    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-pshqj"
Nov 02 15:36:04.609426 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:36:04.609408    5304 scope.go:122] "RemoveContainer" containerID="925c184dcf230cd74bfef28c14af0b95596ad6cd7a419c7948359b1478c56603"
Nov 02 15:36:04.613495 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:36:04.613476    5304 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89710086-0e78-4395-b8f9-e47cce4f7dfe-workload-certs\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:36:04.613580 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:36:04.613498    5304 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b48fb40-a77c-4345-a9d4-776bcdabb078-istio-podinfo\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:36:04.613580 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:36:04.613508    5304 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89710086-0e78-4395-b8f9-e47cce4f7dfe-istio-envoy\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:36:04.613580 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:36:04.613518    5304 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b48fb40-a77c-4345-a9d4-776bcdabb078-istio-envoy\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:36:04.613580 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:36:04.613526    5304 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89710086-0e78-4395-b8f9-e47cce4f7dfe-istiod-ca-cert\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:36:04.613580 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:36:04.613534    5304 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b48fb40-a77c-4345-a9d4-776bcdabb078-istio-token\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:36:04.613580 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:36:04.613542    5304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xmffj\" (UniqueName: \"kubernetes.io/projected/0b48fb40-a77c-4345-a9d4-776bcdabb078-kube-api-access-xmffj\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:36:04.613580 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:36:04.613550    5304 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/89710086-0e78-4395-b8f9-e47cce4f7dfe-istio-token\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:36:04.613580 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:36:04.613560    5304 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89710086-0e78-4395-b8f9-e47cce4f7dfe-istio-podinfo\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:36:04.613580 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:36:04.613569    5304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mfhk7\" (UniqueName: \"kubernetes.io/projected/89710086-0e78-4395-b8f9-e47cce4f7dfe-kube-api-access-mfhk7\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:36:04.620356 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:36:04.620332    5304 scope.go:122] "RemoveContainer" containerID="8dae976028ac237c8abcbbaf74aab1520943f99f8ed0e7c8875888f531c144c2"
Nov 02 15:36:04.620637 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:36:04.620619    5304 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8dae976028ac237c8abcbbaf74aab1520943f99f8ed0e7c8875888f531c144c2\": container with ID starting with 8dae976028ac237c8abcbbaf74aab1520943f99f8ed0e7c8875888f531c144c2 not found: ID does not exist" containerID="8dae976028ac237c8abcbbaf74aab1520943f99f8ed0e7c8875888f531c144c2"
Nov 02 15:36:04.620693 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:36:04.620647    5304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8dae976028ac237c8abcbbaf74aab1520943f99f8ed0e7c8875888f531c144c2"} err="failed to get container status \"8dae976028ac237c8abcbbaf74aab1520943f99f8ed0e7c8875888f531c144c2\": rpc error: code = NotFound desc = could not find container \"8dae976028ac237c8abcbbaf74aab1520943f99f8ed0e7c8875888f531c144c2\": container with ID starting with 8dae976028ac237c8abcbbaf74aab1520943f99f8ed0e7c8875888f531c144c2 not found: ID does not exist"
Nov 02 15:36:04.620693 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:36:04.620665    5304 scope.go:122] "RemoveContainer" containerID="925c184dcf230cd74bfef28c14af0b95596ad6cd7a419c7948359b1478c56603"
Nov 02 15:36:04.620948 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:36:04.620928    5304 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"925c184dcf230cd74bfef28c14af0b95596ad6cd7a419c7948359b1478c56603\": container with ID starting with 925c184dcf230cd74bfef28c14af0b95596ad6cd7a419c7948359b1478c56603 not found: ID does not exist" containerID="925c184dcf230cd74bfef28c14af0b95596ad6cd7a419c7948359b1478c56603"
Nov 02 15:36:04.621000 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:36:04.620952    5304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"925c184dcf230cd74bfef28c14af0b95596ad6cd7a419c7948359b1478c56603"} err="failed to get container status \"925c184dcf230cd74bfef28c14af0b95596ad6cd7a419c7948359b1478c56603\": rpc error: code = NotFound desc = could not find container \"925c184dcf230cd74bfef28c14af0b95596ad6cd7a419c7948359b1478c56603\": container with ID starting with 925c184dcf230cd74bfef28c14af0b95596ad6cd7a419c7948359b1478c56603 not found: ID does not exist"
Nov 02 15:36:04.621000 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:36:04.620966    5304 scope.go:122] "RemoveContainer" containerID="1442a2c5a420e08de2915db274b4fc9b201da678242c7a4d4e215143733d6a93"
Nov 02 15:36:04.631802 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:36:04.631624    5304 scope.go:122] "RemoveContainer" containerID="eecc968f16d732bb7fb9155081d87e858ec89a885a7242f9502bda03fad23c51"
Nov 02 15:36:04.641272 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:36:04.641246    5304 scope.go:122] "RemoveContainer" containerID="1442a2c5a420e08de2915db274b4fc9b201da678242c7a4d4e215143733d6a93"
Nov 02 15:36:04.641546 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:36:04.641519    5304 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1442a2c5a420e08de2915db274b4fc9b201da678242c7a4d4e215143733d6a93\": container with ID starting with 1442a2c5a420e08de2915db274b4fc9b201da678242c7a4d4e215143733d6a93 not found: ID does not exist" containerID="1442a2c5a420e08de2915db274b4fc9b201da678242c7a4d4e215143733d6a93"
Nov 02 15:36:04.641617 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:36:04.641548    5304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1442a2c5a420e08de2915db274b4fc9b201da678242c7a4d4e215143733d6a93"} err="failed to get container status \"1442a2c5a420e08de2915db274b4fc9b201da678242c7a4d4e215143733d6a93\": rpc error: code = NotFound desc = could not find container \"1442a2c5a420e08de2915db274b4fc9b201da678242c7a4d4e215143733d6a93\": container with ID starting with 1442a2c5a420e08de2915db274b4fc9b201da678242c7a4d4e215143733d6a93 not found: ID does not exist"
Nov 02 15:36:04.641617 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:36:04.641565    5304 scope.go:122] "RemoveContainer" containerID="eecc968f16d732bb7fb9155081d87e858ec89a885a7242f9502bda03fad23c51"
Nov 02 15:36:04.641824 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:36:04.641807    5304 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eecc968f16d732bb7fb9155081d87e858ec89a885a7242f9502bda03fad23c51\": container with ID starting with eecc968f16d732bb7fb9155081d87e858ec89a885a7242f9502bda03fad23c51 not found: ID does not exist" containerID="eecc968f16d732bb7fb9155081d87e858ec89a885a7242f9502bda03fad23c51"
Nov 02 15:36:04.641862 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:36:04.641831    5304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eecc968f16d732bb7fb9155081d87e858ec89a885a7242f9502bda03fad23c51"} err="failed to get container status \"eecc968f16d732bb7fb9155081d87e858ec89a885a7242f9502bda03fad23c51\": rpc error: code = NotFound desc = could not find container \"eecc968f16d732bb7fb9155081d87e858ec89a885a7242f9502bda03fad23c51\": container with ID starting with eecc968f16d732bb7fb9155081d87e858ec89a885a7242f9502bda03fad23c51 not found: ID does not exist"
Nov 02 15:36:04.641862 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:36:04.641847    5304 scope.go:122] "RemoveContainer" containerID="7141638a35a6e8d0eeea3c7ac8e21defc3c8db61481fb5e2a002c417397cc317"
Nov 02 15:36:04.654019 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:36:04.653935    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-25tqv"]
Nov 02 15:36:04.665633 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:36:04.665509    5304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-25tqv"]
Nov 02 15:36:04.665709 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:36:04.665691    5304 scope.go:122] "RemoveContainer" containerID="6dd484c90f59525171dfb12122eb7a12a797fd5f1bf98e31a7ef3f7c6170893b"
Nov 02 15:36:04.676782 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:36:04.676763    5304 scope.go:122] "RemoveContainer" containerID="7141638a35a6e8d0eeea3c7ac8e21defc3c8db61481fb5e2a002c417397cc317"
Nov 02 15:36:04.677087 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:36:04.677068    5304 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7141638a35a6e8d0eeea3c7ac8e21defc3c8db61481fb5e2a002c417397cc317\": container with ID starting with 7141638a35a6e8d0eeea3c7ac8e21defc3c8db61481fb5e2a002c417397cc317 not found: ID does not exist" containerID="7141638a35a6e8d0eeea3c7ac8e21defc3c8db61481fb5e2a002c417397cc317"
Nov 02 15:36:04.677157 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:36:04.677096    5304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7141638a35a6e8d0eeea3c7ac8e21defc3c8db61481fb5e2a002c417397cc317"} err="failed to get container status \"7141638a35a6e8d0eeea3c7ac8e21defc3c8db61481fb5e2a002c417397cc317\": rpc error: code = NotFound desc = could not find container \"7141638a35a6e8d0eeea3c7ac8e21defc3c8db61481fb5e2a002c417397cc317\": container with ID starting with 7141638a35a6e8d0eeea3c7ac8e21defc3c8db61481fb5e2a002c417397cc317 not found: ID does not exist"
Nov 02 15:36:04.677157 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:36:04.677117    5304 scope.go:122] "RemoveContainer" containerID="6dd484c90f59525171dfb12122eb7a12a797fd5f1bf98e31a7ef3f7c6170893b"
Nov 02 15:36:04.677413 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:36:04.677393    5304 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6dd484c90f59525171dfb12122eb7a12a797fd5f1bf98e31a7ef3f7c6170893b\": container with ID starting with 6dd484c90f59525171dfb12122eb7a12a797fd5f1bf98e31a7ef3f7c6170893b not found: ID does not exist" containerID="6dd484c90f59525171dfb12122eb7a12a797fd5f1bf98e31a7ef3f7c6170893b"
Nov 02 15:36:04.677462 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:36:04.677420    5304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6dd484c90f59525171dfb12122eb7a12a797fd5f1bf98e31a7ef3f7c6170893b"} err="failed to get container status \"6dd484c90f59525171dfb12122eb7a12a797fd5f1bf98e31a7ef3f7c6170893b\": rpc error: code = NotFound desc = could not find container \"6dd484c90f59525171dfb12122eb7a12a797fd5f1bf98e31a7ef3f7c6170893b\": container with ID starting with 6dd484c90f59525171dfb12122eb7a12a797fd5f1bf98e31a7ef3f7c6170893b not found: ID does not exist"
Nov 02 15:36:04.698295 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:36:04.698257    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-g8tjj"]
Nov 02 15:36:04.703502 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:36:04.703473    5304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-g8tjj"]
Nov 02 15:36:04.738364 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:36:04.738328    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-pshqj"]
Nov 02 15:36:04.745660 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:36:04.745630    5304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-pshqj"]
Nov 02 15:36:06.120001 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:36:06.119969    5304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0b48fb40-a77c-4345-a9d4-776bcdabb078" path="/var/lib/kubelet/pods/0b48fb40-a77c-4345-a9d4-776bcdabb078/volumes"
Nov 02 15:36:06.120593 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:36:06.120576    5304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3ed68512-6850-45d4-af0d-a941c2bf6ec9" path="/var/lib/kubelet/pods/3ed68512-6850-45d4-af0d-a941c2bf6ec9/volumes"
Nov 02 15:36:06.121051 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:36:06.121036    5304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="89710086-0e78-4395-b8f9-e47cce4f7dfe" path="/var/lib/kubelet/pods/89710086-0e78-4395-b8f9-e47cce4f7dfe/volumes"
Nov 02 15:36:07.235036 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:36:07.235003    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-9gbvl"]
Nov 02 15:36:07.235491 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:36:07.235239    5304 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-9gbvl" podUID="7a8625f9-7163-4e64-8e52-6537ebbe404c" containerName="istio-proxy" containerID="cri-o://1e6f17353a53455da6b25fef982bf958f8bf3f468acd79fe65343e426bda100c" gracePeriod=30
Nov 02 15:36:07.320277 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:36:07.320238    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-28f621-dd989bdc-9x66m" podUID="dc422942-41f9-4cee-938f-e9df1b05d0c7" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 15:36:12.539983 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:36:12.539955    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-9gbvl"
Nov 02 15:36:12.627539 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:36:12.627511    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a8625f9-7163-4e64-8e52-6537ebbe404c-kube-api-access-x4v96\" (UniqueName: \"kubernetes.io/projected/7a8625f9-7163-4e64-8e52-6537ebbe404c-kube-api-access-x4v96\") pod \"7a8625f9-7163-4e64-8e52-6537ebbe404c\" (UID: \"7a8625f9-7163-4e64-8e52-6537ebbe404c\") "
Nov 02 15:36:12.627730 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:36:12.627602    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a8625f9-7163-4e64-8e52-6537ebbe404c-istio-token\" (UniqueName: \"kubernetes.io/projected/7a8625f9-7163-4e64-8e52-6537ebbe404c-istio-token\") pod \"7a8625f9-7163-4e64-8e52-6537ebbe404c\" (UID: \"7a8625f9-7163-4e64-8e52-6537ebbe404c\") "
Nov 02 15:36:12.627730 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:36:12.627691    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7a8625f9-7163-4e64-8e52-6537ebbe404c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a8625f9-7163-4e64-8e52-6537ebbe404c-istiod-ca-cert\") pod \"7a8625f9-7163-4e64-8e52-6537ebbe404c\" (UID: \"7a8625f9-7163-4e64-8e52-6537ebbe404c\") "
Nov 02 15:36:12.627894 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:36:12.627779    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a8625f9-7163-4e64-8e52-6537ebbe404c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a8625f9-7163-4e64-8e52-6537ebbe404c-credential-socket\") pod \"7a8625f9-7163-4e64-8e52-6537ebbe404c\" (UID: \"7a8625f9-7163-4e64-8e52-6537ebbe404c\") "
Nov 02 15:36:12.628207 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:36:12.628173    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a8625f9-7163-4e64-8e52-6537ebbe404c-credential-socket" pod "7a8625f9-7163-4e64-8e52-6537ebbe404c" (UID: "7a8625f9-7163-4e64-8e52-6537ebbe404c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:36:12.628207 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:36:12.628195    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a8625f9-7163-4e64-8e52-6537ebbe404c-istiod-ca-cert" pod "7a8625f9-7163-4e64-8e52-6537ebbe404c" (UID: "7a8625f9-7163-4e64-8e52-6537ebbe404c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:36:12.628382 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:36:12.628291    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a8625f9-7163-4e64-8e52-6537ebbe404c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a8625f9-7163-4e64-8e52-6537ebbe404c-workload-socket\") pod \"7a8625f9-7163-4e64-8e52-6537ebbe404c\" (UID: \"7a8625f9-7163-4e64-8e52-6537ebbe404c\") "
Nov 02 15:36:12.628558 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:36:12.628535    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a8625f9-7163-4e64-8e52-6537ebbe404c-workload-socket" pod "7a8625f9-7163-4e64-8e52-6537ebbe404c" (UID: "7a8625f9-7163-4e64-8e52-6537ebbe404c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:36:12.628632 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:36:12.628595    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a8625f9-7163-4e64-8e52-6537ebbe404c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a8625f9-7163-4e64-8e52-6537ebbe404c-istio-envoy\") pod \"7a8625f9-7163-4e64-8e52-6537ebbe404c\" (UID: \"7a8625f9-7163-4e64-8e52-6537ebbe404c\") "
Nov 02 15:36:12.628696 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:36:12.628669    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a8625f9-7163-4e64-8e52-6537ebbe404c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a8625f9-7163-4e64-8e52-6537ebbe404c-istio-data\") pod \"7a8625f9-7163-4e64-8e52-6537ebbe404c\" (UID: \"7a8625f9-7163-4e64-8e52-6537ebbe404c\") "
Nov 02 15:36:12.628755 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:36:12.628734    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a8625f9-7163-4e64-8e52-6537ebbe404c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a8625f9-7163-4e64-8e52-6537ebbe404c-workload-certs\") pod \"7a8625f9-7163-4e64-8e52-6537ebbe404c\" (UID: \"7a8625f9-7163-4e64-8e52-6537ebbe404c\") "
Nov 02 15:36:12.628808 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:36:12.628788    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7a8625f9-7163-4e64-8e52-6537ebbe404c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a8625f9-7163-4e64-8e52-6537ebbe404c-istio-podinfo\") pod \"7a8625f9-7163-4e64-8e52-6537ebbe404c\" (UID: \"7a8625f9-7163-4e64-8e52-6537ebbe404c\") "
Nov 02 15:36:12.629423 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:36:12.629396    5304 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a8625f9-7163-4e64-8e52-6537ebbe404c-workload-socket\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:36:12.629527 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:36:12.629428    5304 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a8625f9-7163-4e64-8e52-6537ebbe404c-istiod-ca-cert\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:36:12.629527 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:36:12.629444    5304 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a8625f9-7163-4e64-8e52-6537ebbe404c-credential-socket\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:36:12.629877 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:36:12.629850    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a8625f9-7163-4e64-8e52-6537ebbe404c-istio-data" pod "7a8625f9-7163-4e64-8e52-6537ebbe404c" (UID: "7a8625f9-7163-4e64-8e52-6537ebbe404c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:36:12.630377 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:36:12.630351    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a8625f9-7163-4e64-8e52-6537ebbe404c-workload-certs" pod "7a8625f9-7163-4e64-8e52-6537ebbe404c" (UID: "7a8625f9-7163-4e64-8e52-6537ebbe404c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:36:12.632319 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:36:12.632293    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a8625f9-7163-4e64-8e52-6537ebbe404c-kube-api-access-x4v96" pod "7a8625f9-7163-4e64-8e52-6537ebbe404c" (UID: "7a8625f9-7163-4e64-8e52-6537ebbe404c"). InnerVolumeSpecName "kube-api-access-x4v96". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:36:12.632424 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:36:12.632328    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a8625f9-7163-4e64-8e52-6537ebbe404c-istio-token" pod "7a8625f9-7163-4e64-8e52-6537ebbe404c" (UID: "7a8625f9-7163-4e64-8e52-6537ebbe404c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:36:12.633535 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:36:12.633513    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7a8625f9-7163-4e64-8e52-6537ebbe404c-istio-podinfo" pod "7a8625f9-7163-4e64-8e52-6537ebbe404c" (UID: "7a8625f9-7163-4e64-8e52-6537ebbe404c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 15:36:12.634007 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:36:12.633984    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a8625f9-7163-4e64-8e52-6537ebbe404c-istio-envoy" pod "7a8625f9-7163-4e64-8e52-6537ebbe404c" (UID: "7a8625f9-7163-4e64-8e52-6537ebbe404c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:36:12.660706 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:36:12.660667    5304 generic.go:356] "Generic (PLEG): container finished" podID="7a8625f9-7163-4e64-8e52-6537ebbe404c" containerID="1e6f17353a53455da6b25fef982bf958f8bf3f468acd79fe65343e426bda100c" exitCode=0
Nov 02 15:36:12.660875 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:36:12.660722    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-9gbvl"
Nov 02 15:36:12.660875 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:36:12.660761    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-9gbvl" event={"ID":"7a8625f9-7163-4e64-8e52-6537ebbe404c","Type":"ContainerDied","Data":"1e6f17353a53455da6b25fef982bf958f8bf3f468acd79fe65343e426bda100c"}
Nov 02 15:36:12.660875 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:36:12.660823    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-9gbvl" event={"ID":"7a8625f9-7163-4e64-8e52-6537ebbe404c","Type":"ContainerDied","Data":"77359c238e91c6da5374d370a28848fdd8f918e1400c28e44ffbf125c1991887"}
Nov 02 15:36:12.660875 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:36:12.660851    5304 scope.go:122] "RemoveContainer" containerID="1e6f17353a53455da6b25fef982bf958f8bf3f468acd79fe65343e426bda100c"
Nov 02 15:36:12.677147 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:36:12.677108    5304 scope.go:122] "RemoveContainer" containerID="1e6f17353a53455da6b25fef982bf958f8bf3f468acd79fe65343e426bda100c"
Nov 02 15:36:12.677374 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:36:12.677355    5304 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e6f17353a53455da6b25fef982bf958f8bf3f468acd79fe65343e426bda100c\": container with ID starting with 1e6f17353a53455da6b25fef982bf958f8bf3f468acd79fe65343e426bda100c not found: ID does not exist" containerID="1e6f17353a53455da6b25fef982bf958f8bf3f468acd79fe65343e426bda100c"
Nov 02 15:36:12.677425 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:36:12.677384    5304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e6f17353a53455da6b25fef982bf958f8bf3f468acd79fe65343e426bda100c"} err="failed to get container status \"1e6f17353a53455da6b25fef982bf958f8bf3f468acd79fe65343e426bda100c\": rpc error: code = NotFound desc = could not find container \"1e6f17353a53455da6b25fef982bf958f8bf3f468acd79fe65343e426bda100c\": container with ID starting with 1e6f17353a53455da6b25fef982bf958f8bf3f468acd79fe65343e426bda100c not found: ID does not exist"
Nov 02 15:36:12.717701 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:36:12.717661    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-9gbvl"]
Nov 02 15:36:12.720756 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:36:12.720726    5304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-9gbvl"]
Nov 02 15:36:12.731072 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:36:12.731043    5304 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a8625f9-7163-4e64-8e52-6537ebbe404c-istio-token\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:36:12.731072 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:36:12.731072    5304 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a8625f9-7163-4e64-8e52-6537ebbe404c-istio-envoy\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:36:12.731259 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:36:12.731081    5304 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a8625f9-7163-4e64-8e52-6537ebbe404c-istio-data\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:36:12.731259 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:36:12.731089    5304 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a8625f9-7163-4e64-8e52-6537ebbe404c-workload-certs\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:36:12.731259 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:36:12.731097    5304 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a8625f9-7163-4e64-8e52-6537ebbe404c-istio-podinfo\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:36:12.731259 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:36:12.731105    5304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x4v96\" (UniqueName: \"kubernetes.io/projected/7a8625f9-7163-4e64-8e52-6537ebbe404c-kube-api-access-x4v96\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:36:14.119303 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:36:14.119265    5304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7a8625f9-7163-4e64-8e52-6537ebbe404c" path="/var/lib/kubelet/pods/7a8625f9-7163-4e64-8e52-6537ebbe404c/volumes"
Nov 02 15:36:17.320349 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:36:17.320305    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-28f621-dd989bdc-9x66m" podUID="dc422942-41f9-4cee-938f-e9df1b05d0c7" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 15:36:24.422277 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:36:24.422213    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-28f621-dd989bdc-9x66m"
Nov 02 15:36:24.498047 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:36:24.498013    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dc422942-41f9-4cee-938f-e9df1b05d0c7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc422942-41f9-4cee-938f-e9df1b05d0c7-plugins-config-volume\") pod \"dc422942-41f9-4cee-938f-e9df1b05d0c7\" (UID: \"dc422942-41f9-4cee-938f-e9df1b05d0c7\") "
Nov 02 15:36:24.498252 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:36:24.498216    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc422942-41f9-4cee-938f-e9df1b05d0c7-kube-api-access-g8xlb\" (UniqueName: \"kubernetes.io/projected/dc422942-41f9-4cee-938f-e9df1b05d0c7-kube-api-access-g8xlb\") pod \"dc422942-41f9-4cee-938f-e9df1b05d0c7\" (UID: \"dc422942-41f9-4cee-938f-e9df1b05d0c7\") "
Nov 02 15:36:24.498466 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:36:24.498441    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc422942-41f9-4cee-938f-e9df1b05d0c7-plugins-config-volume" pod "dc422942-41f9-4cee-938f-e9df1b05d0c7" (UID: "dc422942-41f9-4cee-938f-e9df1b05d0c7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:36:24.498682 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:36:24.498664    5304 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc422942-41f9-4cee-938f-e9df1b05d0c7-plugins-config-volume\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:36:24.500447 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:36:24.500419    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc422942-41f9-4cee-938f-e9df1b05d0c7-kube-api-access-g8xlb" pod "dc422942-41f9-4cee-938f-e9df1b05d0c7" (UID: "dc422942-41f9-4cee-938f-e9df1b05d0c7"). InnerVolumeSpecName "kube-api-access-g8xlb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:36:24.599518 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:36:24.599483    5304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g8xlb\" (UniqueName: \"kubernetes.io/projected/dc422942-41f9-4cee-938f-e9df1b05d0c7-kube-api-access-g8xlb\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:36:24.762824 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:36:24.762729    5304 generic.go:356] "Generic (PLEG): container finished" podID="dc422942-41f9-4cee-938f-e9df1b05d0c7" containerID="6cf882f73d0856520ef43074ea73feb0d1d6276d15f6807cdc1c4dde65dbe01a" exitCode=0
Nov 02 15:36:24.762824 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:36:24.762791    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-28f621-dd989bdc-9x66m"
Nov 02 15:36:24.763077 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:36:24.762813    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-28f621-dd989bdc-9x66m" event={"ID":"dc422942-41f9-4cee-938f-e9df1b05d0c7","Type":"ContainerDied","Data":"6cf882f73d0856520ef43074ea73feb0d1d6276d15f6807cdc1c4dde65dbe01a"}
Nov 02 15:36:24.763077 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:36:24.762906    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-28f621-dd989bdc-9x66m" event={"ID":"dc422942-41f9-4cee-938f-e9df1b05d0c7","Type":"ContainerDied","Data":"acdfe4af7fa49549631af69f5bfd19d95832844d8433670d6ad076c79109f9c5"}
Nov 02 15:36:24.763077 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:36:24.762937    5304 scope.go:122] "RemoveContainer" containerID="6cf882f73d0856520ef43074ea73feb0d1d6276d15f6807cdc1c4dde65dbe01a"
Nov 02 15:36:24.773642 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:36:24.773617    5304 scope.go:122] "RemoveContainer" containerID="6cf882f73d0856520ef43074ea73feb0d1d6276d15f6807cdc1c4dde65dbe01a"
Nov 02 15:36:24.773941 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:36:24.773922    5304 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6cf882f73d0856520ef43074ea73feb0d1d6276d15f6807cdc1c4dde65dbe01a\": container with ID starting with 6cf882f73d0856520ef43074ea73feb0d1d6276d15f6807cdc1c4dde65dbe01a not found: ID does not exist" containerID="6cf882f73d0856520ef43074ea73feb0d1d6276d15f6807cdc1c4dde65dbe01a"
Nov 02 15:36:24.774021 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:36:24.773949    5304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6cf882f73d0856520ef43074ea73feb0d1d6276d15f6807cdc1c4dde65dbe01a"} err="failed to get container status \"6cf882f73d0856520ef43074ea73feb0d1d6276d15f6807cdc1c4dde65dbe01a\": rpc error: code = NotFound desc = could not find container \"6cf882f73d0856520ef43074ea73feb0d1d6276d15f6807cdc1c4dde65dbe01a\": container with ID starting with 6cf882f73d0856520ef43074ea73feb0d1d6276d15f6807cdc1c4dde65dbe01a not found: ID does not exist"
Nov 02 15:36:24.798762 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:36:24.798730    5304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k6fjk" containerName="extractor"
Nov 02 15:36:24.818385 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:36:24.818358    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-28f621-dd989bdc-9x66m"]
Nov 02 15:36:24.829958 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:36:24.829925    5304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-28f621-dd989bdc-9x66m"]
Nov 02 15:36:26.119765 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:36:26.119729    5304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dc422942-41f9-4cee-938f-e9df1b05d0c7" path="/var/lib/kubelet/pods/dc422942-41f9-4cee-938f-e9df1b05d0c7/volumes"
Nov 02 15:36:30.997783 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:36:30.997746    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-28f621-openshift-default-7dd8c5c789-jvmsw"]
Nov 02 15:36:30.998317 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:36:30.998007    5304 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-28f621-openshift-default-7dd8c5c789-jvmsw" podUID="e4bb2a0e-a2da-44e1-ac19-ee296cfff71c" containerName="istio-proxy" containerID="cri-o://cfc0b88601501226138466e5118589e463cd9542b9b211e6989599faf715e907" gracePeriod=30
Nov 02 15:36:34.268379 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:36:34.268335    5304 patch_prober.go:29] interesting pod/e2e-inf-a-28f621-openshift-default-7dd8c5c789-jvmsw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 15:36:34.268806 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:36:34.268421    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-28f621-openshift-default-7dd8c5c789-jvmsw" podUID="e4bb2a0e-a2da-44e1-ac19-ee296cfff71c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 15:36:36.285688 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:36:36.285664    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-28f621-openshift-default-7dd8c5c789-jvmsw"
Nov 02 15:36:36.375767 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:36:36.375722    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4bb2a0e-a2da-44e1-ac19-ee296cfff71c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4bb2a0e-a2da-44e1-ac19-ee296cfff71c-istio-envoy\") pod \"e4bb2a0e-a2da-44e1-ac19-ee296cfff71c\" (UID: \"e4bb2a0e-a2da-44e1-ac19-ee296cfff71c\") "
Nov 02 15:36:36.375967 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:36:36.375815    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e4bb2a0e-a2da-44e1-ac19-ee296cfff71c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4bb2a0e-a2da-44e1-ac19-ee296cfff71c-istio-podinfo\") pod \"e4bb2a0e-a2da-44e1-ac19-ee296cfff71c\" (UID: \"e4bb2a0e-a2da-44e1-ac19-ee296cfff71c\") "
Nov 02 15:36:36.375967 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:36:36.375927    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4bb2a0e-a2da-44e1-ac19-ee296cfff71c-kube-api-access-pw7kj\" (UniqueName: \"kubernetes.io/projected/e4bb2a0e-a2da-44e1-ac19-ee296cfff71c-kube-api-access-pw7kj\") pod \"e4bb2a0e-a2da-44e1-ac19-ee296cfff71c\" (UID: \"e4bb2a0e-a2da-44e1-ac19-ee296cfff71c\") "
Nov 02 15:36:36.376102 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:36:36.375969    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4bb2a0e-a2da-44e1-ac19-ee296cfff71c-istio-token\" (UniqueName: \"kubernetes.io/projected/e4bb2a0e-a2da-44e1-ac19-ee296cfff71c-istio-token\") pod \"e4bb2a0e-a2da-44e1-ac19-ee296cfff71c\" (UID: \"e4bb2a0e-a2da-44e1-ac19-ee296cfff71c\") "
Nov 02 15:36:36.376102 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:36:36.376007    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e4bb2a0e-a2da-44e1-ac19-ee296cfff71c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4bb2a0e-a2da-44e1-ac19-ee296cfff71c-istiod-ca-cert\") pod \"e4bb2a0e-a2da-44e1-ac19-ee296cfff71c\" (UID: \"e4bb2a0e-a2da-44e1-ac19-ee296cfff71c\") "
Nov 02 15:36:36.376102 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:36:36.376075    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4bb2a0e-a2da-44e1-ac19-ee296cfff71c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4bb2a0e-a2da-44e1-ac19-ee296cfff71c-workload-certs\") pod \"e4bb2a0e-a2da-44e1-ac19-ee296cfff71c\" (UID: \"e4bb2a0e-a2da-44e1-ac19-ee296cfff71c\") "
Nov 02 15:36:36.376305 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:36:36.376118    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4bb2a0e-a2da-44e1-ac19-ee296cfff71c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4bb2a0e-a2da-44e1-ac19-ee296cfff71c-workload-socket\") pod \"e4bb2a0e-a2da-44e1-ac19-ee296cfff71c\" (UID: \"e4bb2a0e-a2da-44e1-ac19-ee296cfff71c\") "
Nov 02 15:36:36.376305 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:36:36.376217    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4bb2a0e-a2da-44e1-ac19-ee296cfff71c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4bb2a0e-a2da-44e1-ac19-ee296cfff71c-credential-socket\") pod \"e4bb2a0e-a2da-44e1-ac19-ee296cfff71c\" (UID: \"e4bb2a0e-a2da-44e1-ac19-ee296cfff71c\") "
Nov 02 15:36:36.376305 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:36:36.376293    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4bb2a0e-a2da-44e1-ac19-ee296cfff71c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4bb2a0e-a2da-44e1-ac19-ee296cfff71c-istio-data\") pod \"e4bb2a0e-a2da-44e1-ac19-ee296cfff71c\" (UID: \"e4bb2a0e-a2da-44e1-ac19-ee296cfff71c\") "
Nov 02 15:36:36.376782 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:36:36.376754    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4bb2a0e-a2da-44e1-ac19-ee296cfff71c-workload-certs" pod "e4bb2a0e-a2da-44e1-ac19-ee296cfff71c" (UID: "e4bb2a0e-a2da-44e1-ac19-ee296cfff71c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:36:36.376955 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:36:36.376754    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4bb2a0e-a2da-44e1-ac19-ee296cfff71c-workload-socket" pod "e4bb2a0e-a2da-44e1-ac19-ee296cfff71c" (UID: "e4bb2a0e-a2da-44e1-ac19-ee296cfff71c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:36:36.377075 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:36:36.377023    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4bb2a0e-a2da-44e1-ac19-ee296cfff71c-istio-data" pod "e4bb2a0e-a2da-44e1-ac19-ee296cfff71c" (UID: "e4bb2a0e-a2da-44e1-ac19-ee296cfff71c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:36:36.377075 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:36:36.376774    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4bb2a0e-a2da-44e1-ac19-ee296cfff71c-credential-socket" pod "e4bb2a0e-a2da-44e1-ac19-ee296cfff71c" (UID: "e4bb2a0e-a2da-44e1-ac19-ee296cfff71c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:36:36.377075 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:36:36.377046    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4bb2a0e-a2da-44e1-ac19-ee296cfff71c-istiod-ca-cert" pod "e4bb2a0e-a2da-44e1-ac19-ee296cfff71c" (UID: "e4bb2a0e-a2da-44e1-ac19-ee296cfff71c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:36:36.377382 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:36:36.377360    5304 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4bb2a0e-a2da-44e1-ac19-ee296cfff71c-istiod-ca-cert\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:36:36.377451 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:36:36.377388    5304 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4bb2a0e-a2da-44e1-ac19-ee296cfff71c-workload-certs\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:36:36.377451 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:36:36.377401    5304 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4bb2a0e-a2da-44e1-ac19-ee296cfff71c-workload-socket\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:36:36.377451 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:36:36.377416    5304 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4bb2a0e-a2da-44e1-ac19-ee296cfff71c-credential-socket\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:36:36.377451 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:36:36.377431    5304 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4bb2a0e-a2da-44e1-ac19-ee296cfff71c-istio-data\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:36:36.378386 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:36:36.378354    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e4bb2a0e-a2da-44e1-ac19-ee296cfff71c-istio-podinfo" pod "e4bb2a0e-a2da-44e1-ac19-ee296cfff71c" (UID: "e4bb2a0e-a2da-44e1-ac19-ee296cfff71c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 15:36:36.378889 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:36:36.378860    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4bb2a0e-a2da-44e1-ac19-ee296cfff71c-kube-api-access-pw7kj" pod "e4bb2a0e-a2da-44e1-ac19-ee296cfff71c" (UID: "e4bb2a0e-a2da-44e1-ac19-ee296cfff71c"). InnerVolumeSpecName "kube-api-access-pw7kj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:36:36.379476 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:36:36.379444    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4bb2a0e-a2da-44e1-ac19-ee296cfff71c-istio-token" pod "e4bb2a0e-a2da-44e1-ac19-ee296cfff71c" (UID: "e4bb2a0e-a2da-44e1-ac19-ee296cfff71c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:36:36.379809 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:36:36.379789    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4bb2a0e-a2da-44e1-ac19-ee296cfff71c-istio-envoy" pod "e4bb2a0e-a2da-44e1-ac19-ee296cfff71c" (UID: "e4bb2a0e-a2da-44e1-ac19-ee296cfff71c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:36:36.478648 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:36:36.478613    5304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pw7kj\" (UniqueName: \"kubernetes.io/projected/e4bb2a0e-a2da-44e1-ac19-ee296cfff71c-kube-api-access-pw7kj\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:36:36.478648 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:36:36.478646    5304 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e4bb2a0e-a2da-44e1-ac19-ee296cfff71c-istio-token\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:36:36.478648 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:36:36.478655    5304 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4bb2a0e-a2da-44e1-ac19-ee296cfff71c-istio-envoy\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:36:36.478892 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:36:36.478663    5304 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4bb2a0e-a2da-44e1-ac19-ee296cfff71c-istio-podinfo\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:36:36.849738 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:36:36.849705    5304 generic.go:356] "Generic (PLEG): container finished" podID="e4bb2a0e-a2da-44e1-ac19-ee296cfff71c" containerID="cfc0b88601501226138466e5118589e463cd9542b9b211e6989599faf715e907" exitCode=0
Nov 02 15:36:36.849929 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:36:36.849766    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-28f621-openshift-default-7dd8c5c789-jvmsw"
Nov 02 15:36:36.849929 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:36:36.849779    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-28f621-openshift-default-7dd8c5c789-jvmsw" event={"ID":"e4bb2a0e-a2da-44e1-ac19-ee296cfff71c","Type":"ContainerDied","Data":"cfc0b88601501226138466e5118589e463cd9542b9b211e6989599faf715e907"}
Nov 02 15:36:36.849929 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:36:36.849833    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-28f621-openshift-default-7dd8c5c789-jvmsw" event={"ID":"e4bb2a0e-a2da-44e1-ac19-ee296cfff71c","Type":"ContainerDied","Data":"f0dc7449945750645e479080b03b5dabdbdf3eab1f2ec3f5e06d49702bb5e9b1"}
Nov 02 15:36:36.849929 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:36:36.849854    5304 scope.go:122] "RemoveContainer" containerID="cfc0b88601501226138466e5118589e463cd9542b9b211e6989599faf715e907"
Nov 02 15:36:36.861771 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:36:36.861745    5304 scope.go:122] "RemoveContainer" containerID="7e4a3e30cfa510238243eb2e994d8e6d8236791608cbebd8fb72a14a3396aa63"
Nov 02 15:36:36.871296 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:36:36.871276    5304 scope.go:122] "RemoveContainer" containerID="cfc0b88601501226138466e5118589e463cd9542b9b211e6989599faf715e907"
Nov 02 15:36:36.871572 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:36:36.871551    5304 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cfc0b88601501226138466e5118589e463cd9542b9b211e6989599faf715e907\": container with ID starting with cfc0b88601501226138466e5118589e463cd9542b9b211e6989599faf715e907 not found: ID does not exist" containerID="cfc0b88601501226138466e5118589e463cd9542b9b211e6989599faf715e907"
Nov 02 15:36:36.871630 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:36:36.871582    5304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cfc0b88601501226138466e5118589e463cd9542b9b211e6989599faf715e907"} err="failed to get container status \"cfc0b88601501226138466e5118589e463cd9542b9b211e6989599faf715e907\": rpc error: code = NotFound desc = could not find container \"cfc0b88601501226138466e5118589e463cd9542b9b211e6989599faf715e907\": container with ID starting with cfc0b88601501226138466e5118589e463cd9542b9b211e6989599faf715e907 not found: ID does not exist"
Nov 02 15:36:36.871630 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:36:36.871604    5304 scope.go:122] "RemoveContainer" containerID="7e4a3e30cfa510238243eb2e994d8e6d8236791608cbebd8fb72a14a3396aa63"
Nov 02 15:36:36.871841 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:36:36.871822    5304 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e4a3e30cfa510238243eb2e994d8e6d8236791608cbebd8fb72a14a3396aa63\": container with ID starting with 7e4a3e30cfa510238243eb2e994d8e6d8236791608cbebd8fb72a14a3396aa63 not found: ID does not exist" containerID="7e4a3e30cfa510238243eb2e994d8e6d8236791608cbebd8fb72a14a3396aa63"
Nov 02 15:36:36.871900 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:36:36.871853    5304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e4a3e30cfa510238243eb2e994d8e6d8236791608cbebd8fb72a14a3396aa63"} err="failed to get container status \"7e4a3e30cfa510238243eb2e994d8e6d8236791608cbebd8fb72a14a3396aa63\": rpc error: code = NotFound desc = could not find container \"7e4a3e30cfa510238243eb2e994d8e6d8236791608cbebd8fb72a14a3396aa63\": container with ID starting with 7e4a3e30cfa510238243eb2e994d8e6d8236791608cbebd8fb72a14a3396aa63 not found: ID does not exist"
Nov 02 15:36:36.915229 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:36:36.915194    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-28f621-openshift-default-7dd8c5c789-jvmsw"]
Nov 02 15:36:36.927290 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:36:36.927245    5304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-28f621-openshift-default-7dd8c5c789-jvmsw"]
Nov 02 15:36:38.118570 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:36:38.118537    5304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e4bb2a0e-a2da-44e1-ac19-ee296cfff71c" path="/var/lib/kubelet/pods/e4bb2a0e-a2da-44e1-ac19-ee296cfff71c/volumes"
Nov 02 15:36:54.799440 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:36:54.799400    5304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k6fjk" containerName="extractor"
Nov 02 15:37:04.498171 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:37:04.498114    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-2f8b7c-openshift-default-769cb98f9-xm9dm"]
Nov 02 15:37:04.498640 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:37:04.498374    5304 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-2f8b7c-openshift-default-769cb98f9-xm9dm" podUID="6f8cf102-3f23-49fe-804f-694c50509462" containerName="istio-proxy" containerID="cri-o://8471ba32241bf0c43c3d85ff57b9f5078b985aa12a1d1d918fde5941cf321089" gracePeriod=30
Nov 02 15:37:08.658796 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:37:08.658761    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-cfc63a-6cc5b9494c-scfrq"]
Nov 02 15:37:08.659303 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:37:08.658980    5304 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cfc63a-6cc5b9494c-scfrq" podUID="36c430b0-d26d-46cb-8b8d-f2f2a89aedbd" containerName="maas-api" containerID="cri-o://4d3c47b359719df31679348e0fc7abed238a6562ef2a0f9f8ce25e242c60f161" gracePeriod=30
Nov 02 15:37:08.927854 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:37:08.927779    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cfc63a-6cc5b9494c-scfrq"
Nov 02 15:37:09.057568 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:37:09.057532    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/36c430b0-d26d-46cb-8b8d-f2f2a89aedbd-kube-api-access-j8bp4\" (UniqueName: \"kubernetes.io/projected/36c430b0-d26d-46cb-8b8d-f2f2a89aedbd-kube-api-access-j8bp4\") pod \"36c430b0-d26d-46cb-8b8d-f2f2a89aedbd\" (UID: \"36c430b0-d26d-46cb-8b8d-f2f2a89aedbd\") "
Nov 02 15:37:09.057786 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:37:09.057639    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/36c430b0-d26d-46cb-8b8d-f2f2a89aedbd-metrics-certs\" (UniqueName: \"kubernetes.io/secret/36c430b0-d26d-46cb-8b8d-f2f2a89aedbd-metrics-certs\") pod \"36c430b0-d26d-46cb-8b8d-f2f2a89aedbd\" (UID: \"36c430b0-d26d-46cb-8b8d-f2f2a89aedbd\") "
Nov 02 15:37:09.057786 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:37:09.057704    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/36c430b0-d26d-46cb-8b8d-f2f2a89aedbd-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/36c430b0-d26d-46cb-8b8d-f2f2a89aedbd-openshift-service-ca\") pod \"36c430b0-d26d-46cb-8b8d-f2f2a89aedbd\" (UID: \"36c430b0-d26d-46cb-8b8d-f2f2a89aedbd\") "
Nov 02 15:37:09.057786 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:37:09.057784    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/36c430b0-d26d-46cb-8b8d-f2f2a89aedbd-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/36c430b0-d26d-46cb-8b8d-f2f2a89aedbd-maas-api-tls\") pod \"36c430b0-d26d-46cb-8b8d-f2f2a89aedbd\" (UID: \"36c430b0-d26d-46cb-8b8d-f2f2a89aedbd\") "
Nov 02 15:37:09.058217 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:37:09.058175    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/36c430b0-d26d-46cb-8b8d-f2f2a89aedbd-openshift-service-ca" pod "36c430b0-d26d-46cb-8b8d-f2f2a89aedbd" (UID: "36c430b0-d26d-46cb-8b8d-f2f2a89aedbd"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:37:09.058485 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:37:09.058462    5304 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/36c430b0-d26d-46cb-8b8d-f2f2a89aedbd-openshift-service-ca\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:37:09.060061 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:37:09.060035    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/36c430b0-d26d-46cb-8b8d-f2f2a89aedbd-metrics-certs" pod "36c430b0-d26d-46cb-8b8d-f2f2a89aedbd" (UID: "36c430b0-d26d-46cb-8b8d-f2f2a89aedbd"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:37:09.060435 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:37:09.060398    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/36c430b0-d26d-46cb-8b8d-f2f2a89aedbd-maas-api-tls" pod "36c430b0-d26d-46cb-8b8d-f2f2a89aedbd" (UID: "36c430b0-d26d-46cb-8b8d-f2f2a89aedbd"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:37:09.060435 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:37:09.060418    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36c430b0-d26d-46cb-8b8d-f2f2a89aedbd-kube-api-access-j8bp4" pod "36c430b0-d26d-46cb-8b8d-f2f2a89aedbd" (UID: "36c430b0-d26d-46cb-8b8d-f2f2a89aedbd"). InnerVolumeSpecName "kube-api-access-j8bp4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:37:09.065028 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:37:09.065002    5304 generic.go:356] "Generic (PLEG): container finished" podID="36c430b0-d26d-46cb-8b8d-f2f2a89aedbd" containerID="4d3c47b359719df31679348e0fc7abed238a6562ef2a0f9f8ce25e242c60f161" exitCode=0
Nov 02 15:37:09.065174 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:37:09.065048    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cfc63a-6cc5b9494c-scfrq"
Nov 02 15:37:09.065174 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:37:09.065085    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cfc63a-6cc5b9494c-scfrq" event={"ID":"36c430b0-d26d-46cb-8b8d-f2f2a89aedbd","Type":"ContainerDied","Data":"4d3c47b359719df31679348e0fc7abed238a6562ef2a0f9f8ce25e242c60f161"}
Nov 02 15:37:09.065301 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:37:09.065216    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cfc63a-6cc5b9494c-scfrq" event={"ID":"36c430b0-d26d-46cb-8b8d-f2f2a89aedbd","Type":"ContainerDied","Data":"dfac31c453482b16200fed5103f78d274ed0413a093a7f3492f98d4716bd4972"}
Nov 02 15:37:09.065301 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:37:09.065247    5304 scope.go:122] "RemoveContainer" containerID="4d3c47b359719df31679348e0fc7abed238a6562ef2a0f9f8ce25e242c60f161"
Nov 02 15:37:09.087960 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:37:09.087940    5304 scope.go:122] "RemoveContainer" containerID="4d3c47b359719df31679348e0fc7abed238a6562ef2a0f9f8ce25e242c60f161"
Nov 02 15:37:09.088317 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:37:09.088292    5304 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d3c47b359719df31679348e0fc7abed238a6562ef2a0f9f8ce25e242c60f161\": container with ID starting with 4d3c47b359719df31679348e0fc7abed238a6562ef2a0f9f8ce25e242c60f161 not found: ID does not exist" containerID="4d3c47b359719df31679348e0fc7abed238a6562ef2a0f9f8ce25e242c60f161"
Nov 02 15:37:09.088389 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:37:09.088327    5304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d3c47b359719df31679348e0fc7abed238a6562ef2a0f9f8ce25e242c60f161"} err="failed to get container status \"4d3c47b359719df31679348e0fc7abed238a6562ef2a0f9f8ce25e242c60f161\": rpc error: code = NotFound desc = could not find container \"4d3c47b359719df31679348e0fc7abed238a6562ef2a0f9f8ce25e242c60f161\": container with ID starting with 4d3c47b359719df31679348e0fc7abed238a6562ef2a0f9f8ce25e242c60f161 not found: ID does not exist"
Nov 02 15:37:09.132149 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:37:09.131002    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-cfc63a-6cc5b9494c-scfrq"]
Nov 02 15:37:09.132149 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:37:09.131062    5304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-cfc63a-6cc5b9494c-scfrq"]
Nov 02 15:37:09.160734 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:37:09.160687    5304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j8bp4\" (UniqueName: \"kubernetes.io/projected/36c430b0-d26d-46cb-8b8d-f2f2a89aedbd-kube-api-access-j8bp4\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:37:09.160734 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:37:09.160730    5304 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/36c430b0-d26d-46cb-8b8d-f2f2a89aedbd-metrics-certs\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:37:09.160992 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:37:09.160747    5304 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/36c430b0-d26d-46cb-8b8d-f2f2a89aedbd-maas-api-tls\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:37:09.760886 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:37:09.760856    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-2f8b7c-openshift-default-769cb98f9-xm9dm"
Nov 02 15:37:09.869674 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:37:09.869644    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f8cf102-3f23-49fe-804f-694c50509462-kube-api-access-s4xr8\" (UniqueName: \"kubernetes.io/projected/6f8cf102-3f23-49fe-804f-694c50509462-kube-api-access-s4xr8\") pod \"6f8cf102-3f23-49fe-804f-694c50509462\" (UID: \"6f8cf102-3f23-49fe-804f-694c50509462\") "
Nov 02 15:37:09.869965 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:37:09.869684    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f8cf102-3f23-49fe-804f-694c50509462-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f8cf102-3f23-49fe-804f-694c50509462-workload-socket\") pod \"6f8cf102-3f23-49fe-804f-694c50509462\" (UID: \"6f8cf102-3f23-49fe-804f-694c50509462\") "
Nov 02 15:37:09.869965 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:37:09.869726    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f8cf102-3f23-49fe-804f-694c50509462-istio-token\" (UniqueName: \"kubernetes.io/projected/6f8cf102-3f23-49fe-804f-694c50509462-istio-token\") pod \"6f8cf102-3f23-49fe-804f-694c50509462\" (UID: \"6f8cf102-3f23-49fe-804f-694c50509462\") "
Nov 02 15:37:09.869965 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:37:09.869751    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6f8cf102-3f23-49fe-804f-694c50509462-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f8cf102-3f23-49fe-804f-694c50509462-istio-podinfo\") pod \"6f8cf102-3f23-49fe-804f-694c50509462\" (UID: \"6f8cf102-3f23-49fe-804f-694c50509462\") "
Nov 02 15:37:09.869965 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:37:09.869777    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f8cf102-3f23-49fe-804f-694c50509462-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f8cf102-3f23-49fe-804f-694c50509462-workload-certs\") pod \"6f8cf102-3f23-49fe-804f-694c50509462\" (UID: \"6f8cf102-3f23-49fe-804f-694c50509462\") "
Nov 02 15:37:09.869965 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:37:09.869849    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f8cf102-3f23-49fe-804f-694c50509462-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f8cf102-3f23-49fe-804f-694c50509462-istio-envoy\") pod \"6f8cf102-3f23-49fe-804f-694c50509462\" (UID: \"6f8cf102-3f23-49fe-804f-694c50509462\") "
Nov 02 15:37:09.869965 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:37:09.869931    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f8cf102-3f23-49fe-804f-694c50509462-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f8cf102-3f23-49fe-804f-694c50509462-istio-data\") pod \"6f8cf102-3f23-49fe-804f-694c50509462\" (UID: \"6f8cf102-3f23-49fe-804f-694c50509462\") "
Nov 02 15:37:09.870520 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:37:09.870014    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6f8cf102-3f23-49fe-804f-694c50509462-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f8cf102-3f23-49fe-804f-694c50509462-istiod-ca-cert\") pod \"6f8cf102-3f23-49fe-804f-694c50509462\" (UID: \"6f8cf102-3f23-49fe-804f-694c50509462\") "
Nov 02 15:37:09.870520 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:37:09.870070    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f8cf102-3f23-49fe-804f-694c50509462-workload-socket" pod "6f8cf102-3f23-49fe-804f-694c50509462" (UID: "6f8cf102-3f23-49fe-804f-694c50509462"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:37:09.870520 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:37:09.870159    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f8cf102-3f23-49fe-804f-694c50509462-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f8cf102-3f23-49fe-804f-694c50509462-credential-socket\") pod \"6f8cf102-3f23-49fe-804f-694c50509462\" (UID: \"6f8cf102-3f23-49fe-804f-694c50509462\") "
Nov 02 15:37:09.870520 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:37:09.870184    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f8cf102-3f23-49fe-804f-694c50509462-workload-certs" pod "6f8cf102-3f23-49fe-804f-694c50509462" (UID: "6f8cf102-3f23-49fe-804f-694c50509462"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:37:09.870520 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:37:09.870335    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f8cf102-3f23-49fe-804f-694c50509462-istio-data" pod "6f8cf102-3f23-49fe-804f-694c50509462" (UID: "6f8cf102-3f23-49fe-804f-694c50509462"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:37:09.870520 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:37:09.870467    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f8cf102-3f23-49fe-804f-694c50509462-istiod-ca-cert" pod "6f8cf102-3f23-49fe-804f-694c50509462" (UID: "6f8cf102-3f23-49fe-804f-694c50509462"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:37:09.870807 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:37:09.870472    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f8cf102-3f23-49fe-804f-694c50509462-credential-socket" pod "6f8cf102-3f23-49fe-804f-694c50509462" (UID: "6f8cf102-3f23-49fe-804f-694c50509462"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:37:09.870854 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:37:09.870834    5304 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f8cf102-3f23-49fe-804f-694c50509462-credential-socket\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:37:09.870912 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:37:09.870856    5304 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f8cf102-3f23-49fe-804f-694c50509462-workload-socket\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:37:09.870912 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:37:09.870874    5304 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f8cf102-3f23-49fe-804f-694c50509462-workload-certs\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:37:09.870912 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:37:09.870889    5304 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f8cf102-3f23-49fe-804f-694c50509462-istio-data\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:37:09.870912 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:37:09.870902    5304 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f8cf102-3f23-49fe-804f-694c50509462-istiod-ca-cert\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:37:09.872085 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:37:09.872057    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f8cf102-3f23-49fe-804f-694c50509462-kube-api-access-s4xr8" pod "6f8cf102-3f23-49fe-804f-694c50509462" (UID: "6f8cf102-3f23-49fe-804f-694c50509462"). InnerVolumeSpecName "kube-api-access-s4xr8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:37:09.872351 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:37:09.872330    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f8cf102-3f23-49fe-804f-694c50509462-istio-envoy" pod "6f8cf102-3f23-49fe-804f-694c50509462" (UID: "6f8cf102-3f23-49fe-804f-694c50509462"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:37:09.872528 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:37:09.872503    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6f8cf102-3f23-49fe-804f-694c50509462-istio-podinfo" pod "6f8cf102-3f23-49fe-804f-694c50509462" (UID: "6f8cf102-3f23-49fe-804f-694c50509462"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 15:37:09.872635 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:37:09.872506    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f8cf102-3f23-49fe-804f-694c50509462-istio-token" pod "6f8cf102-3f23-49fe-804f-694c50509462" (UID: "6f8cf102-3f23-49fe-804f-694c50509462"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:37:09.972207 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:37:09.972109    5304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s4xr8\" (UniqueName: \"kubernetes.io/projected/6f8cf102-3f23-49fe-804f-694c50509462-kube-api-access-s4xr8\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:37:09.972207 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:37:09.972202    5304 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f8cf102-3f23-49fe-804f-694c50509462-istio-token\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:37:09.972207 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:37:09.972214    5304 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f8cf102-3f23-49fe-804f-694c50509462-istio-podinfo\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:37:09.972463 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:37:09.972224    5304 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f8cf102-3f23-49fe-804f-694c50509462-istio-envoy\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:37:10.074059 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:37:10.074026    5304 generic.go:356] "Generic (PLEG): container finished" podID="6f8cf102-3f23-49fe-804f-694c50509462" containerID="8471ba32241bf0c43c3d85ff57b9f5078b985aa12a1d1d918fde5941cf321089" exitCode=0
Nov 02 15:37:10.074247 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:37:10.074075    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-2f8b7c-openshift-default-769cb98f9-xm9dm"
Nov 02 15:37:10.074247 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:37:10.074107    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-2f8b7c-openshift-default-769cb98f9-xm9dm" event={"ID":"6f8cf102-3f23-49fe-804f-694c50509462","Type":"ContainerDied","Data":"8471ba32241bf0c43c3d85ff57b9f5078b985aa12a1d1d918fde5941cf321089"}
Nov 02 15:37:10.074247 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:37:10.074169    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-2f8b7c-openshift-default-769cb98f9-xm9dm" event={"ID":"6f8cf102-3f23-49fe-804f-694c50509462","Type":"ContainerDied","Data":"111e895eb5546fb9fc2557c767ac5a6399f9bb0f38c5bb7254e4e575725d44de"}
Nov 02 15:37:10.074247 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:37:10.074198    5304 scope.go:122] "RemoveContainer" containerID="8471ba32241bf0c43c3d85ff57b9f5078b985aa12a1d1d918fde5941cf321089"
Nov 02 15:37:10.084722 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:37:10.084702    5304 scope.go:122] "RemoveContainer" containerID="21a548dc52c0aa6d983371c2a530a668823ee88279a6b33b6df874dfb1cedcfc"
Nov 02 15:37:10.093974 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:37:10.093949    5304 scope.go:122] "RemoveContainer" containerID="8471ba32241bf0c43c3d85ff57b9f5078b985aa12a1d1d918fde5941cf321089"
Nov 02 15:37:10.094368 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:37:10.094343    5304 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8471ba32241bf0c43c3d85ff57b9f5078b985aa12a1d1d918fde5941cf321089\": container with ID starting with 8471ba32241bf0c43c3d85ff57b9f5078b985aa12a1d1d918fde5941cf321089 not found: ID does not exist" containerID="8471ba32241bf0c43c3d85ff57b9f5078b985aa12a1d1d918fde5941cf321089"
Nov 02 15:37:10.094470 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:37:10.094381    5304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8471ba32241bf0c43c3d85ff57b9f5078b985aa12a1d1d918fde5941cf321089"} err="failed to get container status \"8471ba32241bf0c43c3d85ff57b9f5078b985aa12a1d1d918fde5941cf321089\": rpc error: code = NotFound desc = could not find container \"8471ba32241bf0c43c3d85ff57b9f5078b985aa12a1d1d918fde5941cf321089\": container with ID starting with 8471ba32241bf0c43c3d85ff57b9f5078b985aa12a1d1d918fde5941cf321089 not found: ID does not exist"
Nov 02 15:37:10.094470 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:37:10.094406    5304 scope.go:122] "RemoveContainer" containerID="21a548dc52c0aa6d983371c2a530a668823ee88279a6b33b6df874dfb1cedcfc"
Nov 02 15:37:10.094721 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:37:10.094701    5304 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"21a548dc52c0aa6d983371c2a530a668823ee88279a6b33b6df874dfb1cedcfc\": container with ID starting with 21a548dc52c0aa6d983371c2a530a668823ee88279a6b33b6df874dfb1cedcfc not found: ID does not exist" containerID="21a548dc52c0aa6d983371c2a530a668823ee88279a6b33b6df874dfb1cedcfc"
Nov 02 15:37:10.094768 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:37:10.094727    5304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21a548dc52c0aa6d983371c2a530a668823ee88279a6b33b6df874dfb1cedcfc"} err="failed to get container status \"21a548dc52c0aa6d983371c2a530a668823ee88279a6b33b6df874dfb1cedcfc\": rpc error: code = NotFound desc = could not find container \"21a548dc52c0aa6d983371c2a530a668823ee88279a6b33b6df874dfb1cedcfc\": container with ID starting with 21a548dc52c0aa6d983371c2a530a668823ee88279a6b33b6df874dfb1cedcfc not found: ID does not exist"
Nov 02 15:37:10.118617 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:37:10.118582    5304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="36c430b0-d26d-46cb-8b8d-f2f2a89aedbd" path="/var/lib/kubelet/pods/36c430b0-d26d-46cb-8b8d-f2f2a89aedbd/volumes"
Nov 02 15:37:10.131380 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:37:10.131308    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-2f8b7c-openshift-default-769cb98f9-xm9dm"]
Nov 02 15:37:10.142878 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:37:10.142849    5304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-2f8b7c-openshift-default-769cb98f9-xm9dm"]
Nov 02 15:37:12.118943 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:37:12.118906    5304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6f8cf102-3f23-49fe-804f-694c50509462" path="/var/lib/kubelet/pods/6f8cf102-3f23-49fe-804f-694c50509462/volumes"
Nov 02 15:37:15.635237 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:37:15.635206    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-d35a86-75458f47fc-hd67p"]
Nov 02 15:37:19.757006 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:37:19.756969    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-80ec03-openshift-default-66dfdbcf85-b4ltb"]
Nov 02 15:37:19.757542 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:37:19.757278    5304 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-80ec03-openshift-default-66dfdbcf85-b4ltb" podUID="1f9d68a5-7f9c-489b-9289-2f85b7889c2c" containerName="istio-proxy" containerID="cri-o://a2e6e9aa05bc75198000808de8a0a4e21fef1cc7311a0b5ccebfcd1ef4518d52" gracePeriod=30
Nov 02 15:37:20.636081 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:37:20.636023    5304 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-d35a86-75458f47fc-hd67p" podUID="5b81a985-2325-4b54-80a7-8dc2034e5596" containerName="payload-processing" containerID="cri-o://c12b7ea1b85d12e31c42bef5c27266eeceda95d0a931309285a4b0f3bb9e10fc" gracePeriod=25
Nov 02 15:37:21.283309 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:37:21.283271    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-d35a86-75458f47fc-hd67p" podUID="5b81a985-2325-4b54-80a7-8dc2034e5596" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 15:37:24.799253 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:37:24.799218    5304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k6fjk" containerName="extractor"
Nov 02 15:37:25.029216 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:37:25.029153    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-80ec03-openshift-default-66dfdbcf85-b4ltb"
Nov 02 15:37:25.078812 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:37:25.078781    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f9d68a5-7f9c-489b-9289-2f85b7889c2c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f9d68a5-7f9c-489b-9289-2f85b7889c2c-credential-socket\") pod \"1f9d68a5-7f9c-489b-9289-2f85b7889c2c\" (UID: \"1f9d68a5-7f9c-489b-9289-2f85b7889c2c\") "
Nov 02 15:37:25.079075 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:37:25.078868    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f9d68a5-7f9c-489b-9289-2f85b7889c2c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f9d68a5-7f9c-489b-9289-2f85b7889c2c-istio-envoy\") pod \"1f9d68a5-7f9c-489b-9289-2f85b7889c2c\" (UID: \"1f9d68a5-7f9c-489b-9289-2f85b7889c2c\") "
Nov 02 15:37:25.079075 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:37:25.078905    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f9d68a5-7f9c-489b-9289-2f85b7889c2c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f9d68a5-7f9c-489b-9289-2f85b7889c2c-workload-socket\") pod \"1f9d68a5-7f9c-489b-9289-2f85b7889c2c\" (UID: \"1f9d68a5-7f9c-489b-9289-2f85b7889c2c\") "
Nov 02 15:37:25.079075 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:37:25.078963    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f9d68a5-7f9c-489b-9289-2f85b7889c2c-kube-api-access-tt84c\" (UniqueName: \"kubernetes.io/projected/1f9d68a5-7f9c-489b-9289-2f85b7889c2c-kube-api-access-tt84c\") pod \"1f9d68a5-7f9c-489b-9289-2f85b7889c2c\" (UID: \"1f9d68a5-7f9c-489b-9289-2f85b7889c2c\") "
Nov 02 15:37:25.079075 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:37:25.078990    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f9d68a5-7f9c-489b-9289-2f85b7889c2c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f9d68a5-7f9c-489b-9289-2f85b7889c2c-workload-certs\") pod \"1f9d68a5-7f9c-489b-9289-2f85b7889c2c\" (UID: \"1f9d68a5-7f9c-489b-9289-2f85b7889c2c\") "
Nov 02 15:37:25.079075 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:37:25.079021    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1f9d68a5-7f9c-489b-9289-2f85b7889c2c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f9d68a5-7f9c-489b-9289-2f85b7889c2c-istiod-ca-cert\") pod \"1f9d68a5-7f9c-489b-9289-2f85b7889c2c\" (UID: \"1f9d68a5-7f9c-489b-9289-2f85b7889c2c\") "
Nov 02 15:37:25.079075 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:37:25.079056    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f9d68a5-7f9c-489b-9289-2f85b7889c2c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f9d68a5-7f9c-489b-9289-2f85b7889c2c-istio-data\") pod \"1f9d68a5-7f9c-489b-9289-2f85b7889c2c\" (UID: \"1f9d68a5-7f9c-489b-9289-2f85b7889c2c\") "
Nov 02 15:37:25.079467 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:37:25.079121    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f9d68a5-7f9c-489b-9289-2f85b7889c2c-credential-socket" pod "1f9d68a5-7f9c-489b-9289-2f85b7889c2c" (UID: "1f9d68a5-7f9c-489b-9289-2f85b7889c2c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:37:25.079467 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:37:25.079173    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f9d68a5-7f9c-489b-9289-2f85b7889c2c-istio-token\" (UniqueName: \"kubernetes.io/projected/1f9d68a5-7f9c-489b-9289-2f85b7889c2c-istio-token\") pod \"1f9d68a5-7f9c-489b-9289-2f85b7889c2c\" (UID: \"1f9d68a5-7f9c-489b-9289-2f85b7889c2c\") "
Nov 02 15:37:25.079467 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:37:25.079209    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1f9d68a5-7f9c-489b-9289-2f85b7889c2c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f9d68a5-7f9c-489b-9289-2f85b7889c2c-istio-podinfo\") pod \"1f9d68a5-7f9c-489b-9289-2f85b7889c2c\" (UID: \"1f9d68a5-7f9c-489b-9289-2f85b7889c2c\") "
Nov 02 15:37:25.079467 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:37:25.079260    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f9d68a5-7f9c-489b-9289-2f85b7889c2c-workload-socket" pod "1f9d68a5-7f9c-489b-9289-2f85b7889c2c" (UID: "1f9d68a5-7f9c-489b-9289-2f85b7889c2c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:37:25.079467 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:37:25.079459    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f9d68a5-7f9c-489b-9289-2f85b7889c2c-istio-data" pod "1f9d68a5-7f9c-489b-9289-2f85b7889c2c" (UID: "1f9d68a5-7f9c-489b-9289-2f85b7889c2c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:37:25.079712 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:37:25.079493    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f9d68a5-7f9c-489b-9289-2f85b7889c2c-istiod-ca-cert" pod "1f9d68a5-7f9c-489b-9289-2f85b7889c2c" (UID: "1f9d68a5-7f9c-489b-9289-2f85b7889c2c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:37:25.079712 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:37:25.079651    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f9d68a5-7f9c-489b-9289-2f85b7889c2c-workload-certs" pod "1f9d68a5-7f9c-489b-9289-2f85b7889c2c" (UID: "1f9d68a5-7f9c-489b-9289-2f85b7889c2c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:37:25.079941 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:37:25.079921    5304 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f9d68a5-7f9c-489b-9289-2f85b7889c2c-workload-socket\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:37:25.080001 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:37:25.079948    5304 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f9d68a5-7f9c-489b-9289-2f85b7889c2c-workload-certs\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:37:25.080001 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:37:25.079964    5304 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f9d68a5-7f9c-489b-9289-2f85b7889c2c-istiod-ca-cert\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:37:25.080001 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:37:25.079977    5304 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f9d68a5-7f9c-489b-9289-2f85b7889c2c-istio-data\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:37:25.080001 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:37:25.079990    5304 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f9d68a5-7f9c-489b-9289-2f85b7889c2c-credential-socket\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:37:25.081412 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:37:25.081390    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f9d68a5-7f9c-489b-9289-2f85b7889c2c-kube-api-access-tt84c" pod "1f9d68a5-7f9c-489b-9289-2f85b7889c2c" (UID: "1f9d68a5-7f9c-489b-9289-2f85b7889c2c"). InnerVolumeSpecName "kube-api-access-tt84c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:37:25.081489 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:37:25.081436    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1f9d68a5-7f9c-489b-9289-2f85b7889c2c-istio-podinfo" pod "1f9d68a5-7f9c-489b-9289-2f85b7889c2c" (UID: "1f9d68a5-7f9c-489b-9289-2f85b7889c2c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 15:37:25.081699 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:37:25.081670    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f9d68a5-7f9c-489b-9289-2f85b7889c2c-istio-token" pod "1f9d68a5-7f9c-489b-9289-2f85b7889c2c" (UID: "1f9d68a5-7f9c-489b-9289-2f85b7889c2c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:37:25.081699 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:37:25.081681    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f9d68a5-7f9c-489b-9289-2f85b7889c2c-istio-envoy" pod "1f9d68a5-7f9c-489b-9289-2f85b7889c2c" (UID: "1f9d68a5-7f9c-489b-9289-2f85b7889c2c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:37:25.164043 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:37:25.164008    5304 generic.go:356] "Generic (PLEG): container finished" podID="1f9d68a5-7f9c-489b-9289-2f85b7889c2c" containerID="a2e6e9aa05bc75198000808de8a0a4e21fef1cc7311a0b5ccebfcd1ef4518d52" exitCode=0
Nov 02 15:37:25.164221 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:37:25.164063    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-80ec03-openshift-default-66dfdbcf85-b4ltb"
Nov 02 15:37:25.164221 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:37:25.164092    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-80ec03-openshift-default-66dfdbcf85-b4ltb" event={"ID":"1f9d68a5-7f9c-489b-9289-2f85b7889c2c","Type":"ContainerDied","Data":"a2e6e9aa05bc75198000808de8a0a4e21fef1cc7311a0b5ccebfcd1ef4518d52"}
Nov 02 15:37:25.164221 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:37:25.164176    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-80ec03-openshift-default-66dfdbcf85-b4ltb" event={"ID":"1f9d68a5-7f9c-489b-9289-2f85b7889c2c","Type":"ContainerDied","Data":"c2318f9bfe896666b0892b745d6798be109ccd624bea6c987bcc744f05d042e6"}
Nov 02 15:37:25.164221 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:37:25.164191    5304 scope.go:122] "RemoveContainer" containerID="a2e6e9aa05bc75198000808de8a0a4e21fef1cc7311a0b5ccebfcd1ef4518d52"
Nov 02 15:37:25.176005 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:37:25.175980    5304 scope.go:122] "RemoveContainer" containerID="ca4848c3f3277089ca035016514b458f2450114dc6a79e6b0ef5191a3518abc8"
Nov 02 15:37:25.182027 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:37:25.182002    5304 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f9d68a5-7f9c-489b-9289-2f85b7889c2c-istio-podinfo\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:37:25.182152 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:37:25.182032    5304 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f9d68a5-7f9c-489b-9289-2f85b7889c2c-istio-envoy\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:37:25.182152 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:37:25.182050    5304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tt84c\" (UniqueName: \"kubernetes.io/projected/1f9d68a5-7f9c-489b-9289-2f85b7889c2c-kube-api-access-tt84c\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:37:25.182152 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:37:25.182064    5304 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f9d68a5-7f9c-489b-9289-2f85b7889c2c-istio-token\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:37:25.187251 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:37:25.187223    5304 scope.go:122] "RemoveContainer" containerID="a2e6e9aa05bc75198000808de8a0a4e21fef1cc7311a0b5ccebfcd1ef4518d52"
Nov 02 15:37:25.187648 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:37:25.187624    5304 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a2e6e9aa05bc75198000808de8a0a4e21fef1cc7311a0b5ccebfcd1ef4518d52\": container with ID starting with a2e6e9aa05bc75198000808de8a0a4e21fef1cc7311a0b5ccebfcd1ef4518d52 not found: ID does not exist" containerID="a2e6e9aa05bc75198000808de8a0a4e21fef1cc7311a0b5ccebfcd1ef4518d52"
Nov 02 15:37:25.187752 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:37:25.187660    5304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a2e6e9aa05bc75198000808de8a0a4e21fef1cc7311a0b5ccebfcd1ef4518d52"} err="failed to get container status \"a2e6e9aa05bc75198000808de8a0a4e21fef1cc7311a0b5ccebfcd1ef4518d52\": rpc error: code = NotFound desc = could not find container \"a2e6e9aa05bc75198000808de8a0a4e21fef1cc7311a0b5ccebfcd1ef4518d52\": container with ID starting with a2e6e9aa05bc75198000808de8a0a4e21fef1cc7311a0b5ccebfcd1ef4518d52 not found: ID does not exist"
Nov 02 15:37:25.187752 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:37:25.187685    5304 scope.go:122] "RemoveContainer" containerID="ca4848c3f3277089ca035016514b458f2450114dc6a79e6b0ef5191a3518abc8"
Nov 02 15:37:25.188505 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:37:25.188120    5304 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca4848c3f3277089ca035016514b458f2450114dc6a79e6b0ef5191a3518abc8\": container with ID starting with ca4848c3f3277089ca035016514b458f2450114dc6a79e6b0ef5191a3518abc8 not found: ID does not exist" containerID="ca4848c3f3277089ca035016514b458f2450114dc6a79e6b0ef5191a3518abc8"
Nov 02 15:37:25.188505 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:37:25.188177    5304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca4848c3f3277089ca035016514b458f2450114dc6a79e6b0ef5191a3518abc8"} err="failed to get container status \"ca4848c3f3277089ca035016514b458f2450114dc6a79e6b0ef5191a3518abc8\": rpc error: code = NotFound desc = could not find container \"ca4848c3f3277089ca035016514b458f2450114dc6a79e6b0ef5191a3518abc8\": container with ID starting with ca4848c3f3277089ca035016514b458f2450114dc6a79e6b0ef5191a3518abc8 not found: ID does not exist"
Nov 02 15:37:25.227625 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:37:25.227590    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-80ec03-openshift-default-66dfdbcf85-b4ltb"]
Nov 02 15:37:25.230034 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:37:25.230008    5304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-80ec03-openshift-default-66dfdbcf85-b4ltb"]
Nov 02 15:37:26.118852 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:37:26.118815    5304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1f9d68a5-7f9c-489b-9289-2f85b7889c2c" path="/var/lib/kubelet/pods/1f9d68a5-7f9c-489b-9289-2f85b7889c2c/volumes"
Nov 02 15:37:31.282534 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:37:31.282490    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-d35a86-75458f47fc-hd67p" podUID="5b81a985-2325-4b54-80a7-8dc2034e5596" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 15:37:40.915572 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:37:40.915545    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-d35a86-75458f47fc-hd67p"
Nov 02 15:37:41.094277 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:37:41.094244    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5b81a985-2325-4b54-80a7-8dc2034e5596-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b81a985-2325-4b54-80a7-8dc2034e5596-plugins-config-volume\") pod \"5b81a985-2325-4b54-80a7-8dc2034e5596\" (UID: \"5b81a985-2325-4b54-80a7-8dc2034e5596\") "
Nov 02 15:37:41.094457 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:37:41.094310    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b81a985-2325-4b54-80a7-8dc2034e5596-kube-api-access-trkwg\" (UniqueName: \"kubernetes.io/projected/5b81a985-2325-4b54-80a7-8dc2034e5596-kube-api-access-trkwg\") pod \"5b81a985-2325-4b54-80a7-8dc2034e5596\" (UID: \"5b81a985-2325-4b54-80a7-8dc2034e5596\") "
Nov 02 15:37:41.094686 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:37:41.094664    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b81a985-2325-4b54-80a7-8dc2034e5596-plugins-config-volume" pod "5b81a985-2325-4b54-80a7-8dc2034e5596" (UID: "5b81a985-2325-4b54-80a7-8dc2034e5596"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:37:41.094791 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:37:41.094778    5304 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b81a985-2325-4b54-80a7-8dc2034e5596-plugins-config-volume\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:37:41.096414 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:37:41.096395    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b81a985-2325-4b54-80a7-8dc2034e5596-kube-api-access-trkwg" pod "5b81a985-2325-4b54-80a7-8dc2034e5596" (UID: "5b81a985-2325-4b54-80a7-8dc2034e5596"). InnerVolumeSpecName "kube-api-access-trkwg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:37:41.195704 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:37:41.195610    5304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-trkwg\" (UniqueName: \"kubernetes.io/projected/5b81a985-2325-4b54-80a7-8dc2034e5596-kube-api-access-trkwg\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:37:41.269266 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:37:41.269223    5304 generic.go:356] "Generic (PLEG): container finished" podID="5b81a985-2325-4b54-80a7-8dc2034e5596" containerID="c12b7ea1b85d12e31c42bef5c27266eeceda95d0a931309285a4b0f3bb9e10fc" exitCode=0
Nov 02 15:37:41.269433 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:37:41.269269    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-d35a86-75458f47fc-hd67p"
Nov 02 15:37:41.269433 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:37:41.269269    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-d35a86-75458f47fc-hd67p" event={"ID":"5b81a985-2325-4b54-80a7-8dc2034e5596","Type":"ContainerDied","Data":"c12b7ea1b85d12e31c42bef5c27266eeceda95d0a931309285a4b0f3bb9e10fc"}
Nov 02 15:37:41.269433 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:37:41.269394    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-d35a86-75458f47fc-hd67p" event={"ID":"5b81a985-2325-4b54-80a7-8dc2034e5596","Type":"ContainerDied","Data":"4d6373d72b57fa10989ad7bdc6956ced50e8a2deb1bf86b0902d53b2775c624f"}
Nov 02 15:37:41.269433 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:37:41.269422    5304 scope.go:122] "RemoveContainer" containerID="c12b7ea1b85d12e31c42bef5c27266eeceda95d0a931309285a4b0f3bb9e10fc"
Nov 02 15:37:41.279235 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:37:41.279210    5304 scope.go:122] "RemoveContainer" containerID="c12b7ea1b85d12e31c42bef5c27266eeceda95d0a931309285a4b0f3bb9e10fc"
Nov 02 15:37:41.279473 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:37:41.279456    5304 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c12b7ea1b85d12e31c42bef5c27266eeceda95d0a931309285a4b0f3bb9e10fc\": container with ID starting with c12b7ea1b85d12e31c42bef5c27266eeceda95d0a931309285a4b0f3bb9e10fc not found: ID does not exist" containerID="c12b7ea1b85d12e31c42bef5c27266eeceda95d0a931309285a4b0f3bb9e10fc"
Nov 02 15:37:41.279528 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:37:41.279481    5304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c12b7ea1b85d12e31c42bef5c27266eeceda95d0a931309285a4b0f3bb9e10fc"} err="failed to get container status \"c12b7ea1b85d12e31c42bef5c27266eeceda95d0a931309285a4b0f3bb9e10fc\": rpc error: code = NotFound desc = could not find container \"c12b7ea1b85d12e31c42bef5c27266eeceda95d0a931309285a4b0f3bb9e10fc\": container with ID starting with c12b7ea1b85d12e31c42bef5c27266eeceda95d0a931309285a4b0f3bb9e10fc not found: ID does not exist"
Nov 02 15:37:41.320966 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:37:41.320932    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-d35a86-75458f47fc-hd67p"]
Nov 02 15:37:41.326223 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:37:41.326193    5304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-d35a86-75458f47fc-hd67p"]
Nov 02 15:37:42.118730 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:37:42.118687    5304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5b81a985-2325-4b54-80a7-8dc2034e5596" path="/var/lib/kubelet/pods/5b81a985-2325-4b54-80a7-8dc2034e5596/volumes"
Nov 02 15:37:43.377865 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:37:43.377825    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-69ff88-openshift-default-866dfd5ff7-74bk2"]
Nov 02 15:37:43.378282 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:37:43.378115    5304 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-69ff88-openshift-default-866dfd5ff7-74bk2" podUID="301b87ba-1380-4110-96cd-5e2a057cae4d" containerName="istio-proxy" containerID="cri-o://c52095f80a6a806c607449558de0019e7eba5c9efca4153c3fcfc3a77a707a9c" gracePeriod=30
Nov 02 15:37:48.653160 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:37:48.653060    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-69ff88-openshift-default-866dfd5ff7-74bk2"
Nov 02 15:37:48.693504 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:37:48.693472    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/301b87ba-1380-4110-96cd-5e2a057cae4d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/301b87ba-1380-4110-96cd-5e2a057cae4d-workload-certs\") pod \"301b87ba-1380-4110-96cd-5e2a057cae4d\" (UID: \"301b87ba-1380-4110-96cd-5e2a057cae4d\") "
Nov 02 15:37:48.693713 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:37:48.693515    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/301b87ba-1380-4110-96cd-5e2a057cae4d-istio-token\" (UniqueName: \"kubernetes.io/projected/301b87ba-1380-4110-96cd-5e2a057cae4d-istio-token\") pod \"301b87ba-1380-4110-96cd-5e2a057cae4d\" (UID: \"301b87ba-1380-4110-96cd-5e2a057cae4d\") "
Nov 02 15:37:48.693713 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:37:48.693544    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/301b87ba-1380-4110-96cd-5e2a057cae4d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/301b87ba-1380-4110-96cd-5e2a057cae4d-credential-socket\") pod \"301b87ba-1380-4110-96cd-5e2a057cae4d\" (UID: \"301b87ba-1380-4110-96cd-5e2a057cae4d\") "
Nov 02 15:37:48.693713 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:37:48.693579    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/301b87ba-1380-4110-96cd-5e2a057cae4d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/301b87ba-1380-4110-96cd-5e2a057cae4d-istiod-ca-cert\") pod \"301b87ba-1380-4110-96cd-5e2a057cae4d\" (UID: \"301b87ba-1380-4110-96cd-5e2a057cae4d\") "
Nov 02 15:37:48.693713 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:37:48.693634    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/301b87ba-1380-4110-96cd-5e2a057cae4d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/301b87ba-1380-4110-96cd-5e2a057cae4d-workload-socket\") pod \"301b87ba-1380-4110-96cd-5e2a057cae4d\" (UID: \"301b87ba-1380-4110-96cd-5e2a057cae4d\") "
Nov 02 15:37:48.693968 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:37:48.693741    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/301b87ba-1380-4110-96cd-5e2a057cae4d-kube-api-access-9fbwt\" (UniqueName: \"kubernetes.io/projected/301b87ba-1380-4110-96cd-5e2a057cae4d-kube-api-access-9fbwt\") pod \"301b87ba-1380-4110-96cd-5e2a057cae4d\" (UID: \"301b87ba-1380-4110-96cd-5e2a057cae4d\") "
Nov 02 15:37:48.693968 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:37:48.693769    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/301b87ba-1380-4110-96cd-5e2a057cae4d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/301b87ba-1380-4110-96cd-5e2a057cae4d-istio-podinfo\") pod \"301b87ba-1380-4110-96cd-5e2a057cae4d\" (UID: \"301b87ba-1380-4110-96cd-5e2a057cae4d\") "
Nov 02 15:37:48.693968 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:37:48.693798    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/301b87ba-1380-4110-96cd-5e2a057cae4d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/301b87ba-1380-4110-96cd-5e2a057cae4d-istio-data\") pod \"301b87ba-1380-4110-96cd-5e2a057cae4d\" (UID: \"301b87ba-1380-4110-96cd-5e2a057cae4d\") "
Nov 02 15:37:48.693968 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:37:48.693831    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/301b87ba-1380-4110-96cd-5e2a057cae4d-workload-certs" pod "301b87ba-1380-4110-96cd-5e2a057cae4d" (UID: "301b87ba-1380-4110-96cd-5e2a057cae4d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:37:48.693968 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:37:48.693894    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/301b87ba-1380-4110-96cd-5e2a057cae4d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/301b87ba-1380-4110-96cd-5e2a057cae4d-istio-envoy\") pod \"301b87ba-1380-4110-96cd-5e2a057cae4d\" (UID: \"301b87ba-1380-4110-96cd-5e2a057cae4d\") "
Nov 02 15:37:48.693968 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:37:48.693906    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/301b87ba-1380-4110-96cd-5e2a057cae4d-credential-socket" pod "301b87ba-1380-4110-96cd-5e2a057cae4d" (UID: "301b87ba-1380-4110-96cd-5e2a057cae4d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:37:48.693968 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:37:48.693957    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/301b87ba-1380-4110-96cd-5e2a057cae4d-workload-socket" pod "301b87ba-1380-4110-96cd-5e2a057cae4d" (UID: "301b87ba-1380-4110-96cd-5e2a057cae4d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:37:48.694392 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:37:48.694169    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/301b87ba-1380-4110-96cd-5e2a057cae4d-istiod-ca-cert" pod "301b87ba-1380-4110-96cd-5e2a057cae4d" (UID: "301b87ba-1380-4110-96cd-5e2a057cae4d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:37:48.694474 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:37:48.694452    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/301b87ba-1380-4110-96cd-5e2a057cae4d-istio-data" pod "301b87ba-1380-4110-96cd-5e2a057cae4d" (UID: "301b87ba-1380-4110-96cd-5e2a057cae4d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:37:48.694668 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:37:48.694652    5304 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/301b87ba-1380-4110-96cd-5e2a057cae4d-workload-certs\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:37:48.694743 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:37:48.694674    5304 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/301b87ba-1380-4110-96cd-5e2a057cae4d-credential-socket\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:37:48.694743 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:37:48.694688    5304 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/301b87ba-1380-4110-96cd-5e2a057cae4d-istiod-ca-cert\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:37:48.694743 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:37:48.694700    5304 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/301b87ba-1380-4110-96cd-5e2a057cae4d-workload-socket\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:37:48.694743 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:37:48.694712    5304 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/301b87ba-1380-4110-96cd-5e2a057cae4d-istio-data\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:37:48.696326 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:37:48.696296    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/301b87ba-1380-4110-96cd-5e2a057cae4d-kube-api-access-9fbwt" pod "301b87ba-1380-4110-96cd-5e2a057cae4d" (UID: "301b87ba-1380-4110-96cd-5e2a057cae4d"). InnerVolumeSpecName "kube-api-access-9fbwt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:37:48.696431 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:37:48.696323    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/301b87ba-1380-4110-96cd-5e2a057cae4d-istio-podinfo" pod "301b87ba-1380-4110-96cd-5e2a057cae4d" (UID: "301b87ba-1380-4110-96cd-5e2a057cae4d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 15:37:48.696431 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:37:48.696379    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/301b87ba-1380-4110-96cd-5e2a057cae4d-istio-token" pod "301b87ba-1380-4110-96cd-5e2a057cae4d" (UID: "301b87ba-1380-4110-96cd-5e2a057cae4d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:37:48.696431 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:37:48.696383    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/301b87ba-1380-4110-96cd-5e2a057cae4d-istio-envoy" pod "301b87ba-1380-4110-96cd-5e2a057cae4d" (UID: "301b87ba-1380-4110-96cd-5e2a057cae4d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:37:48.796228 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:37:48.796188    5304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9fbwt\" (UniqueName: \"kubernetes.io/projected/301b87ba-1380-4110-96cd-5e2a057cae4d-kube-api-access-9fbwt\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:37:48.796228 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:37:48.796220    5304 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/301b87ba-1380-4110-96cd-5e2a057cae4d-istio-podinfo\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:37:48.796228 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:37:48.796233    5304 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/301b87ba-1380-4110-96cd-5e2a057cae4d-istio-envoy\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:37:48.796479 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:37:48.796244    5304 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/301b87ba-1380-4110-96cd-5e2a057cae4d-istio-token\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:37:49.319849 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:37:49.319815    5304 generic.go:356] "Generic (PLEG): container finished" podID="301b87ba-1380-4110-96cd-5e2a057cae4d" containerID="c52095f80a6a806c607449558de0019e7eba5c9efca4153c3fcfc3a77a707a9c" exitCode=0
Nov 02 15:37:49.320101 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:37:49.319863    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-69ff88-openshift-default-866dfd5ff7-74bk2"
Nov 02 15:37:49.320101 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:37:49.319897    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-69ff88-openshift-default-866dfd5ff7-74bk2" event={"ID":"301b87ba-1380-4110-96cd-5e2a057cae4d","Type":"ContainerDied","Data":"c52095f80a6a806c607449558de0019e7eba5c9efca4153c3fcfc3a77a707a9c"}
Nov 02 15:37:49.320101 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:37:49.319951    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-69ff88-openshift-default-866dfd5ff7-74bk2" event={"ID":"301b87ba-1380-4110-96cd-5e2a057cae4d","Type":"ContainerDied","Data":"c2c6f4d0f21c59b911436725df9698c09933bbf38e942cbf595faa9cbf988e35"}
Nov 02 15:37:49.320101 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:37:49.319980    5304 scope.go:122] "RemoveContainer" containerID="c52095f80a6a806c607449558de0019e7eba5c9efca4153c3fcfc3a77a707a9c"
Nov 02 15:37:49.330988 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:37:49.330972    5304 scope.go:122] "RemoveContainer" containerID="c52095f80a6a806c607449558de0019e7eba5c9efca4153c3fcfc3a77a707a9c"
Nov 02 15:37:49.331282 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:37:49.331261    5304 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c52095f80a6a806c607449558de0019e7eba5c9efca4153c3fcfc3a77a707a9c\": container with ID starting with c52095f80a6a806c607449558de0019e7eba5c9efca4153c3fcfc3a77a707a9c not found: ID does not exist" containerID="c52095f80a6a806c607449558de0019e7eba5c9efca4153c3fcfc3a77a707a9c"
Nov 02 15:37:49.331355 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:37:49.331294    5304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c52095f80a6a806c607449558de0019e7eba5c9efca4153c3fcfc3a77a707a9c"} err="failed to get container status \"c52095f80a6a806c607449558de0019e7eba5c9efca4153c3fcfc3a77a707a9c\": rpc error: code = NotFound desc = could not find container \"c52095f80a6a806c607449558de0019e7eba5c9efca4153c3fcfc3a77a707a9c\": container with ID starting with c52095f80a6a806c607449558de0019e7eba5c9efca4153c3fcfc3a77a707a9c not found: ID does not exist"
Nov 02 15:37:49.386112 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:37:49.386066    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-69ff88-openshift-default-866dfd5ff7-74bk2"]
Nov 02 15:37:49.398098 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:37:49.398066    5304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-69ff88-openshift-default-866dfd5ff7-74bk2"]
Nov 02 15:37:50.120189 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:37:50.120157    5304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="301b87ba-1380-4110-96cd-5e2a057cae4d" path="/var/lib/kubelet/pods/301b87ba-1380-4110-96cd-5e2a057cae4d/volumes"
Nov 02 15:37:52.994500 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:37:52.994462    5304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-6f2e0c-openshift-default-767db5d9b7-6jmzj"]
Nov 02 15:37:52.995255 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:37:52.995234    5304 cpu_manager.go:414] "containerMap: removing container" podUID="dc422942-41f9-4cee-938f-e9df1b05d0c7" containerName="payload-processing"
Nov 02 15:37:52.995255 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:37:52.995256    5304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc422942-41f9-4cee-938f-e9df1b05d0c7" containerName="payload-processing"
Nov 02 15:37:52.995460 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:37:52.995270    5304 cpu_manager.go:414] "containerMap: removing container" podUID="6f8cf102-3f23-49fe-804f-694c50509462" containerName="istio-proxy"
Nov 02 15:37:52.995460 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:37:52.995277    5304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f8cf102-3f23-49fe-804f-694c50509462" containerName="istio-proxy"
Nov 02 15:37:52.995460 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:37:52.995290    5304 cpu_manager.go:414] "containerMap: removing container" podUID="c1051c4a-02ca-40b8-b8de-180bd2ca598b" containerName="istio-proxy"
Nov 02 15:37:52.995460 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:37:52.995299    5304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1051c4a-02ca-40b8-b8de-180bd2ca598b" containerName="istio-proxy"
Nov 02 15:37:52.995460 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:37:52.995310    5304 cpu_manager.go:414] "containerMap: removing container" podUID="5b81a985-2325-4b54-80a7-8dc2034e5596" containerName="payload-processing"
Nov 02 15:37:52.995460 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:37:52.995317    5304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b81a985-2325-4b54-80a7-8dc2034e5596" containerName="payload-processing"
Nov 02 15:37:52.995460 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:37:52.995335    5304 cpu_manager.go:414] "containerMap: removing container" podUID="e4bb2a0e-a2da-44e1-ac19-ee296cfff71c" containerName="istio-proxy"
Nov 02 15:37:52.995460 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:37:52.995342    5304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4bb2a0e-a2da-44e1-ac19-ee296cfff71c" containerName="istio-proxy"
Nov 02 15:37:52.995460 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:37:52.995351    5304 cpu_manager.go:414] "containerMap: removing container" podUID="1f9d68a5-7f9c-489b-9289-2f85b7889c2c" containerName="istio-proxy"
Nov 02 15:37:52.995460 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:37:52.995358    5304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f9d68a5-7f9c-489b-9289-2f85b7889c2c" containerName="istio-proxy"
Nov 02 15:37:52.995460 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:37:52.995368    5304 cpu_manager.go:414] "containerMap: removing container" podUID="1f9d68a5-7f9c-489b-9289-2f85b7889c2c" containerName="istio-proxy"
Nov 02 15:37:52.995460 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:37:52.995374    5304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f9d68a5-7f9c-489b-9289-2f85b7889c2c" containerName="istio-proxy"
Nov 02 15:37:52.995460 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:37:52.995393    5304 cpu_manager.go:414] "containerMap: removing container" podUID="6f8cf102-3f23-49fe-804f-694c50509462" containerName="istio-proxy"
Nov 02 15:37:52.995460 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:37:52.995400    5304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f8cf102-3f23-49fe-804f-694c50509462" containerName="istio-proxy"
Nov 02 15:37:52.995460 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:37:52.995409    5304 cpu_manager.go:414] "containerMap: removing container" podUID="0b48fb40-a77c-4345-a9d4-776bcdabb078" containerName="istio-proxy"
Nov 02 15:37:52.995460 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:37:52.995417    5304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b48fb40-a77c-4345-a9d4-776bcdabb078" containerName="istio-proxy"
Nov 02 15:37:52.995460 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:37:52.995426    5304 cpu_manager.go:414] "containerMap: removing container" podUID="89710086-0e78-4395-b8f9-e47cce4f7dfe" containerName="istio-proxy"
Nov 02 15:37:52.995460 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:37:52.995433    5304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89710086-0e78-4395-b8f9-e47cce4f7dfe" containerName="istio-proxy"
Nov 02 15:37:52.995460 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:37:52.995442    5304 cpu_manager.go:414] "containerMap: removing container" podUID="e4bb2a0e-a2da-44e1-ac19-ee296cfff71c" containerName="istio-proxy"
Nov 02 15:37:52.995460 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:37:52.995449    5304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4bb2a0e-a2da-44e1-ac19-ee296cfff71c" containerName="istio-proxy"
Nov 02 15:37:52.996489 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:37:52.995486    5304 cpu_manager.go:414] "containerMap: removing container" podUID="3ed68512-6850-45d4-af0d-a941c2bf6ec9" containerName="istio-proxy"
Nov 02 15:37:52.996489 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:37:52.995494    5304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ed68512-6850-45d4-af0d-a941c2bf6ec9" containerName="istio-proxy"
Nov 02 15:37:52.996489 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:37:52.995506    5304 cpu_manager.go:414] "containerMap: removing container" podUID="36c430b0-d26d-46cb-8b8d-f2f2a89aedbd" containerName="maas-api"
Nov 02 15:37:52.996489 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:37:52.995513    5304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="36c430b0-d26d-46cb-8b8d-f2f2a89aedbd" containerName="maas-api"
Nov 02 15:37:52.996489 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:37:52.995529    5304 cpu_manager.go:414] "containerMap: removing container" podUID="3ed68512-6850-45d4-af0d-a941c2bf6ec9" containerName="istio-proxy"
Nov 02 15:37:52.996489 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:37:52.995536    5304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ed68512-6850-45d4-af0d-a941c2bf6ec9" containerName="istio-proxy"
Nov 02 15:37:52.996489 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:37:52.995550    5304 cpu_manager.go:414] "containerMap: removing container" podUID="301b87ba-1380-4110-96cd-5e2a057cae4d" containerName="istio-proxy"
Nov 02 15:37:52.996489 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:37:52.995558    5304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="301b87ba-1380-4110-96cd-5e2a057cae4d" containerName="istio-proxy"
Nov 02 15:37:52.996489 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:37:52.995573    5304 cpu_manager.go:414] "containerMap: removing container" podUID="7a8625f9-7163-4e64-8e52-6537ebbe404c" containerName="istio-proxy"
Nov 02 15:37:52.996489 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:37:52.995580    5304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a8625f9-7163-4e64-8e52-6537ebbe404c" containerName="istio-proxy"
Nov 02 15:37:52.996489 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:37:52.995593    5304 cpu_manager.go:414] "containerMap: removing container" podUID="c1051c4a-02ca-40b8-b8de-180bd2ca598b" containerName="istio-proxy"
Nov 02 15:37:52.996489 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:37:52.995601    5304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1051c4a-02ca-40b8-b8de-180bd2ca598b" containerName="istio-proxy"
Nov 02 15:37:52.996489 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:37:52.995612    5304 cpu_manager.go:414] "containerMap: removing container" podUID="89710086-0e78-4395-b8f9-e47cce4f7dfe" containerName="istio-proxy"
Nov 02 15:37:52.996489 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:37:52.995619    5304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89710086-0e78-4395-b8f9-e47cce4f7dfe" containerName="istio-proxy"
Nov 02 15:37:52.996489 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:37:52.995743    5304 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ed68512-6850-45d4-af0d-a941c2bf6ec9" containerName="istio-proxy"
Nov 02 15:37:52.996489 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:37:52.995754    5304 memory_manager.go:365] "RemoveStaleState removing state" podUID="36c430b0-d26d-46cb-8b8d-f2f2a89aedbd" containerName="maas-api"
Nov 02 15:37:52.996489 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:37:52.995766    5304 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b48fb40-a77c-4345-a9d4-776bcdabb078" containerName="istio-proxy"
Nov 02 15:37:52.996489 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:37:52.995777    5304 memory_manager.go:365] "RemoveStaleState removing state" podUID="301b87ba-1380-4110-96cd-5e2a057cae4d" containerName="istio-proxy"
Nov 02 15:37:52.996489 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:37:52.995785    5304 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a8625f9-7163-4e64-8e52-6537ebbe404c" containerName="istio-proxy"
Nov 02 15:37:52.996489 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:37:52.995792    5304 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1051c4a-02ca-40b8-b8de-180bd2ca598b" containerName="istio-proxy"
Nov 02 15:37:52.996489 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:37:52.995801    5304 memory_manager.go:365] "RemoveStaleState removing state" podUID="89710086-0e78-4395-b8f9-e47cce4f7dfe" containerName="istio-proxy"
Nov 02 15:37:52.996489 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:37:52.995809    5304 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc422942-41f9-4cee-938f-e9df1b05d0c7" containerName="payload-processing"
Nov 02 15:37:52.996489 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:37:52.995819    5304 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f8cf102-3f23-49fe-804f-694c50509462" containerName="istio-proxy"
Nov 02 15:37:52.996489 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:37:52.995828    5304 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1051c4a-02ca-40b8-b8de-180bd2ca598b" containerName="istio-proxy"
Nov 02 15:37:52.996489 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:37:52.995838    5304 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b81a985-2325-4b54-80a7-8dc2034e5596" containerName="payload-processing"
Nov 02 15:37:52.996489 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:37:52.995847    5304 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4bb2a0e-a2da-44e1-ac19-ee296cfff71c" containerName="istio-proxy"
Nov 02 15:37:52.996489 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:37:52.995855    5304 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f9d68a5-7f9c-489b-9289-2f85b7889c2c" containerName="istio-proxy"
Nov 02 15:37:52.996489 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:37:52.995862    5304 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f9d68a5-7f9c-489b-9289-2f85b7889c2c" containerName="istio-proxy"
Nov 02 15:37:52.996489 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:37:52.995872    5304 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f8cf102-3f23-49fe-804f-694c50509462" containerName="istio-proxy"
Nov 02 15:37:52.996489 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:37:52.995882    5304 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4bb2a0e-a2da-44e1-ac19-ee296cfff71c" containerName="istio-proxy"
Nov 02 15:37:53.001426 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:37:53.001398    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-6f2e0c-openshift-default-767db5d9b7-6jmzj"
Nov 02 15:37:53.018282 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:37:53.018258    5304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-6f2e0c-openshift-default-dockercfg-mpw2f\""
Nov 02 15:37:53.031974 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:37:53.031946    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-6f2e0c-openshift-default-767db5d9b7-6jmzj"]
Nov 02 15:37:53.046061 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:37:53.046025    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9819202d-bd76-4f6f-b077-59ec79c0c50a-istio-data\") pod \"e2e-shared-a-w2-6f2e0c-openshift-default-767db5d9b7-6jmzj\" (UID: \"9819202d-bd76-4f6f-b077-59ec79c0c50a\") " pod="openshift-ingress/e2e-shared-a-w2-6f2e0c-openshift-default-767db5d9b7-6jmzj"
Nov 02 15:37:53.046266 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:37:53.046095    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9819202d-bd76-4f6f-b077-59ec79c0c50a-workload-socket\") pod \"e2e-shared-a-w2-6f2e0c-openshift-default-767db5d9b7-6jmzj\" (UID: \"9819202d-bd76-4f6f-b077-59ec79c0c50a\") " pod="openshift-ingress/e2e-shared-a-w2-6f2e0c-openshift-default-767db5d9b7-6jmzj"
Nov 02 15:37:53.046266 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:37:53.046239    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9819202d-bd76-4f6f-b077-59ec79c0c50a-istio-token\") pod \"e2e-shared-a-w2-6f2e0c-openshift-default-767db5d9b7-6jmzj\" (UID: \"9819202d-bd76-4f6f-b077-59ec79c0c50a\") " pod="openshift-ingress/e2e-shared-a-w2-6f2e0c-openshift-default-767db5d9b7-6jmzj"
Nov 02 15:37:53.046396 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:37:53.046360    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6d28d\" (UniqueName: \"kubernetes.io/projected/9819202d-bd76-4f6f-b077-59ec79c0c50a-kube-api-access-6d28d\") pod \"e2e-shared-a-w2-6f2e0c-openshift-default-767db5d9b7-6jmzj\" (UID: \"9819202d-bd76-4f6f-b077-59ec79c0c50a\") " pod="openshift-ingress/e2e-shared-a-w2-6f2e0c-openshift-default-767db5d9b7-6jmzj"
Nov 02 15:37:53.046444 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:37:53.046396    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9819202d-bd76-4f6f-b077-59ec79c0c50a-workload-certs\") pod \"e2e-shared-a-w2-6f2e0c-openshift-default-767db5d9b7-6jmzj\" (UID: \"9819202d-bd76-4f6f-b077-59ec79c0c50a\") " pod="openshift-ingress/e2e-shared-a-w2-6f2e0c-openshift-default-767db5d9b7-6jmzj"
Nov 02 15:37:53.046488 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:37:53.046463    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9819202d-bd76-4f6f-b077-59ec79c0c50a-istio-podinfo\") pod \"e2e-shared-a-w2-6f2e0c-openshift-default-767db5d9b7-6jmzj\" (UID: \"9819202d-bd76-4f6f-b077-59ec79c0c50a\") " pod="openshift-ingress/e2e-shared-a-w2-6f2e0c-openshift-default-767db5d9b7-6jmzj"
Nov 02 15:37:53.046526 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:37:53.046503    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9819202d-bd76-4f6f-b077-59ec79c0c50a-credential-socket\") pod \"e2e-shared-a-w2-6f2e0c-openshift-default-767db5d9b7-6jmzj\" (UID: \"9819202d-bd76-4f6f-b077-59ec79c0c50a\") " pod="openshift-ingress/e2e-shared-a-w2-6f2e0c-openshift-default-767db5d9b7-6jmzj"
Nov 02 15:37:53.046567 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:37:53.046529    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9819202d-bd76-4f6f-b077-59ec79c0c50a-istiod-ca-cert\") pod \"e2e-shared-a-w2-6f2e0c-openshift-default-767db5d9b7-6jmzj\" (UID: \"9819202d-bd76-4f6f-b077-59ec79c0c50a\") " pod="openshift-ingress/e2e-shared-a-w2-6f2e0c-openshift-default-767db5d9b7-6jmzj"
Nov 02 15:37:53.046616 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:37:53.046553    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9819202d-bd76-4f6f-b077-59ec79c0c50a-istio-envoy\") pod \"e2e-shared-a-w2-6f2e0c-openshift-default-767db5d9b7-6jmzj\" (UID: \"9819202d-bd76-4f6f-b077-59ec79c0c50a\") " pod="openshift-ingress/e2e-shared-a-w2-6f2e0c-openshift-default-767db5d9b7-6jmzj"
Nov 02 15:37:53.151797 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:37:53.151732    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9819202d-bd76-4f6f-b077-59ec79c0c50a-workload-socket\") pod \"e2e-shared-a-w2-6f2e0c-openshift-default-767db5d9b7-6jmzj\" (UID: \"9819202d-bd76-4f6f-b077-59ec79c0c50a\") " pod="openshift-ingress/e2e-shared-a-w2-6f2e0c-openshift-default-767db5d9b7-6jmzj"
Nov 02 15:37:53.152022 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:37:53.151842    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9819202d-bd76-4f6f-b077-59ec79c0c50a-istio-token\") pod \"e2e-shared-a-w2-6f2e0c-openshift-default-767db5d9b7-6jmzj\" (UID: \"9819202d-bd76-4f6f-b077-59ec79c0c50a\") " pod="openshift-ingress/e2e-shared-a-w2-6f2e0c-openshift-default-767db5d9b7-6jmzj"
Nov 02 15:37:53.152022 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:37:53.151975    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6d28d\" (UniqueName: \"kubernetes.io/projected/9819202d-bd76-4f6f-b077-59ec79c0c50a-kube-api-access-6d28d\") pod \"e2e-shared-a-w2-6f2e0c-openshift-default-767db5d9b7-6jmzj\" (UID: \"9819202d-bd76-4f6f-b077-59ec79c0c50a\") " pod="openshift-ingress/e2e-shared-a-w2-6f2e0c-openshift-default-767db5d9b7-6jmzj"
Nov 02 15:37:53.152022 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:37:53.152011    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9819202d-bd76-4f6f-b077-59ec79c0c50a-workload-certs\") pod \"e2e-shared-a-w2-6f2e0c-openshift-default-767db5d9b7-6jmzj\" (UID: \"9819202d-bd76-4f6f-b077-59ec79c0c50a\") " pod="openshift-ingress/e2e-shared-a-w2-6f2e0c-openshift-default-767db5d9b7-6jmzj"
Nov 02 15:37:53.152282 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:37:53.152044    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9819202d-bd76-4f6f-b077-59ec79c0c50a-istio-podinfo\") pod \"e2e-shared-a-w2-6f2e0c-openshift-default-767db5d9b7-6jmzj\" (UID: \"9819202d-bd76-4f6f-b077-59ec79c0c50a\") " pod="openshift-ingress/e2e-shared-a-w2-6f2e0c-openshift-default-767db5d9b7-6jmzj"
Nov 02 15:37:53.152282 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:37:53.152071    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9819202d-bd76-4f6f-b077-59ec79c0c50a-credential-socket\") pod \"e2e-shared-a-w2-6f2e0c-openshift-default-767db5d9b7-6jmzj\" (UID: \"9819202d-bd76-4f6f-b077-59ec79c0c50a\") " pod="openshift-ingress/e2e-shared-a-w2-6f2e0c-openshift-default-767db5d9b7-6jmzj"
Nov 02 15:37:53.152282 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:37:53.152095    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9819202d-bd76-4f6f-b077-59ec79c0c50a-istiod-ca-cert\") pod \"e2e-shared-a-w2-6f2e0c-openshift-default-767db5d9b7-6jmzj\" (UID: \"9819202d-bd76-4f6f-b077-59ec79c0c50a\") " pod="openshift-ingress/e2e-shared-a-w2-6f2e0c-openshift-default-767db5d9b7-6jmzj"
Nov 02 15:37:53.152282 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:37:53.152118    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9819202d-bd76-4f6f-b077-59ec79c0c50a-istio-envoy\") pod \"e2e-shared-a-w2-6f2e0c-openshift-default-767db5d9b7-6jmzj\" (UID: \"9819202d-bd76-4f6f-b077-59ec79c0c50a\") " pod="openshift-ingress/e2e-shared-a-w2-6f2e0c-openshift-default-767db5d9b7-6jmzj"
Nov 02 15:37:53.152548 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:37:53.152290    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9819202d-bd76-4f6f-b077-59ec79c0c50a-istio-data\") pod \"e2e-shared-a-w2-6f2e0c-openshift-default-767db5d9b7-6jmzj\" (UID: \"9819202d-bd76-4f6f-b077-59ec79c0c50a\") " pod="openshift-ingress/e2e-shared-a-w2-6f2e0c-openshift-default-767db5d9b7-6jmzj"
Nov 02 15:37:53.152804 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:37:53.152779    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9819202d-bd76-4f6f-b077-59ec79c0c50a-istio-data\") pod \"e2e-shared-a-w2-6f2e0c-openshift-default-767db5d9b7-6jmzj\" (UID: \"9819202d-bd76-4f6f-b077-59ec79c0c50a\") " pod="openshift-ingress/e2e-shared-a-w2-6f2e0c-openshift-default-767db5d9b7-6jmzj"
Nov 02 15:37:53.154397 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:37:53.153008    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9819202d-bd76-4f6f-b077-59ec79c0c50a-workload-certs\") pod \"e2e-shared-a-w2-6f2e0c-openshift-default-767db5d9b7-6jmzj\" (UID: \"9819202d-bd76-4f6f-b077-59ec79c0c50a\") " pod="openshift-ingress/e2e-shared-a-w2-6f2e0c-openshift-default-767db5d9b7-6jmzj"
Nov 02 15:37:53.154562 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:37:53.153229    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9819202d-bd76-4f6f-b077-59ec79c0c50a-credential-socket\") pod \"e2e-shared-a-w2-6f2e0c-openshift-default-767db5d9b7-6jmzj\" (UID: \"9819202d-bd76-4f6f-b077-59ec79c0c50a\") " pod="openshift-ingress/e2e-shared-a-w2-6f2e0c-openshift-default-767db5d9b7-6jmzj"
Nov 02 15:37:53.154656 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:37:53.153310    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9819202d-bd76-4f6f-b077-59ec79c0c50a-workload-socket\") pod \"e2e-shared-a-w2-6f2e0c-openshift-default-767db5d9b7-6jmzj\" (UID: \"9819202d-bd76-4f6f-b077-59ec79c0c50a\") " pod="openshift-ingress/e2e-shared-a-w2-6f2e0c-openshift-default-767db5d9b7-6jmzj"
Nov 02 15:37:53.154749 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:37:53.154321    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9819202d-bd76-4f6f-b077-59ec79c0c50a-istiod-ca-cert\") pod \"e2e-shared-a-w2-6f2e0c-openshift-default-767db5d9b7-6jmzj\" (UID: \"9819202d-bd76-4f6f-b077-59ec79c0c50a\") " pod="openshift-ingress/e2e-shared-a-w2-6f2e0c-openshift-default-767db5d9b7-6jmzj"
Nov 02 15:37:53.157581 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:37:53.157550    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9819202d-bd76-4f6f-b077-59ec79c0c50a-istio-envoy\") pod \"e2e-shared-a-w2-6f2e0c-openshift-default-767db5d9b7-6jmzj\" (UID: \"9819202d-bd76-4f6f-b077-59ec79c0c50a\") " pod="openshift-ingress/e2e-shared-a-w2-6f2e0c-openshift-default-767db5d9b7-6jmzj"
Nov 02 15:37:53.157965 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:37:53.157939    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9819202d-bd76-4f6f-b077-59ec79c0c50a-istio-podinfo\") pod \"e2e-shared-a-w2-6f2e0c-openshift-default-767db5d9b7-6jmzj\" (UID: \"9819202d-bd76-4f6f-b077-59ec79c0c50a\") " pod="openshift-ingress/e2e-shared-a-w2-6f2e0c-openshift-default-767db5d9b7-6jmzj"
Nov 02 15:37:53.181411 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:37:53.181375    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9819202d-bd76-4f6f-b077-59ec79c0c50a-istio-token\") pod \"e2e-shared-a-w2-6f2e0c-openshift-default-767db5d9b7-6jmzj\" (UID: \"9819202d-bd76-4f6f-b077-59ec79c0c50a\") " pod="openshift-ingress/e2e-shared-a-w2-6f2e0c-openshift-default-767db5d9b7-6jmzj"
Nov 02 15:37:53.181595 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:37:53.181576    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6d28d\" (UniqueName: \"kubernetes.io/projected/9819202d-bd76-4f6f-b077-59ec79c0c50a-kube-api-access-6d28d\") pod \"e2e-shared-a-w2-6f2e0c-openshift-default-767db5d9b7-6jmzj\" (UID: \"9819202d-bd76-4f6f-b077-59ec79c0c50a\") " pod="openshift-ingress/e2e-shared-a-w2-6f2e0c-openshift-default-767db5d9b7-6jmzj"
Nov 02 15:37:53.316076 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:37:53.315981    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-6f2e0c-openshift-default-767db5d9b7-6jmzj"
Nov 02 15:37:53.650357 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:37:53.650316    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-6f2e0c-openshift-default-767db5d9b7-6jmzj"]
Nov 02 15:37:53.653741 ip-10-0-135-138 kubenswrapper[5304]: W1102 15:37:53.653711    5304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9819202d_bd76_4f6f_b077_59ec79c0c50a.slice/crio-b81724df4b358b287df51821862409e240c265e9e2efd3d7a25c037717459339 WatchSource:0}: Error finding container b81724df4b358b287df51821862409e240c265e9e2efd3d7a25c037717459339: Status 404 returned error can't find the container with id b81724df4b358b287df51821862409e240c265e9e2efd3d7a25c037717459339
Nov 02 15:37:53.656435 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:37:53.656401    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:37:53.656503 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:37:53.656471    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:37:53.656539 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:37:53.656511    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:37:54.362399 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:37:54.362341    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-6f2e0c-openshift-default-767db5d9b7-6jmzj" event={"ID":"9819202d-bd76-4f6f-b077-59ec79c0c50a","Type":"ContainerStarted","Data":"46d8b1f7cadd9125636d38189583e802ef17f667738eec2f0998339ef29ba8f9"}
Nov 02 15:37:54.362829 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:37:54.362411    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-6f2e0c-openshift-default-767db5d9b7-6jmzj" event={"ID":"9819202d-bd76-4f6f-b077-59ec79c0c50a","Type":"ContainerStarted","Data":"b81724df4b358b287df51821862409e240c265e9e2efd3d7a25c037717459339"}
Nov 02 15:37:54.459830 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:37:54.459772    5304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-6f2e0c-openshift-default-767db5d9b7-6jmzj" podStartSLOduration=2.459757927 podStartE2EDuration="2.459757927s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:37:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:37:54.446295762 +0000 UTC m=+2474.440120172" watchObservedRunningTime="2026-11-02 15:37:54.459757927 +0000 UTC m=+2474.453582336"
Nov 02 15:37:54.799988 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:37:54.799909    5304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k6fjk" containerName="extractor"
Nov 02 15:37:55.317061 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:37:55.317031    5304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-6f2e0c-openshift-default-767db5d9b7-6jmzj"
Nov 02 15:37:55.325523 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:37:55.325485    5304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-6f2e0c-openshift-default-767db5d9b7-6jmzj"
Nov 02 15:37:55.369269 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:37:55.369239    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-6f2e0c-openshift-default-767db5d9b7-6jmzj"
Nov 02 15:37:55.370316 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:37:55.370293    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-6f2e0c-openshift-default-767db5d9b7-6jmzj"
Nov 02 15:37:58.951439 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:37:58.951405    5304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-6f2e0c-6568cc97f7-c7h8r"]
Nov 02 15:37:58.951976 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:37:58.951961    5304 cpu_manager.go:414] "containerMap: removing container" podUID="0b48fb40-a77c-4345-a9d4-776bcdabb078" containerName="istio-proxy"
Nov 02 15:37:58.952026 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:37:58.951979    5304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b48fb40-a77c-4345-a9d4-776bcdabb078" containerName="istio-proxy"
Nov 02 15:37:58.952112 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:37:58.952102    5304 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ed68512-6850-45d4-af0d-a941c2bf6ec9" containerName="istio-proxy"
Nov 02 15:37:58.952166 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:37:58.952115    5304 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b48fb40-a77c-4345-a9d4-776bcdabb078" containerName="istio-proxy"
Nov 02 15:37:58.952166 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:37:58.952134    5304 memory_manager.go:365] "RemoveStaleState removing state" podUID="89710086-0e78-4395-b8f9-e47cce4f7dfe" containerName="istio-proxy"
Nov 02 15:37:58.955482 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:37:58.955468    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-6f2e0c-6568cc97f7-c7h8r"
Nov 02 15:37:58.985665 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:37:58.985632    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-6f2e0c-6568cc97f7-c7h8r"]
Nov 02 15:37:59.028011 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:37:59.027973    5304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-6f2e0c-dockercfg-gwq65\""
Nov 02 15:37:59.028213 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:37:59.028075    5304 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-6f2e0c\""
Nov 02 15:37:59.032035 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:37:59.031996    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c90c9e1-7216-4ffb-b513-f5ca7fec318a-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-6f2e0c-6568cc97f7-c7h8r\" (UID: \"9c90c9e1-7216-4ffb-b513-f5ca7fec318a\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-6f2e0c-6568cc97f7-c7h8r"
Nov 02 15:37:59.032401 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:37:59.032377    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4bpfq\" (UniqueName: \"kubernetes.io/projected/9c90c9e1-7216-4ffb-b513-f5ca7fec318a-kube-api-access-4bpfq\") pod \"payload-processing-e2e-shared-a-w2-6f2e0c-6568cc97f7-c7h8r\" (UID: \"9c90c9e1-7216-4ffb-b513-f5ca7fec318a\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-6f2e0c-6568cc97f7-c7h8r"
Nov 02 15:37:59.134087 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:37:59.134053    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c90c9e1-7216-4ffb-b513-f5ca7fec318a-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-6f2e0c-6568cc97f7-c7h8r\" (UID: \"9c90c9e1-7216-4ffb-b513-f5ca7fec318a\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-6f2e0c-6568cc97f7-c7h8r"
Nov 02 15:37:59.134298 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:37:59.134177    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4bpfq\" (UniqueName: \"kubernetes.io/projected/9c90c9e1-7216-4ffb-b513-f5ca7fec318a-kube-api-access-4bpfq\") pod \"payload-processing-e2e-shared-a-w2-6f2e0c-6568cc97f7-c7h8r\" (UID: \"9c90c9e1-7216-4ffb-b513-f5ca7fec318a\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-6f2e0c-6568cc97f7-c7h8r"
Nov 02 15:37:59.134814 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:37:59.134787    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c90c9e1-7216-4ffb-b513-f5ca7fec318a-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-6f2e0c-6568cc97f7-c7h8r\" (UID: \"9c90c9e1-7216-4ffb-b513-f5ca7fec318a\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-6f2e0c-6568cc97f7-c7h8r"
Nov 02 15:37:59.157933 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:37:59.157903    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4bpfq\" (UniqueName: \"kubernetes.io/projected/9c90c9e1-7216-4ffb-b513-f5ca7fec318a-kube-api-access-4bpfq\") pod \"payload-processing-e2e-shared-a-w2-6f2e0c-6568cc97f7-c7h8r\" (UID: \"9c90c9e1-7216-4ffb-b513-f5ca7fec318a\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-6f2e0c-6568cc97f7-c7h8r"
Nov 02 15:37:59.269667 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:37:59.269581    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-6f2e0c-6568cc97f7-c7h8r"
Nov 02 15:37:59.579170 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:37:59.579138    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-6f2e0c-6568cc97f7-c7h8r"]
Nov 02 15:37:59.582189 ip-10-0-135-138 kubenswrapper[5304]: W1102 15:37:59.582158    5304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c90c9e1_7216_4ffb_b513_f5ca7fec318a.slice/crio-6a10d8720260ebc50803b06f2c87a6b6c7da679a62837a307c1cc7e510bc1498 WatchSource:0}: Error finding container 6a10d8720260ebc50803b06f2c87a6b6c7da679a62837a307c1cc7e510bc1498: Status 404 returned error can't find the container with id 6a10d8720260ebc50803b06f2c87a6b6c7da679a62837a307c1cc7e510bc1498
Nov 02 15:38:00.402927 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:38:00.402869    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-6f2e0c-6568cc97f7-c7h8r" event={"ID":"9c90c9e1-7216-4ffb-b513-f5ca7fec318a","Type":"ContainerStarted","Data":"31267b61932edcb7a3e585accc0405f9f7597f0b04201107e586506712fcd64d"}
Nov 02 15:38:00.402927 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:38:00.402933    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-6f2e0c-6568cc97f7-c7h8r"
Nov 02 15:38:00.403392 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:38:00.402951    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-6f2e0c-6568cc97f7-c7h8r" event={"ID":"9c90c9e1-7216-4ffb-b513-f5ca7fec318a","Type":"ContainerStarted","Data":"6a10d8720260ebc50803b06f2c87a6b6c7da679a62837a307c1cc7e510bc1498"}
Nov 02 15:38:00.448072 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:38:00.448000    5304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-6f2e0c-6568cc97f7-c7h8r" podStartSLOduration=2.44798175 podStartE2EDuration="2.44798175s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:37:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:38:00.446673971 +0000 UTC m=+2480.440498383" watchObservedRunningTime="2026-11-02 15:38:00.44798175 +0000 UTC m=+2480.441806158"
Nov 02 15:38:11.411710 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:38:11.411680    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-6f2e0c-6568cc97f7-c7h8r"
Nov 02 15:38:14.757795 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:38:14.757758    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-2e4ebc-6d869c5c4b-7zfcq"]
Nov 02 15:38:14.758278 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:38:14.758022    5304 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2e4ebc-6d869c5c4b-7zfcq" podUID="a48f863f-1237-46a0-9949-0ddecd8b350c" containerName="maas-api" containerID="cri-o://431bf7259acaecd2ae83969a0f007f36be1e2938e991dbaadfc12461845918df" gracePeriod=30
Nov 02 15:38:15.039809 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:38:15.039781    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2e4ebc-6d869c5c4b-7zfcq"
Nov 02 15:38:15.144460 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:38:15.144424    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a48f863f-1237-46a0-9949-0ddecd8b350c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a48f863f-1237-46a0-9949-0ddecd8b350c-metrics-certs\") pod \"a48f863f-1237-46a0-9949-0ddecd8b350c\" (UID: \"a48f863f-1237-46a0-9949-0ddecd8b350c\") "
Nov 02 15:38:15.144663 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:38:15.144595    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a48f863f-1237-46a0-9949-0ddecd8b350c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a48f863f-1237-46a0-9949-0ddecd8b350c-openshift-service-ca\") pod \"a48f863f-1237-46a0-9949-0ddecd8b350c\" (UID: \"a48f863f-1237-46a0-9949-0ddecd8b350c\") "
Nov 02 15:38:15.144745 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:38:15.144686    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a48f863f-1237-46a0-9949-0ddecd8b350c-kube-api-access-gfjg7\" (UniqueName: \"kubernetes.io/projected/a48f863f-1237-46a0-9949-0ddecd8b350c-kube-api-access-gfjg7\") pod \"a48f863f-1237-46a0-9949-0ddecd8b350c\" (UID: \"a48f863f-1237-46a0-9949-0ddecd8b350c\") "
Nov 02 15:38:15.144745 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:38:15.144705    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a48f863f-1237-46a0-9949-0ddecd8b350c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a48f863f-1237-46a0-9949-0ddecd8b350c-maas-api-tls\") pod \"a48f863f-1237-46a0-9949-0ddecd8b350c\" (UID: \"a48f863f-1237-46a0-9949-0ddecd8b350c\") "
Nov 02 15:38:15.144982 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:38:15.144948    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a48f863f-1237-46a0-9949-0ddecd8b350c-openshift-service-ca" pod "a48f863f-1237-46a0-9949-0ddecd8b350c" (UID: "a48f863f-1237-46a0-9949-0ddecd8b350c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:38:15.145383 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:38:15.145349    5304 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a48f863f-1237-46a0-9949-0ddecd8b350c-openshift-service-ca\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:38:15.146824 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:38:15.146796    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a48f863f-1237-46a0-9949-0ddecd8b350c-kube-api-access-gfjg7" pod "a48f863f-1237-46a0-9949-0ddecd8b350c" (UID: "a48f863f-1237-46a0-9949-0ddecd8b350c"). InnerVolumeSpecName "kube-api-access-gfjg7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:38:15.146950 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:38:15.146819    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a48f863f-1237-46a0-9949-0ddecd8b350c-metrics-certs" pod "a48f863f-1237-46a0-9949-0ddecd8b350c" (UID: "a48f863f-1237-46a0-9949-0ddecd8b350c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:38:15.146950 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:38:15.146890    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a48f863f-1237-46a0-9949-0ddecd8b350c-maas-api-tls" pod "a48f863f-1237-46a0-9949-0ddecd8b350c" (UID: "a48f863f-1237-46a0-9949-0ddecd8b350c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:38:15.247484 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:38:15.247438    5304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gfjg7\" (UniqueName: \"kubernetes.io/projected/a48f863f-1237-46a0-9949-0ddecd8b350c-kube-api-access-gfjg7\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:38:15.247484 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:38:15.247480    5304 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a48f863f-1237-46a0-9949-0ddecd8b350c-maas-api-tls\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:38:15.247484 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:38:15.247494    5304 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a48f863f-1237-46a0-9949-0ddecd8b350c-metrics-certs\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:38:15.489372 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:38:15.489332    5304 generic.go:356] "Generic (PLEG): container finished" podID="a48f863f-1237-46a0-9949-0ddecd8b350c" containerID="431bf7259acaecd2ae83969a0f007f36be1e2938e991dbaadfc12461845918df" exitCode=0
Nov 02 15:38:15.489643 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:38:15.489379    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2e4ebc-6d869c5c4b-7zfcq"
Nov 02 15:38:15.489643 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:38:15.489417    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2e4ebc-6d869c5c4b-7zfcq" event={"ID":"a48f863f-1237-46a0-9949-0ddecd8b350c","Type":"ContainerDied","Data":"431bf7259acaecd2ae83969a0f007f36be1e2938e991dbaadfc12461845918df"}
Nov 02 15:38:15.489643 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:38:15.489478    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2e4ebc-6d869c5c4b-7zfcq" event={"ID":"a48f863f-1237-46a0-9949-0ddecd8b350c","Type":"ContainerDied","Data":"3ae1e80f99060bc6eee04d86151fd8a3b0a5d758bd50c22ba76da68f79fd7fef"}
Nov 02 15:38:15.489643 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:38:15.489509    5304 scope.go:122] "RemoveContainer" containerID="431bf7259acaecd2ae83969a0f007f36be1e2938e991dbaadfc12461845918df"
Nov 02 15:38:15.500366 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:38:15.500347    5304 scope.go:122] "RemoveContainer" containerID="431bf7259acaecd2ae83969a0f007f36be1e2938e991dbaadfc12461845918df"
Nov 02 15:38:15.500620 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:38:15.500604    5304 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"431bf7259acaecd2ae83969a0f007f36be1e2938e991dbaadfc12461845918df\": container with ID starting with 431bf7259acaecd2ae83969a0f007f36be1e2938e991dbaadfc12461845918df not found: ID does not exist" containerID="431bf7259acaecd2ae83969a0f007f36be1e2938e991dbaadfc12461845918df"
Nov 02 15:38:15.500670 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:38:15.500626    5304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"431bf7259acaecd2ae83969a0f007f36be1e2938e991dbaadfc12461845918df"} err="failed to get container status \"431bf7259acaecd2ae83969a0f007f36be1e2938e991dbaadfc12461845918df\": rpc error: code = NotFound desc = could not find container \"431bf7259acaecd2ae83969a0f007f36be1e2938e991dbaadfc12461845918df\": container with ID starting with 431bf7259acaecd2ae83969a0f007f36be1e2938e991dbaadfc12461845918df not found: ID does not exist"
Nov 02 15:38:15.549609 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:38:15.549580    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-2e4ebc-6d869c5c4b-7zfcq"]
Nov 02 15:38:15.561353 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:38:15.561327    5304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-2e4ebc-6d869c5c4b-7zfcq"]
Nov 02 15:38:16.118324 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:38:16.118290    5304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a48f863f-1237-46a0-9949-0ddecd8b350c" path="/var/lib/kubelet/pods/a48f863f-1237-46a0-9949-0ddecd8b350c/volumes"
Nov 02 15:38:24.325925 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:38:24.325895    5304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-f6f1a0-openshift-default-cd57f47bf-2p5bx"]
Nov 02 15:38:24.328327 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:38:24.326434    5304 cpu_manager.go:414] "containerMap: removing container" podUID="a48f863f-1237-46a0-9949-0ddecd8b350c" containerName="maas-api"
Nov 02 15:38:24.328327 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:38:24.326449    5304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a48f863f-1237-46a0-9949-0ddecd8b350c" containerName="maas-api"
Nov 02 15:38:24.328327 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:38:24.326570    5304 memory_manager.go:365] "RemoveStaleState removing state" podUID="a48f863f-1237-46a0-9949-0ddecd8b350c" containerName="maas-api"
Nov 02 15:38:24.329572 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:38:24.329557    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-f6f1a0-openshift-default-cd57f47bf-2p5bx"
Nov 02 15:38:24.346664 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:38:24.346641    5304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-f6f1a0-openshift-default-dockercfg-49pnn\""
Nov 02 15:38:24.375670 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:38:24.375636    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-f6f1a0-openshift-default-cd57f47bf-2p5bx"]
Nov 02 15:38:24.459394 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:38:24.459357    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55aea2b2-3109-4d02-a3d5-c65835f4d6bf-workload-certs\") pod \"e2e-shared-a-w3-f6f1a0-openshift-default-cd57f47bf-2p5bx\" (UID: \"55aea2b2-3109-4d02-a3d5-c65835f4d6bf\") " pod="openshift-ingress/e2e-shared-a-w3-f6f1a0-openshift-default-cd57f47bf-2p5bx"
Nov 02 15:38:24.459594 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:38:24.459415    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55aea2b2-3109-4d02-a3d5-c65835f4d6bf-credential-socket\") pod \"e2e-shared-a-w3-f6f1a0-openshift-default-cd57f47bf-2p5bx\" (UID: \"55aea2b2-3109-4d02-a3d5-c65835f4d6bf\") " pod="openshift-ingress/e2e-shared-a-w3-f6f1a0-openshift-default-cd57f47bf-2p5bx"
Nov 02 15:38:24.459594 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:38:24.459548    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8qbxk\" (UniqueName: \"kubernetes.io/projected/55aea2b2-3109-4d02-a3d5-c65835f4d6bf-kube-api-access-8qbxk\") pod \"e2e-shared-a-w3-f6f1a0-openshift-default-cd57f47bf-2p5bx\" (UID: \"55aea2b2-3109-4d02-a3d5-c65835f4d6bf\") " pod="openshift-ingress/e2e-shared-a-w3-f6f1a0-openshift-default-cd57f47bf-2p5bx"
Nov 02 15:38:24.459688 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:38:24.459605    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/55aea2b2-3109-4d02-a3d5-c65835f4d6bf-istio-token\") pod \"e2e-shared-a-w3-f6f1a0-openshift-default-cd57f47bf-2p5bx\" (UID: \"55aea2b2-3109-4d02-a3d5-c65835f4d6bf\") " pod="openshift-ingress/e2e-shared-a-w3-f6f1a0-openshift-default-cd57f47bf-2p5bx"
Nov 02 15:38:24.459764 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:38:24.459739    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55aea2b2-3109-4d02-a3d5-c65835f4d6bf-istiod-ca-cert\") pod \"e2e-shared-a-w3-f6f1a0-openshift-default-cd57f47bf-2p5bx\" (UID: \"55aea2b2-3109-4d02-a3d5-c65835f4d6bf\") " pod="openshift-ingress/e2e-shared-a-w3-f6f1a0-openshift-default-cd57f47bf-2p5bx"
Nov 02 15:38:24.459844 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:38:24.459829    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55aea2b2-3109-4d02-a3d5-c65835f4d6bf-istio-envoy\") pod \"e2e-shared-a-w3-f6f1a0-openshift-default-cd57f47bf-2p5bx\" (UID: \"55aea2b2-3109-4d02-a3d5-c65835f4d6bf\") " pod="openshift-ingress/e2e-shared-a-w3-f6f1a0-openshift-default-cd57f47bf-2p5bx"
Nov 02 15:38:24.459906 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:38:24.459857    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55aea2b2-3109-4d02-a3d5-c65835f4d6bf-workload-socket\") pod \"e2e-shared-a-w3-f6f1a0-openshift-default-cd57f47bf-2p5bx\" (UID: \"55aea2b2-3109-4d02-a3d5-c65835f4d6bf\") " pod="openshift-ingress/e2e-shared-a-w3-f6f1a0-openshift-default-cd57f47bf-2p5bx"
Nov 02 15:38:24.459906 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:38:24.459897    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55aea2b2-3109-4d02-a3d5-c65835f4d6bf-istio-data\") pod \"e2e-shared-a-w3-f6f1a0-openshift-default-cd57f47bf-2p5bx\" (UID: \"55aea2b2-3109-4d02-a3d5-c65835f4d6bf\") " pod="openshift-ingress/e2e-shared-a-w3-f6f1a0-openshift-default-cd57f47bf-2p5bx"
Nov 02 15:38:24.460014 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:38:24.459983    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55aea2b2-3109-4d02-a3d5-c65835f4d6bf-istio-podinfo\") pod \"e2e-shared-a-w3-f6f1a0-openshift-default-cd57f47bf-2p5bx\" (UID: \"55aea2b2-3109-4d02-a3d5-c65835f4d6bf\") " pod="openshift-ingress/e2e-shared-a-w3-f6f1a0-openshift-default-cd57f47bf-2p5bx"
Nov 02 15:38:24.561070 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:38:24.561030    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55aea2b2-3109-4d02-a3d5-c65835f4d6bf-istio-envoy\") pod \"e2e-shared-a-w3-f6f1a0-openshift-default-cd57f47bf-2p5bx\" (UID: \"55aea2b2-3109-4d02-a3d5-c65835f4d6bf\") " pod="openshift-ingress/e2e-shared-a-w3-f6f1a0-openshift-default-cd57f47bf-2p5bx"
Nov 02 15:38:24.561285 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:38:24.561080    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55aea2b2-3109-4d02-a3d5-c65835f4d6bf-workload-socket\") pod \"e2e-shared-a-w3-f6f1a0-openshift-default-cd57f47bf-2p5bx\" (UID: \"55aea2b2-3109-4d02-a3d5-c65835f4d6bf\") " pod="openshift-ingress/e2e-shared-a-w3-f6f1a0-openshift-default-cd57f47bf-2p5bx"
Nov 02 15:38:24.561285 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:38:24.561137    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55aea2b2-3109-4d02-a3d5-c65835f4d6bf-istio-data\") pod \"e2e-shared-a-w3-f6f1a0-openshift-default-cd57f47bf-2p5bx\" (UID: \"55aea2b2-3109-4d02-a3d5-c65835f4d6bf\") " pod="openshift-ingress/e2e-shared-a-w3-f6f1a0-openshift-default-cd57f47bf-2p5bx"
Nov 02 15:38:24.561285 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:38:24.561178    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55aea2b2-3109-4d02-a3d5-c65835f4d6bf-istio-podinfo\") pod \"e2e-shared-a-w3-f6f1a0-openshift-default-cd57f47bf-2p5bx\" (UID: \"55aea2b2-3109-4d02-a3d5-c65835f4d6bf\") " pod="openshift-ingress/e2e-shared-a-w3-f6f1a0-openshift-default-cd57f47bf-2p5bx"
Nov 02 15:38:24.561285 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:38:24.561237    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55aea2b2-3109-4d02-a3d5-c65835f4d6bf-workload-certs\") pod \"e2e-shared-a-w3-f6f1a0-openshift-default-cd57f47bf-2p5bx\" (UID: \"55aea2b2-3109-4d02-a3d5-c65835f4d6bf\") " pod="openshift-ingress/e2e-shared-a-w3-f6f1a0-openshift-default-cd57f47bf-2p5bx"
Nov 02 15:38:24.561285 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:38:24.561279    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55aea2b2-3109-4d02-a3d5-c65835f4d6bf-credential-socket\") pod \"e2e-shared-a-w3-f6f1a0-openshift-default-cd57f47bf-2p5bx\" (UID: \"55aea2b2-3109-4d02-a3d5-c65835f4d6bf\") " pod="openshift-ingress/e2e-shared-a-w3-f6f1a0-openshift-default-cd57f47bf-2p5bx"
Nov 02 15:38:24.561566 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:38:24.561317    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8qbxk\" (UniqueName: \"kubernetes.io/projected/55aea2b2-3109-4d02-a3d5-c65835f4d6bf-kube-api-access-8qbxk\") pod \"e2e-shared-a-w3-f6f1a0-openshift-default-cd57f47bf-2p5bx\" (UID: \"55aea2b2-3109-4d02-a3d5-c65835f4d6bf\") " pod="openshift-ingress/e2e-shared-a-w3-f6f1a0-openshift-default-cd57f47bf-2p5bx"
Nov 02 15:38:24.561566 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:38:24.561358    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/55aea2b2-3109-4d02-a3d5-c65835f4d6bf-istio-token\") pod \"e2e-shared-a-w3-f6f1a0-openshift-default-cd57f47bf-2p5bx\" (UID: \"55aea2b2-3109-4d02-a3d5-c65835f4d6bf\") " pod="openshift-ingress/e2e-shared-a-w3-f6f1a0-openshift-default-cd57f47bf-2p5bx"
Nov 02 15:38:24.561566 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:38:24.561462    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55aea2b2-3109-4d02-a3d5-c65835f4d6bf-istiod-ca-cert\") pod \"e2e-shared-a-w3-f6f1a0-openshift-default-cd57f47bf-2p5bx\" (UID: \"55aea2b2-3109-4d02-a3d5-c65835f4d6bf\") " pod="openshift-ingress/e2e-shared-a-w3-f6f1a0-openshift-default-cd57f47bf-2p5bx"
Nov 02 15:38:24.561976 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:38:24.561942    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55aea2b2-3109-4d02-a3d5-c65835f4d6bf-istio-data\") pod \"e2e-shared-a-w3-f6f1a0-openshift-default-cd57f47bf-2p5bx\" (UID: \"55aea2b2-3109-4d02-a3d5-c65835f4d6bf\") " pod="openshift-ingress/e2e-shared-a-w3-f6f1a0-openshift-default-cd57f47bf-2p5bx"
Nov 02 15:38:24.562321 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:38:24.562279    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55aea2b2-3109-4d02-a3d5-c65835f4d6bf-workload-socket\") pod \"e2e-shared-a-w3-f6f1a0-openshift-default-cd57f47bf-2p5bx\" (UID: \"55aea2b2-3109-4d02-a3d5-c65835f4d6bf\") " pod="openshift-ingress/e2e-shared-a-w3-f6f1a0-openshift-default-cd57f47bf-2p5bx"
Nov 02 15:38:24.562886 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:38:24.562855    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55aea2b2-3109-4d02-a3d5-c65835f4d6bf-credential-socket\") pod \"e2e-shared-a-w3-f6f1a0-openshift-default-cd57f47bf-2p5bx\" (UID: \"55aea2b2-3109-4d02-a3d5-c65835f4d6bf\") " pod="openshift-ingress/e2e-shared-a-w3-f6f1a0-openshift-default-cd57f47bf-2p5bx"
Nov 02 15:38:24.569006 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:38:24.565515    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55aea2b2-3109-4d02-a3d5-c65835f4d6bf-istiod-ca-cert\") pod \"e2e-shared-a-w3-f6f1a0-openshift-default-cd57f47bf-2p5bx\" (UID: \"55aea2b2-3109-4d02-a3d5-c65835f4d6bf\") " pod="openshift-ingress/e2e-shared-a-w3-f6f1a0-openshift-default-cd57f47bf-2p5bx"
Nov 02 15:38:24.569006 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:38:24.567279    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55aea2b2-3109-4d02-a3d5-c65835f4d6bf-workload-certs\") pod \"e2e-shared-a-w3-f6f1a0-openshift-default-cd57f47bf-2p5bx\" (UID: \"55aea2b2-3109-4d02-a3d5-c65835f4d6bf\") " pod="openshift-ingress/e2e-shared-a-w3-f6f1a0-openshift-default-cd57f47bf-2p5bx"
Nov 02 15:38:24.569006 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:38:24.567839    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55aea2b2-3109-4d02-a3d5-c65835f4d6bf-istio-podinfo\") pod \"e2e-shared-a-w3-f6f1a0-openshift-default-cd57f47bf-2p5bx\" (UID: \"55aea2b2-3109-4d02-a3d5-c65835f4d6bf\") " pod="openshift-ingress/e2e-shared-a-w3-f6f1a0-openshift-default-cd57f47bf-2p5bx"
Nov 02 15:38:24.569006 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:38:24.568745    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55aea2b2-3109-4d02-a3d5-c65835f4d6bf-istio-envoy\") pod \"e2e-shared-a-w3-f6f1a0-openshift-default-cd57f47bf-2p5bx\" (UID: \"55aea2b2-3109-4d02-a3d5-c65835f4d6bf\") " pod="openshift-ingress/e2e-shared-a-w3-f6f1a0-openshift-default-cd57f47bf-2p5bx"
Nov 02 15:38:24.592009 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:38:24.591975    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8qbxk\" (UniqueName: \"kubernetes.io/projected/55aea2b2-3109-4d02-a3d5-c65835f4d6bf-kube-api-access-8qbxk\") pod \"e2e-shared-a-w3-f6f1a0-openshift-default-cd57f47bf-2p5bx\" (UID: \"55aea2b2-3109-4d02-a3d5-c65835f4d6bf\") " pod="openshift-ingress/e2e-shared-a-w3-f6f1a0-openshift-default-cd57f47bf-2p5bx"
Nov 02 15:38:24.592383 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:38:24.592362    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/55aea2b2-3109-4d02-a3d5-c65835f4d6bf-istio-token\") pod \"e2e-shared-a-w3-f6f1a0-openshift-default-cd57f47bf-2p5bx\" (UID: \"55aea2b2-3109-4d02-a3d5-c65835f4d6bf\") " pod="openshift-ingress/e2e-shared-a-w3-f6f1a0-openshift-default-cd57f47bf-2p5bx"
Nov 02 15:38:24.643378 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:38:24.643344    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-f6f1a0-openshift-default-cd57f47bf-2p5bx"
Nov 02 15:38:24.798730 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:38:24.798682    5304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k6fjk" containerName="extractor"
Nov 02 15:38:24.970159 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:38:24.970046    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-f6f1a0-openshift-default-cd57f47bf-2p5bx"]
Nov 02 15:38:24.973916 ip-10-0-135-138 kubenswrapper[5304]: W1102 15:38:24.973891    5304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55aea2b2_3109_4d02_a3d5_c65835f4d6bf.slice/crio-0e8682bd4b5937755a2e6cbd56a44e749b4ca1609acb12e265c477fccc9b1c0e WatchSource:0}: Error finding container 0e8682bd4b5937755a2e6cbd56a44e749b4ca1609acb12e265c477fccc9b1c0e: Status 404 returned error can't find the container with id 0e8682bd4b5937755a2e6cbd56a44e749b4ca1609acb12e265c477fccc9b1c0e
Nov 02 15:38:24.976569 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:38:24.976528    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:38:24.976676 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:38:24.976605    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:38:24.976676 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:38:24.976640    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:38:25.553233 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:38:25.553194    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-f6f1a0-openshift-default-cd57f47bf-2p5bx" event={"ID":"55aea2b2-3109-4d02-a3d5-c65835f4d6bf","Type":"ContainerStarted","Data":"fcc4d456dad10bb63c54e6a834b29b56725b6ca722ddf1fd64f22249c1641da9"}
Nov 02 15:38:25.553738 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:38:25.553245    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-f6f1a0-openshift-default-cd57f47bf-2p5bx" event={"ID":"55aea2b2-3109-4d02-a3d5-c65835f4d6bf","Type":"ContainerStarted","Data":"0e8682bd4b5937755a2e6cbd56a44e749b4ca1609acb12e265c477fccc9b1c0e"}
Nov 02 15:38:25.643605 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:38:25.643573    5304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-f6f1a0-openshift-default-cd57f47bf-2p5bx"
Nov 02 15:38:26.650088 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:38:26.650061    5304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-f6f1a0-openshift-default-cd57f47bf-2p5bx"
Nov 02 15:38:26.707633 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:38:26.707581    5304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-f6f1a0-openshift-default-cd57f47bf-2p5bx" podStartSLOduration=2.7075682150000002 podStartE2EDuration="2.707568215s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:38:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:38:25.626655816 +0000 UTC m=+2505.620480225" watchObservedRunningTime="2026-11-02 15:38:26.707568215 +0000 UTC m=+2506.701392623"
Nov 02 15:38:27.564607 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:38:27.564581    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-f6f1a0-openshift-default-cd57f47bf-2p5bx"
Nov 02 15:38:27.565660 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:38:27.565639    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-f6f1a0-openshift-default-cd57f47bf-2p5bx"
Nov 02 15:38:31.100774 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:38:31.100735    5304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-f6f1a0-bb944cbdc-m2w4j"]
Nov 02 15:38:31.104691 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:38:31.104674    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-f6f1a0-bb944cbdc-m2w4j"
Nov 02 15:38:31.169638 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:38:31.169605    5304 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-f6f1a0\""
Nov 02 15:38:31.170044 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:38:31.170016    5304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-f6f1a0-dockercfg-7pz4z\""
Nov 02 15:38:31.180288 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:38:31.180256    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-f6f1a0-bb944cbdc-m2w4j"]
Nov 02 15:38:31.274037 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:38:31.274001    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0fc51d17-635f-4f36-90d6-f2c546a457ba-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-f6f1a0-bb944cbdc-m2w4j\" (UID: \"0fc51d17-635f-4f36-90d6-f2c546a457ba\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-f6f1a0-bb944cbdc-m2w4j"
Nov 02 15:38:31.274266 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:38:31.274066    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zvrh5\" (UniqueName: \"kubernetes.io/projected/0fc51d17-635f-4f36-90d6-f2c546a457ba-kube-api-access-zvrh5\") pod \"payload-processing-e2e-shared-a-w3-f6f1a0-bb944cbdc-m2w4j\" (UID: \"0fc51d17-635f-4f36-90d6-f2c546a457ba\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-f6f1a0-bb944cbdc-m2w4j"
Nov 02 15:38:31.375849 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:38:31.375759    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0fc51d17-635f-4f36-90d6-f2c546a457ba-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-f6f1a0-bb944cbdc-m2w4j\" (UID: \"0fc51d17-635f-4f36-90d6-f2c546a457ba\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-f6f1a0-bb944cbdc-m2w4j"
Nov 02 15:38:31.375849 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:38:31.375830    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvrh5\" (UniqueName: \"kubernetes.io/projected/0fc51d17-635f-4f36-90d6-f2c546a457ba-kube-api-access-zvrh5\") pod \"payload-processing-e2e-shared-a-w3-f6f1a0-bb944cbdc-m2w4j\" (UID: \"0fc51d17-635f-4f36-90d6-f2c546a457ba\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-f6f1a0-bb944cbdc-m2w4j"
Nov 02 15:38:31.376602 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:38:31.376574    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0fc51d17-635f-4f36-90d6-f2c546a457ba-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-f6f1a0-bb944cbdc-m2w4j\" (UID: \"0fc51d17-635f-4f36-90d6-f2c546a457ba\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-f6f1a0-bb944cbdc-m2w4j"
Nov 02 15:38:31.399685 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:38:31.399653    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvrh5\" (UniqueName: \"kubernetes.io/projected/0fc51d17-635f-4f36-90d6-f2c546a457ba-kube-api-access-zvrh5\") pod \"payload-processing-e2e-shared-a-w3-f6f1a0-bb944cbdc-m2w4j\" (UID: \"0fc51d17-635f-4f36-90d6-f2c546a457ba\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-f6f1a0-bb944cbdc-m2w4j"
Nov 02 15:38:31.481853 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:38:31.481816    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-f6f1a0-bb944cbdc-m2w4j"
Nov 02 15:38:31.812023 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:38:31.811950    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-f6f1a0-bb944cbdc-m2w4j"]
Nov 02 15:38:31.816846 ip-10-0-135-138 kubenswrapper[5304]: W1102 15:38:31.816813    5304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0fc51d17_635f_4f36_90d6_f2c546a457ba.slice/crio-b390064ff57c54db0dd21ea71f3e46818cdef48f162e1dde5a3dd5b519636623 WatchSource:0}: Error finding container b390064ff57c54db0dd21ea71f3e46818cdef48f162e1dde5a3dd5b519636623: Status 404 returned error can't find the container with id b390064ff57c54db0dd21ea71f3e46818cdef48f162e1dde5a3dd5b519636623
Nov 02 15:38:32.613749 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:38:32.613703    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-f6f1a0-bb944cbdc-m2w4j" event={"ID":"0fc51d17-635f-4f36-90d6-f2c546a457ba","Type":"ContainerStarted","Data":"8f9bf1e387ebe321eb770ccd6dbd945057133d3e238474c297ed03ced5bd7bcf"}
Nov 02 15:38:32.614211 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:38:32.613776    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-f6f1a0-bb944cbdc-m2w4j" event={"ID":"0fc51d17-635f-4f36-90d6-f2c546a457ba","Type":"ContainerStarted","Data":"b390064ff57c54db0dd21ea71f3e46818cdef48f162e1dde5a3dd5b519636623"}
Nov 02 15:38:32.614211 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:38:32.613803    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-f6f1a0-bb944cbdc-m2w4j"
Nov 02 15:38:32.692698 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:38:32.692627    5304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-f6f1a0-bb944cbdc-m2w4j" podStartSLOduration=1.692610416 podStartE2EDuration="1.692610416s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:38:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:38:32.680226872 +0000 UTC m=+2512.674051282" watchObservedRunningTime="2026-11-02 15:38:32.692610416 +0000 UTC m=+2512.686434826"
Nov 02 15:38:43.623193 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:38:43.623151    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-f6f1a0-bb944cbdc-m2w4j"
Nov 02 15:38:54.798663 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:38:54.798571    5304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k6fjk" containerName="extractor"
Nov 02 15:39:22.995481 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:39:22.995442    5304 generic.go:356] "Generic (PLEG): container finished" podID="9819202d-bd76-4f6f-b077-59ec79c0c50a" containerID="46d8b1f7cadd9125636d38189583e802ef17f667738eec2f0998339ef29ba8f9" exitCode=0
Nov 02 15:39:22.995950 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:39:22.995515    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-6f2e0c-openshift-default-767db5d9b7-6jmzj" event={"ID":"9819202d-bd76-4f6f-b077-59ec79c0c50a","Type":"ContainerDied","Data":"46d8b1f7cadd9125636d38189583e802ef17f667738eec2f0998339ef29ba8f9"}
Nov 02 15:39:22.996000 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:39:22.995956    5304 scope.go:122] "RemoveContainer" containerID="46d8b1f7cadd9125636d38189583e802ef17f667738eec2f0998339ef29ba8f9"
Nov 02 15:39:22.997661 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:39:22.997633    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:39:22.997724 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:39:22.997702    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:39:22.997765 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:39:22.997737    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:39:24.003504 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:39:24.003472    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-6f2e0c-openshift-default-767db5d9b7-6jmzj" event={"ID":"9819202d-bd76-4f6f-b077-59ec79c0c50a","Type":"ContainerStarted","Data":"c48074e374290de95f1882f29f363dfe1128af91498a3b92a31a69287ba520af"}
Nov 02 15:39:24.317041 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:39:24.316952    5304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-6f2e0c-openshift-default-767db5d9b7-6jmzj"
Nov 02 15:39:24.318336 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:39:24.318310    5304 patch_prober.go:29] interesting pod/e2e-shared-a-w2-6f2e0c-openshift-default-767db5d9b7-6jmzj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" start-of-body=
Nov 02 15:39:24.318452 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:39:24.318379    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-6f2e0c-openshift-default-767db5d9b7-6jmzj" podUID="9819202d-bd76-4f6f-b077-59ec79c0c50a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused"
Nov 02 15:39:24.800119 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:39:24.800078    5304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k6fjk" containerName="extractor"
Nov 02 15:39:25.317527 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:39:25.317494    5304 patch_prober.go:29] interesting pod/e2e-shared-a-w2-6f2e0c-openshift-default-767db5d9b7-6jmzj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" start-of-body=
Nov 02 15:39:25.317935 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:39:25.317556    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-6f2e0c-openshift-default-767db5d9b7-6jmzj" podUID="9819202d-bd76-4f6f-b077-59ec79c0c50a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused"
Nov 02 15:39:25.369551 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:39:25.369514    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-6f2e0c-openshift-default-767db5d9b7-6jmzj"
Nov 02 15:39:26.317603 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:39:26.317566    5304 patch_prober.go:29] interesting pod/e2e-shared-a-w2-6f2e0c-openshift-default-767db5d9b7-6jmzj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" start-of-body=
Nov 02 15:39:26.318002 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:39:26.317618    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-6f2e0c-openshift-default-767db5d9b7-6jmzj" podUID="9819202d-bd76-4f6f-b077-59ec79c0c50a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused"
Nov 02 15:39:27.317746 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:39:27.317708    5304 patch_prober.go:29] interesting pod/e2e-shared-a-w2-6f2e0c-openshift-default-767db5d9b7-6jmzj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" start-of-body=
Nov 02 15:39:27.318173 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:39:27.317766    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-6f2e0c-openshift-default-767db5d9b7-6jmzj" podUID="9819202d-bd76-4f6f-b077-59ec79c0c50a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused"
Nov 02 15:39:28.316767 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:39:28.316729    5304 patch_prober.go:29] interesting pod/e2e-shared-a-w2-6f2e0c-openshift-default-767db5d9b7-6jmzj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" start-of-body=
Nov 02 15:39:28.316945 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:39:28.316798    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-6f2e0c-openshift-default-767db5d9b7-6jmzj" podUID="9819202d-bd76-4f6f-b077-59ec79c0c50a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused"
Nov 02 15:39:29.322246 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:39:29.322211    5304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-6f2e0c-openshift-default-767db5d9b7-6jmzj"
Nov 02 15:39:29.323185 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:39:29.323167    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-6f2e0c-openshift-default-767db5d9b7-6jmzj"
Nov 02 15:39:39.502574 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:39:39.502541    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w4-501738/auth-test-model-501738-kserve-85cfb4c9cc-xkzq5"]
Nov 02 15:39:44.849990 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:39:44.849948    5304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-9cfd1f-openshift-default-6fbb7b57d9-hh28t"]
Nov 02 15:39:44.853070 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:39:44.853049    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-9cfd1f-openshift-default-6fbb7b57d9-hh28t"
Nov 02 15:39:44.871331 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:39:44.871306    5304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-9cfd1f-openshift-default-dockercfg-fjhdb\""
Nov 02 15:39:44.887189 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:39:44.887156    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-9cfd1f-openshift-default-6fbb7b57d9-hh28t"]
Nov 02 15:39:44.907889 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:39:44.907855    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d665d588-97c3-4797-be57-c426b3d905ff-workload-certs\") pod \"e2e-ipp-a-9cfd1f-openshift-default-6fbb7b57d9-hh28t\" (UID: \"d665d588-97c3-4797-be57-c426b3d905ff\") " pod="openshift-ingress/e2e-ipp-a-9cfd1f-openshift-default-6fbb7b57d9-hh28t"
Nov 02 15:39:44.908089 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:39:44.907904    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d665d588-97c3-4797-be57-c426b3d905ff-istiod-ca-cert\") pod \"e2e-ipp-a-9cfd1f-openshift-default-6fbb7b57d9-hh28t\" (UID: \"d665d588-97c3-4797-be57-c426b3d905ff\") " pod="openshift-ingress/e2e-ipp-a-9cfd1f-openshift-default-6fbb7b57d9-hh28t"
Nov 02 15:39:44.908089 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:39:44.908057    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d665d588-97c3-4797-be57-c426b3d905ff-istio-token\") pod \"e2e-ipp-a-9cfd1f-openshift-default-6fbb7b57d9-hh28t\" (UID: \"d665d588-97c3-4797-be57-c426b3d905ff\") " pod="openshift-ingress/e2e-ipp-a-9cfd1f-openshift-default-6fbb7b57d9-hh28t"
Nov 02 15:39:44.908255 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:39:44.908095    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d665d588-97c3-4797-be57-c426b3d905ff-credential-socket\") pod \"e2e-ipp-a-9cfd1f-openshift-default-6fbb7b57d9-hh28t\" (UID: \"d665d588-97c3-4797-be57-c426b3d905ff\") " pod="openshift-ingress/e2e-ipp-a-9cfd1f-openshift-default-6fbb7b57d9-hh28t"
Nov 02 15:39:44.908255 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:39:44.908218    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d665d588-97c3-4797-be57-c426b3d905ff-istio-data\") pod \"e2e-ipp-a-9cfd1f-openshift-default-6fbb7b57d9-hh28t\" (UID: \"d665d588-97c3-4797-be57-c426b3d905ff\") " pod="openshift-ingress/e2e-ipp-a-9cfd1f-openshift-default-6fbb7b57d9-hh28t"
Nov 02 15:39:44.908369 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:39:44.908342    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d665d588-97c3-4797-be57-c426b3d905ff-workload-socket\") pod \"e2e-ipp-a-9cfd1f-openshift-default-6fbb7b57d9-hh28t\" (UID: \"d665d588-97c3-4797-be57-c426b3d905ff\") " pod="openshift-ingress/e2e-ipp-a-9cfd1f-openshift-default-6fbb7b57d9-hh28t"
Nov 02 15:39:44.908441 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:39:44.908423    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d665d588-97c3-4797-be57-c426b3d905ff-istio-envoy\") pod \"e2e-ipp-a-9cfd1f-openshift-default-6fbb7b57d9-hh28t\" (UID: \"d665d588-97c3-4797-be57-c426b3d905ff\") " pod="openshift-ingress/e2e-ipp-a-9cfd1f-openshift-default-6fbb7b57d9-hh28t"
Nov 02 15:39:44.908518 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:39:44.908472    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dj54j\" (UniqueName: \"kubernetes.io/projected/d665d588-97c3-4797-be57-c426b3d905ff-kube-api-access-dj54j\") pod \"e2e-ipp-a-9cfd1f-openshift-default-6fbb7b57d9-hh28t\" (UID: \"d665d588-97c3-4797-be57-c426b3d905ff\") " pod="openshift-ingress/e2e-ipp-a-9cfd1f-openshift-default-6fbb7b57d9-hh28t"
Nov 02 15:39:44.908650 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:39:44.908536    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d665d588-97c3-4797-be57-c426b3d905ff-istio-podinfo\") pod \"e2e-ipp-a-9cfd1f-openshift-default-6fbb7b57d9-hh28t\" (UID: \"d665d588-97c3-4797-be57-c426b3d905ff\") " pod="openshift-ingress/e2e-ipp-a-9cfd1f-openshift-default-6fbb7b57d9-hh28t"
Nov 02 15:39:45.010197 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:39:45.010165    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d665d588-97c3-4797-be57-c426b3d905ff-istio-data\") pod \"e2e-ipp-a-9cfd1f-openshift-default-6fbb7b57d9-hh28t\" (UID: \"d665d588-97c3-4797-be57-c426b3d905ff\") " pod="openshift-ingress/e2e-ipp-a-9cfd1f-openshift-default-6fbb7b57d9-hh28t"
Nov 02 15:39:45.010358 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:39:45.010310    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d665d588-97c3-4797-be57-c426b3d905ff-workload-socket\") pod \"e2e-ipp-a-9cfd1f-openshift-default-6fbb7b57d9-hh28t\" (UID: \"d665d588-97c3-4797-be57-c426b3d905ff\") " pod="openshift-ingress/e2e-ipp-a-9cfd1f-openshift-default-6fbb7b57d9-hh28t"
Nov 02 15:39:45.010402 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:39:45.010363    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d665d588-97c3-4797-be57-c426b3d905ff-istio-envoy\") pod \"e2e-ipp-a-9cfd1f-openshift-default-6fbb7b57d9-hh28t\" (UID: \"d665d588-97c3-4797-be57-c426b3d905ff\") " pod="openshift-ingress/e2e-ipp-a-9cfd1f-openshift-default-6fbb7b57d9-hh28t"
Nov 02 15:39:45.010402 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:39:45.010388    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dj54j\" (UniqueName: \"kubernetes.io/projected/d665d588-97c3-4797-be57-c426b3d905ff-kube-api-access-dj54j\") pod \"e2e-ipp-a-9cfd1f-openshift-default-6fbb7b57d9-hh28t\" (UID: \"d665d588-97c3-4797-be57-c426b3d905ff\") " pod="openshift-ingress/e2e-ipp-a-9cfd1f-openshift-default-6fbb7b57d9-hh28t"
Nov 02 15:39:45.010487 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:39:45.010419    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d665d588-97c3-4797-be57-c426b3d905ff-istio-podinfo\") pod \"e2e-ipp-a-9cfd1f-openshift-default-6fbb7b57d9-hh28t\" (UID: \"d665d588-97c3-4797-be57-c426b3d905ff\") " pod="openshift-ingress/e2e-ipp-a-9cfd1f-openshift-default-6fbb7b57d9-hh28t"
Nov 02 15:39:45.010548 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:39:45.010524    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d665d588-97c3-4797-be57-c426b3d905ff-istio-data\") pod \"e2e-ipp-a-9cfd1f-openshift-default-6fbb7b57d9-hh28t\" (UID: \"d665d588-97c3-4797-be57-c426b3d905ff\") " pod="openshift-ingress/e2e-ipp-a-9cfd1f-openshift-default-6fbb7b57d9-hh28t"
Nov 02 15:39:45.010609 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:39:45.010533    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d665d588-97c3-4797-be57-c426b3d905ff-workload-certs\") pod \"e2e-ipp-a-9cfd1f-openshift-default-6fbb7b57d9-hh28t\" (UID: \"d665d588-97c3-4797-be57-c426b3d905ff\") " pod="openshift-ingress/e2e-ipp-a-9cfd1f-openshift-default-6fbb7b57d9-hh28t"
Nov 02 15:39:45.010666 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:39:45.010607    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d665d588-97c3-4797-be57-c426b3d905ff-istiod-ca-cert\") pod \"e2e-ipp-a-9cfd1f-openshift-default-6fbb7b57d9-hh28t\" (UID: \"d665d588-97c3-4797-be57-c426b3d905ff\") " pod="openshift-ingress/e2e-ipp-a-9cfd1f-openshift-default-6fbb7b57d9-hh28t"
Nov 02 15:39:45.010726 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:39:45.010665    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d665d588-97c3-4797-be57-c426b3d905ff-istio-token\") pod \"e2e-ipp-a-9cfd1f-openshift-default-6fbb7b57d9-hh28t\" (UID: \"d665d588-97c3-4797-be57-c426b3d905ff\") " pod="openshift-ingress/e2e-ipp-a-9cfd1f-openshift-default-6fbb7b57d9-hh28t"
Nov 02 15:39:45.010726 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:39:45.010680    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d665d588-97c3-4797-be57-c426b3d905ff-workload-socket\") pod \"e2e-ipp-a-9cfd1f-openshift-default-6fbb7b57d9-hh28t\" (UID: \"d665d588-97c3-4797-be57-c426b3d905ff\") " pod="openshift-ingress/e2e-ipp-a-9cfd1f-openshift-default-6fbb7b57d9-hh28t"
Nov 02 15:39:45.010726 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:39:45.010691    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d665d588-97c3-4797-be57-c426b3d905ff-credential-socket\") pod \"e2e-ipp-a-9cfd1f-openshift-default-6fbb7b57d9-hh28t\" (UID: \"d665d588-97c3-4797-be57-c426b3d905ff\") " pod="openshift-ingress/e2e-ipp-a-9cfd1f-openshift-default-6fbb7b57d9-hh28t"
Nov 02 15:39:45.010897 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:39:45.010784    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d665d588-97c3-4797-be57-c426b3d905ff-workload-certs\") pod \"e2e-ipp-a-9cfd1f-openshift-default-6fbb7b57d9-hh28t\" (UID: \"d665d588-97c3-4797-be57-c426b3d905ff\") " pod="openshift-ingress/e2e-ipp-a-9cfd1f-openshift-default-6fbb7b57d9-hh28t"
Nov 02 15:39:45.011074 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:39:45.011049    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d665d588-97c3-4797-be57-c426b3d905ff-credential-socket\") pod \"e2e-ipp-a-9cfd1f-openshift-default-6fbb7b57d9-hh28t\" (UID: \"d665d588-97c3-4797-be57-c426b3d905ff\") " pod="openshift-ingress/e2e-ipp-a-9cfd1f-openshift-default-6fbb7b57d9-hh28t"
Nov 02 15:39:45.011365 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:39:45.011348    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d665d588-97c3-4797-be57-c426b3d905ff-istiod-ca-cert\") pod \"e2e-ipp-a-9cfd1f-openshift-default-6fbb7b57d9-hh28t\" (UID: \"d665d588-97c3-4797-be57-c426b3d905ff\") " pod="openshift-ingress/e2e-ipp-a-9cfd1f-openshift-default-6fbb7b57d9-hh28t"
Nov 02 15:39:45.012753 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:39:45.012733    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d665d588-97c3-4797-be57-c426b3d905ff-istio-envoy\") pod \"e2e-ipp-a-9cfd1f-openshift-default-6fbb7b57d9-hh28t\" (UID: \"d665d588-97c3-4797-be57-c426b3d905ff\") " pod="openshift-ingress/e2e-ipp-a-9cfd1f-openshift-default-6fbb7b57d9-hh28t"
Nov 02 15:39:45.012950 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:39:45.012932    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d665d588-97c3-4797-be57-c426b3d905ff-istio-podinfo\") pod \"e2e-ipp-a-9cfd1f-openshift-default-6fbb7b57d9-hh28t\" (UID: \"d665d588-97c3-4797-be57-c426b3d905ff\") " pod="openshift-ingress/e2e-ipp-a-9cfd1f-openshift-default-6fbb7b57d9-hh28t"
Nov 02 15:39:45.037343 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:39:45.037314    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d665d588-97c3-4797-be57-c426b3d905ff-istio-token\") pod \"e2e-ipp-a-9cfd1f-openshift-default-6fbb7b57d9-hh28t\" (UID: \"d665d588-97c3-4797-be57-c426b3d905ff\") " pod="openshift-ingress/e2e-ipp-a-9cfd1f-openshift-default-6fbb7b57d9-hh28t"
Nov 02 15:39:45.037518 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:39:45.037388    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dj54j\" (UniqueName: \"kubernetes.io/projected/d665d588-97c3-4797-be57-c426b3d905ff-kube-api-access-dj54j\") pod \"e2e-ipp-a-9cfd1f-openshift-default-6fbb7b57d9-hh28t\" (UID: \"d665d588-97c3-4797-be57-c426b3d905ff\") " pod="openshift-ingress/e2e-ipp-a-9cfd1f-openshift-default-6fbb7b57d9-hh28t"
Nov 02 15:39:45.169434 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:39:45.169356    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-9cfd1f-openshift-default-6fbb7b57d9-hh28t"
Nov 02 15:39:45.546659 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:39:45.546628    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-9cfd1f-openshift-default-6fbb7b57d9-hh28t"]
Nov 02 15:39:45.548036 ip-10-0-135-138 kubenswrapper[5304]: W1102 15:39:45.548011    5304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd665d588_97c3_4797_be57_c426b3d905ff.slice/crio-352deb3e37dd4d1391328f659027d810c3d22db470f34c40a7fdced015cac2d3 WatchSource:0}: Error finding container 352deb3e37dd4d1391328f659027d810c3d22db470f34c40a7fdced015cac2d3: Status 404 returned error can't find the container with id 352deb3e37dd4d1391328f659027d810c3d22db470f34c40a7fdced015cac2d3
Nov 02 15:39:45.551001 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:39:45.550966    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:39:45.551073 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:39:45.551037    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:39:45.551073 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:39:45.551068    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:39:46.193032 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:39:46.192991    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-9cfd1f-openshift-default-6fbb7b57d9-hh28t" event={"ID":"d665d588-97c3-4797-be57-c426b3d905ff","Type":"ContainerStarted","Data":"844d78510fb78a43c3c7c69d7bafedfd7f55e584f99d15174806ac946b6c56a3"}
Nov 02 15:39:46.193550 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:39:46.193050    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-9cfd1f-openshift-default-6fbb7b57d9-hh28t" event={"ID":"d665d588-97c3-4797-be57-c426b3d905ff","Type":"ContainerStarted","Data":"352deb3e37dd4d1391328f659027d810c3d22db470f34c40a7fdced015cac2d3"}
Nov 02 15:39:46.268951 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:39:46.268898    5304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-9cfd1f-openshift-default-6fbb7b57d9-hh28t" podStartSLOduration=2.268881906 podStartE2EDuration="2.268881906s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:39:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:39:46.254624596 +0000 UTC m=+2586.248449008" watchObservedRunningTime="2026-11-02 15:39:46.268881906 +0000 UTC m=+2586.262706376"
Nov 02 15:39:47.170297 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:39:47.170252    5304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-9cfd1f-openshift-default-6fbb7b57d9-hh28t"
Nov 02 15:39:47.180747 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:39:47.180710    5304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-9cfd1f-openshift-default-6fbb7b57d9-hh28t"
Nov 02 15:39:47.199683 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:39:47.199645    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-9cfd1f-openshift-default-6fbb7b57d9-hh28t"
Nov 02 15:39:47.200774 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:39:47.200751    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-9cfd1f-openshift-default-6fbb7b57d9-hh28t"
Nov 02 15:39:54.524220 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:39:54.524104    5304 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w4-501738/auth-test-model-501738-kserve-85cfb4c9cc-xkzq5" podUID="c4240adb-0390-4339-a752-209bdd4b9b50" containerName="main" containerID="cri-o://ddf8a0cc36dfc36aa248d0dc8b1655e9b489a379dd0261d665c76d1101b36ca7" gracePeriod=45
Nov 02 15:39:54.806079 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:39:54.806036    5304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k6fjk" containerName="extractor"
Nov 02 15:39:54.821520 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:39:54.821492    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-501738/auth-test-model-501738-kserve-85cfb4c9cc-xkzq5"
Nov 02 15:39:54.940016 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:39:54.939985    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4240adb-0390-4339-a752-209bdd4b9b50-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c4240adb-0390-4339-a752-209bdd4b9b50-tmp-dir\") pod \"c4240adb-0390-4339-a752-209bdd4b9b50\" (UID: \"c4240adb-0390-4339-a752-209bdd4b9b50\") "
Nov 02 15:39:54.940231 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:39:54.940053    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4240adb-0390-4339-a752-209bdd4b9b50-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c4240adb-0390-4339-a752-209bdd4b9b50-model-cache\") pod \"c4240adb-0390-4339-a752-209bdd4b9b50\" (UID: \"c4240adb-0390-4339-a752-209bdd4b9b50\") "
Nov 02 15:39:54.940231 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:39:54.940179    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4240adb-0390-4339-a752-209bdd4b9b50-kube-api-access-kwwpg\" (UniqueName: \"kubernetes.io/projected/c4240adb-0390-4339-a752-209bdd4b9b50-kube-api-access-kwwpg\") pod \"c4240adb-0390-4339-a752-209bdd4b9b50\" (UID: \"c4240adb-0390-4339-a752-209bdd4b9b50\") "
Nov 02 15:39:54.940367 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:39:54.940268    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c4240adb-0390-4339-a752-209bdd4b9b50-tls-certs\" (UniqueName: \"kubernetes.io/secret/c4240adb-0390-4339-a752-209bdd4b9b50-tls-certs\") pod \"c4240adb-0390-4339-a752-209bdd4b9b50\" (UID: \"c4240adb-0390-4339-a752-209bdd4b9b50\") "
Nov 02 15:39:54.940367 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:39:54.940274    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4240adb-0390-4339-a752-209bdd4b9b50-tmp-dir" pod "c4240adb-0390-4339-a752-209bdd4b9b50" (UID: "c4240adb-0390-4339-a752-209bdd4b9b50"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:39:54.940367 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:39:54.940327    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4240adb-0390-4339-a752-209bdd4b9b50-dshm\" (UniqueName: \"kubernetes.io/empty-dir/c4240adb-0390-4339-a752-209bdd4b9b50-dshm\") pod \"c4240adb-0390-4339-a752-209bdd4b9b50\" (UID: \"c4240adb-0390-4339-a752-209bdd4b9b50\") "
Nov 02 15:39:54.940367 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:39:54.940353    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4240adb-0390-4339-a752-209bdd4b9b50-home\" (UniqueName: \"kubernetes.io/empty-dir/c4240adb-0390-4339-a752-209bdd4b9b50-home\") pod \"c4240adb-0390-4339-a752-209bdd4b9b50\" (UID: \"c4240adb-0390-4339-a752-209bdd4b9b50\") "
Nov 02 15:39:54.940599 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:39:54.940357    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4240adb-0390-4339-a752-209bdd4b9b50-model-cache" pod "c4240adb-0390-4339-a752-209bdd4b9b50" (UID: "c4240adb-0390-4339-a752-209bdd4b9b50"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:39:54.940684 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:39:54.940665    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4240adb-0390-4339-a752-209bdd4b9b50-home" pod "c4240adb-0390-4339-a752-209bdd4b9b50" (UID: "c4240adb-0390-4339-a752-209bdd4b9b50"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:39:54.940817 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:39:54.940799    5304 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c4240adb-0390-4339-a752-209bdd4b9b50-home\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:39:54.940887 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:39:54.940840    5304 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c4240adb-0390-4339-a752-209bdd4b9b50-tmp-dir\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:39:54.940887 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:39:54.940858    5304 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c4240adb-0390-4339-a752-209bdd4b9b50-model-cache\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:39:54.942639 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:39:54.942580    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4240adb-0390-4339-a752-209bdd4b9b50-kube-api-access-kwwpg" pod "c4240adb-0390-4339-a752-209bdd4b9b50" (UID: "c4240adb-0390-4339-a752-209bdd4b9b50"). InnerVolumeSpecName "kube-api-access-kwwpg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:39:54.942846 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:39:54.942822    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4240adb-0390-4339-a752-209bdd4b9b50-dshm" pod "c4240adb-0390-4339-a752-209bdd4b9b50" (UID: "c4240adb-0390-4339-a752-209bdd4b9b50"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:39:54.942951 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:39:54.942850    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c4240adb-0390-4339-a752-209bdd4b9b50-tls-certs" pod "c4240adb-0390-4339-a752-209bdd4b9b50" (UID: "c4240adb-0390-4339-a752-209bdd4b9b50"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:39:55.042038 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:39:55.041949    5304 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c4240adb-0390-4339-a752-209bdd4b9b50-dshm\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:39:55.042038 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:39:55.041988    5304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kwwpg\" (UniqueName: \"kubernetes.io/projected/c4240adb-0390-4339-a752-209bdd4b9b50-kube-api-access-kwwpg\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:39:55.042038 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:39:55.042002    5304 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c4240adb-0390-4339-a752-209bdd4b9b50-tls-certs\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:39:55.288254 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:39:55.288218    5304 generic.go:356] "Generic (PLEG): container finished" podID="c4240adb-0390-4339-a752-209bdd4b9b50" containerID="ddf8a0cc36dfc36aa248d0dc8b1655e9b489a379dd0261d665c76d1101b36ca7" exitCode=0
Nov 02 15:39:55.288459 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:39:55.288277    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-501738/auth-test-model-501738-kserve-85cfb4c9cc-xkzq5"
Nov 02 15:39:55.288459 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:39:55.288310    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-501738/auth-test-model-501738-kserve-85cfb4c9cc-xkzq5" event={"ID":"c4240adb-0390-4339-a752-209bdd4b9b50","Type":"ContainerDied","Data":"ddf8a0cc36dfc36aa248d0dc8b1655e9b489a379dd0261d665c76d1101b36ca7"}
Nov 02 15:39:55.288459 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:39:55.288363    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-501738/auth-test-model-501738-kserve-85cfb4c9cc-xkzq5" event={"ID":"c4240adb-0390-4339-a752-209bdd4b9b50","Type":"ContainerDied","Data":"8d3b36d1aed724d6343a57d5c75026a6bb4dcf8a8c758b460ce586851acbf9c4"}
Nov 02 15:39:55.288459 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:39:55.288391    5304 scope.go:122] "RemoveContainer" containerID="ddf8a0cc36dfc36aa248d0dc8b1655e9b489a379dd0261d665c76d1101b36ca7"
Nov 02 15:39:55.300416 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:39:55.300394    5304 scope.go:122] "RemoveContainer" containerID="ddf8a0cc36dfc36aa248d0dc8b1655e9b489a379dd0261d665c76d1101b36ca7"
Nov 02 15:39:55.300811 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:39:55.300787    5304 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ddf8a0cc36dfc36aa248d0dc8b1655e9b489a379dd0261d665c76d1101b36ca7\": container with ID starting with ddf8a0cc36dfc36aa248d0dc8b1655e9b489a379dd0261d665c76d1101b36ca7 not found: ID does not exist" containerID="ddf8a0cc36dfc36aa248d0dc8b1655e9b489a379dd0261d665c76d1101b36ca7"
Nov 02 15:39:55.300875 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:39:55.300823    5304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ddf8a0cc36dfc36aa248d0dc8b1655e9b489a379dd0261d665c76d1101b36ca7"} err="failed to get container status \"ddf8a0cc36dfc36aa248d0dc8b1655e9b489a379dd0261d665c76d1101b36ca7\": rpc error: code = NotFound desc = could not find container \"ddf8a0cc36dfc36aa248d0dc8b1655e9b489a379dd0261d665c76d1101b36ca7\": container with ID starting with ddf8a0cc36dfc36aa248d0dc8b1655e9b489a379dd0261d665c76d1101b36ca7 not found: ID does not exist"
Nov 02 15:39:55.347346 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:39:55.347308    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w4-501738/auth-test-model-501738-kserve-85cfb4c9cc-xkzq5"]
Nov 02 15:39:55.358891 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:39:55.358855    5304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w4-501738/auth-test-model-501738-kserve-85cfb4c9cc-xkzq5"]
Nov 02 15:39:56.121720 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:39:56.121689    5304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c4240adb-0390-4339-a752-209bdd4b9b50" path="/var/lib/kubelet/pods/c4240adb-0390-4339-a752-209bdd4b9b50/volumes"
Nov 02 15:39:58.330555 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:39:58.329807    5304 patch_prober.go:29] interesting pod/e2e-shared-a-w3-f6f1a0-openshift-default-cd57f47bf-2p5bx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.157:15021/healthz/ready\": read tcp 10.133.0.2:43816->10.133.0.157:15021: read: connection reset by peer" start-of-body=
Nov 02 15:39:58.330555 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:39:58.329877    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w3-f6f1a0-openshift-default-cd57f47bf-2p5bx" podUID="55aea2b2-3109-4d02-a3d5-c65835f4d6bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.157:15021/healthz/ready\": read tcp 10.133.0.2:43816->10.133.0.157:15021: read: connection reset by peer"
Nov 02 15:39:59.315799 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:39:59.315764    5304 generic.go:356] "Generic (PLEG): container finished" podID="55aea2b2-3109-4d02-a3d5-c65835f4d6bf" containerID="fcc4d456dad10bb63c54e6a834b29b56725b6ca722ddf1fd64f22249c1641da9" exitCode=0
Nov 02 15:39:59.315980 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:39:59.315847    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-f6f1a0-openshift-default-cd57f47bf-2p5bx" event={"ID":"55aea2b2-3109-4d02-a3d5-c65835f4d6bf","Type":"ContainerDied","Data":"fcc4d456dad10bb63c54e6a834b29b56725b6ca722ddf1fd64f22249c1641da9"}
Nov 02 15:39:59.316403 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:39:59.316388    5304 scope.go:122] "RemoveContainer" containerID="fcc4d456dad10bb63c54e6a834b29b56725b6ca722ddf1fd64f22249c1641da9"
Nov 02 15:39:59.318858 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:39:59.318828    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:39:59.318980 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:39:59.318902    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:39:59.318980 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:39:59.318935    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:40:00.324614 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:40:00.324579    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-f6f1a0-openshift-default-cd57f47bf-2p5bx" event={"ID":"55aea2b2-3109-4d02-a3d5-c65835f4d6bf","Type":"ContainerStarted","Data":"92d99e959f1dcec0272ffbe25756aab6e0c5b705c3507c2463e1d0eeb51f793b"}
Nov 02 15:40:00.652603 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:40:00.652570    5304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-f6f1a0-openshift-default-cd57f47bf-2p5bx"
Nov 02 15:40:00.653028 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:40:00.652923    5304 patch_prober.go:29] interesting pod/e2e-shared-a-w3-f6f1a0-openshift-default-cd57f47bf-2p5bx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" start-of-body=
Nov 02 15:40:00.653241 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:40:00.653217    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-f6f1a0-openshift-default-cd57f47bf-2p5bx" podUID="55aea2b2-3109-4d02-a3d5-c65835f4d6bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused"
Nov 02 15:40:01.650157 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:40:01.649757    5304 patch_prober.go:29] interesting pod/e2e-shared-a-w3-f6f1a0-openshift-default-cd57f47bf-2p5bx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" start-of-body=
Nov 02 15:40:01.650157 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:40:01.649820    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-f6f1a0-openshift-default-cd57f47bf-2p5bx" podUID="55aea2b2-3109-4d02-a3d5-c65835f4d6bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused"
Nov 02 15:40:02.645605 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:40:02.645559    5304 patch_prober.go:29] interesting pod/e2e-shared-a-w3-f6f1a0-openshift-default-cd57f47bf-2p5bx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" start-of-body=
Nov 02 15:40:02.645948 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:40:02.645916    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-f6f1a0-openshift-default-cd57f47bf-2p5bx" podUID="55aea2b2-3109-4d02-a3d5-c65835f4d6bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused"
Nov 02 15:40:03.649178 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:40:03.648706    5304 patch_prober.go:29] interesting pod/e2e-shared-a-w3-f6f1a0-openshift-default-cd57f47bf-2p5bx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" start-of-body=
Nov 02 15:40:03.649178 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:40:03.648775    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-f6f1a0-openshift-default-cd57f47bf-2p5bx" podUID="55aea2b2-3109-4d02-a3d5-c65835f4d6bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused"
Nov 02 15:40:04.649586 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:40:04.649544    5304 patch_prober.go:29] interesting pod/e2e-shared-a-w3-f6f1a0-openshift-default-cd57f47bf-2p5bx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" start-of-body=
Nov 02 15:40:04.650289 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:40:04.650250    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-f6f1a0-openshift-default-cd57f47bf-2p5bx" podUID="55aea2b2-3109-4d02-a3d5-c65835f4d6bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused"
Nov 02 15:40:05.643849 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:40:05.643809    5304 patch_prober.go:29] interesting pod/e2e-shared-a-w3-f6f1a0-openshift-default-cd57f47bf-2p5bx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" start-of-body=
Nov 02 15:40:05.644041 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:40:05.643884    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-f6f1a0-openshift-default-cd57f47bf-2p5bx" podUID="55aea2b2-3109-4d02-a3d5-c65835f4d6bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused"
Nov 02 15:40:06.655692 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:40:06.655653    5304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-f6f1a0-openshift-default-cd57f47bf-2p5bx"
Nov 02 15:40:06.657190 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:40:06.657165    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-f6f1a0-openshift-default-cd57f47bf-2p5bx"
Nov 02 15:40:06.659739 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:40:06.659713    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-f6f1a0-openshift-default-cd57f47bf-2p5bx"
Nov 02 15:40:14.775873 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:40:14.775834    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w1-fda646/auth-test-model-fda646-kserve-568f9ddd9-t6lxf"]
Nov 02 15:40:15.202252 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:40:15.202218    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w6-b955c5/auth-test-model-b955c5-kserve-7965dcb68f-75hkb"]
Nov 02 15:40:24.799473 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:40:24.799389    5304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k6fjk" containerName="extractor"
Nov 02 15:40:29.797861 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:40:29.797810    5304 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w1-fda646/auth-test-model-fda646-kserve-568f9ddd9-t6lxf" podUID="7e84c4e2-b5e2-4ba8-a6e8-8090a06ac68c" containerName="main" containerID="cri-o://3aaad7829e41605fe646a6dd017ea705ffd0d8207c90df5eea30c87ee418c27f" gracePeriod=45
Nov 02 15:40:30.068347 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:40:30.068321    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-fda646/auth-test-model-fda646-kserve-568f9ddd9-t6lxf"
Nov 02 15:40:30.153381 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:40:30.153343    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e84c4e2-b5e2-4ba8-a6e8-8090a06ac68c-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e84c4e2-b5e2-4ba8-a6e8-8090a06ac68c-tmp-dir\") pod \"7e84c4e2-b5e2-4ba8-a6e8-8090a06ac68c\" (UID: \"7e84c4e2-b5e2-4ba8-a6e8-8090a06ac68c\") "
Nov 02 15:40:30.153569 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:40:30.153430    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e84c4e2-b5e2-4ba8-a6e8-8090a06ac68c-kube-api-access-xh978\" (UniqueName: \"kubernetes.io/projected/7e84c4e2-b5e2-4ba8-a6e8-8090a06ac68c-kube-api-access-xh978\") pod \"7e84c4e2-b5e2-4ba8-a6e8-8090a06ac68c\" (UID: \"7e84c4e2-b5e2-4ba8-a6e8-8090a06ac68c\") "
Nov 02 15:40:30.153569 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:40:30.153468    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e84c4e2-b5e2-4ba8-a6e8-8090a06ac68c-dshm\" (UniqueName: \"kubernetes.io/empty-dir/7e84c4e2-b5e2-4ba8-a6e8-8090a06ac68c-dshm\") pod \"7e84c4e2-b5e2-4ba8-a6e8-8090a06ac68c\" (UID: \"7e84c4e2-b5e2-4ba8-a6e8-8090a06ac68c\") "
Nov 02 15:40:30.153712 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:40:30.153654    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e84c4e2-b5e2-4ba8-a6e8-8090a06ac68c-tmp-dir" pod "7e84c4e2-b5e2-4ba8-a6e8-8090a06ac68c" (UID: "7e84c4e2-b5e2-4ba8-a6e8-8090a06ac68c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:40:30.153712 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:40:30.153675    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e84c4e2-b5e2-4ba8-a6e8-8090a06ac68c-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7e84c4e2-b5e2-4ba8-a6e8-8090a06ac68c-model-cache\") pod \"7e84c4e2-b5e2-4ba8-a6e8-8090a06ac68c\" (UID: \"7e84c4e2-b5e2-4ba8-a6e8-8090a06ac68c\") "
Nov 02 15:40:30.153826 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:40:30.153772    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7e84c4e2-b5e2-4ba8-a6e8-8090a06ac68c-tls-certs\" (UniqueName: \"kubernetes.io/secret/7e84c4e2-b5e2-4ba8-a6e8-8090a06ac68c-tls-certs\") pod \"7e84c4e2-b5e2-4ba8-a6e8-8090a06ac68c\" (UID: \"7e84c4e2-b5e2-4ba8-a6e8-8090a06ac68c\") "
Nov 02 15:40:30.153917 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:40:30.153884    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e84c4e2-b5e2-4ba8-a6e8-8090a06ac68c-model-cache" pod "7e84c4e2-b5e2-4ba8-a6e8-8090a06ac68c" (UID: "7e84c4e2-b5e2-4ba8-a6e8-8090a06ac68c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:40:30.153974 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:40:30.153945    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e84c4e2-b5e2-4ba8-a6e8-8090a06ac68c-home\" (UniqueName: \"kubernetes.io/empty-dir/7e84c4e2-b5e2-4ba8-a6e8-8090a06ac68c-home\") pod \"7e84c4e2-b5e2-4ba8-a6e8-8090a06ac68c\" (UID: \"7e84c4e2-b5e2-4ba8-a6e8-8090a06ac68c\") "
Nov 02 15:40:30.154315 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:40:30.154273    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e84c4e2-b5e2-4ba8-a6e8-8090a06ac68c-home" pod "7e84c4e2-b5e2-4ba8-a6e8-8090a06ac68c" (UID: "7e84c4e2-b5e2-4ba8-a6e8-8090a06ac68c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:40:30.154604 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:40:30.154586    5304 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7e84c4e2-b5e2-4ba8-a6e8-8090a06ac68c-home\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:40:30.154656 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:40:30.154610    5304 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e84c4e2-b5e2-4ba8-a6e8-8090a06ac68c-tmp-dir\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:40:30.154656 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:40:30.154626    5304 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7e84c4e2-b5e2-4ba8-a6e8-8090a06ac68c-model-cache\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:40:30.155803 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:40:30.155782    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e84c4e2-b5e2-4ba8-a6e8-8090a06ac68c-kube-api-access-xh978" pod "7e84c4e2-b5e2-4ba8-a6e8-8090a06ac68c" (UID: "7e84c4e2-b5e2-4ba8-a6e8-8090a06ac68c"). InnerVolumeSpecName "kube-api-access-xh978". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:40:30.156280 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:40:30.156255    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e84c4e2-b5e2-4ba8-a6e8-8090a06ac68c-dshm" pod "7e84c4e2-b5e2-4ba8-a6e8-8090a06ac68c" (UID: "7e84c4e2-b5e2-4ba8-a6e8-8090a06ac68c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:40:30.156391 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:40:30.156328    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7e84c4e2-b5e2-4ba8-a6e8-8090a06ac68c-tls-certs" pod "7e84c4e2-b5e2-4ba8-a6e8-8090a06ac68c" (UID: "7e84c4e2-b5e2-4ba8-a6e8-8090a06ac68c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:40:30.223119 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:40:30.223081    5304 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w6-b955c5/auth-test-model-b955c5-kserve-7965dcb68f-75hkb" podUID="61fd91e0-c9e9-4336-b47c-81f9e2799829" containerName="main" containerID="cri-o://c49f2acccdf4a9e0984df2047dd2c9c747bf749d124acdc5570d94cf4b56949b" gracePeriod=45
Nov 02 15:40:30.255449 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:40:30.255417    5304 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e84c4e2-b5e2-4ba8-a6e8-8090a06ac68c-tls-certs\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:40:30.255449 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:40:30.255445    5304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xh978\" (UniqueName: \"kubernetes.io/projected/7e84c4e2-b5e2-4ba8-a6e8-8090a06ac68c-kube-api-access-xh978\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:40:30.255449 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:40:30.255455    5304 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7e84c4e2-b5e2-4ba8-a6e8-8090a06ac68c-dshm\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:40:30.481888 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:40:30.481861    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w6-b955c5/auth-test-model-b955c5-kserve-7965dcb68f-75hkb"
Nov 02 15:40:30.524135 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:40:30.524089    5304 generic.go:356] "Generic (PLEG): container finished" podID="7e84c4e2-b5e2-4ba8-a6e8-8090a06ac68c" containerID="3aaad7829e41605fe646a6dd017ea705ffd0d8207c90df5eea30c87ee418c27f" exitCode=0
Nov 02 15:40:30.524317 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:40:30.524151    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-fda646/auth-test-model-fda646-kserve-568f9ddd9-t6lxf"
Nov 02 15:40:30.524317 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:40:30.524162    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-fda646/auth-test-model-fda646-kserve-568f9ddd9-t6lxf" event={"ID":"7e84c4e2-b5e2-4ba8-a6e8-8090a06ac68c","Type":"ContainerDied","Data":"3aaad7829e41605fe646a6dd017ea705ffd0d8207c90df5eea30c87ee418c27f"}
Nov 02 15:40:30.524317 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:40:30.524208    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-fda646/auth-test-model-fda646-kserve-568f9ddd9-t6lxf" event={"ID":"7e84c4e2-b5e2-4ba8-a6e8-8090a06ac68c","Type":"ContainerDied","Data":"0220bd62c1c608cb1f9736b7f9ab56d1fd1eded19095e11fae4b99943e64a9bb"}
Nov 02 15:40:30.524317 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:40:30.524231    5304 scope.go:122] "RemoveContainer" containerID="3aaad7829e41605fe646a6dd017ea705ffd0d8207c90df5eea30c87ee418c27f"
Nov 02 15:40:30.525835 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:40:30.525799    5304 generic.go:356] "Generic (PLEG): container finished" podID="61fd91e0-c9e9-4336-b47c-81f9e2799829" containerID="c49f2acccdf4a9e0984df2047dd2c9c747bf749d124acdc5570d94cf4b56949b" exitCode=0
Nov 02 15:40:30.525950 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:40:30.525858    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w6-b955c5/auth-test-model-b955c5-kserve-7965dcb68f-75hkb"
Nov 02 15:40:30.525950 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:40:30.525884    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w6-b955c5/auth-test-model-b955c5-kserve-7965dcb68f-75hkb" event={"ID":"61fd91e0-c9e9-4336-b47c-81f9e2799829","Type":"ContainerDied","Data":"c49f2acccdf4a9e0984df2047dd2c9c747bf749d124acdc5570d94cf4b56949b"}
Nov 02 15:40:30.525950 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:40:30.525924    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w6-b955c5/auth-test-model-b955c5-kserve-7965dcb68f-75hkb" event={"ID":"61fd91e0-c9e9-4336-b47c-81f9e2799829","Type":"ContainerDied","Data":"cc0da4b8b6b71f7e6e9dd4313cf201a74b3688e300789d8c2564705fe9ef4814"}
Nov 02 15:40:30.538716 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:40:30.538694    5304 scope.go:122] "RemoveContainer" containerID="3aaad7829e41605fe646a6dd017ea705ffd0d8207c90df5eea30c87ee418c27f"
Nov 02 15:40:30.539005 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:40:30.538981    5304 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3aaad7829e41605fe646a6dd017ea705ffd0d8207c90df5eea30c87ee418c27f\": container with ID starting with 3aaad7829e41605fe646a6dd017ea705ffd0d8207c90df5eea30c87ee418c27f not found: ID does not exist" containerID="3aaad7829e41605fe646a6dd017ea705ffd0d8207c90df5eea30c87ee418c27f"
Nov 02 15:40:30.539102 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:40:30.539012    5304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3aaad7829e41605fe646a6dd017ea705ffd0d8207c90df5eea30c87ee418c27f"} err="failed to get container status \"3aaad7829e41605fe646a6dd017ea705ffd0d8207c90df5eea30c87ee418c27f\": rpc error: code = NotFound desc = could not find container \"3aaad7829e41605fe646a6dd017ea705ffd0d8207c90df5eea30c87ee418c27f\": container with ID starting with 3aaad7829e41605fe646a6dd017ea705ffd0d8207c90df5eea30c87ee418c27f not found: ID does not exist"
Nov 02 15:40:30.539102 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:40:30.539031    5304 scope.go:122] "RemoveContainer" containerID="c49f2acccdf4a9e0984df2047dd2c9c747bf749d124acdc5570d94cf4b56949b"
Nov 02 15:40:30.549359 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:40:30.549338    5304 scope.go:122] "RemoveContainer" containerID="c49f2acccdf4a9e0984df2047dd2c9c747bf749d124acdc5570d94cf4b56949b"
Nov 02 15:40:30.549681 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:40:30.549657    5304 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c49f2acccdf4a9e0984df2047dd2c9c747bf749d124acdc5570d94cf4b56949b\": container with ID starting with c49f2acccdf4a9e0984df2047dd2c9c747bf749d124acdc5570d94cf4b56949b not found: ID does not exist" containerID="c49f2acccdf4a9e0984df2047dd2c9c747bf749d124acdc5570d94cf4b56949b"
Nov 02 15:40:30.549742 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:40:30.549692    5304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c49f2acccdf4a9e0984df2047dd2c9c747bf749d124acdc5570d94cf4b56949b"} err="failed to get container status \"c49f2acccdf4a9e0984df2047dd2c9c747bf749d124acdc5570d94cf4b56949b\": rpc error: code = NotFound desc = could not find container \"c49f2acccdf4a9e0984df2047dd2c9c747bf749d124acdc5570d94cf4b56949b\": container with ID starting with c49f2acccdf4a9e0984df2047dd2c9c747bf749d124acdc5570d94cf4b56949b not found: ID does not exist"
Nov 02 15:40:30.602843 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:40:30.602761    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w1-fda646/auth-test-model-fda646-kserve-568f9ddd9-t6lxf"]
Nov 02 15:40:30.606588 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:40:30.606559    5304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w1-fda646/auth-test-model-fda646-kserve-568f9ddd9-t6lxf"]
Nov 02 15:40:30.660778 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:40:30.660740    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61fd91e0-c9e9-4336-b47c-81f9e2799829-dshm\" (UniqueName: \"kubernetes.io/empty-dir/61fd91e0-c9e9-4336-b47c-81f9e2799829-dshm\") pod \"61fd91e0-c9e9-4336-b47c-81f9e2799829\" (UID: \"61fd91e0-c9e9-4336-b47c-81f9e2799829\") "
Nov 02 15:40:30.660952 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:40:30.660787    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/61fd91e0-c9e9-4336-b47c-81f9e2799829-tls-certs\" (UniqueName: \"kubernetes.io/secret/61fd91e0-c9e9-4336-b47c-81f9e2799829-tls-certs\") pod \"61fd91e0-c9e9-4336-b47c-81f9e2799829\" (UID: \"61fd91e0-c9e9-4336-b47c-81f9e2799829\") "
Nov 02 15:40:30.661011 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:40:30.661000    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61fd91e0-c9e9-4336-b47c-81f9e2799829-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/61fd91e0-c9e9-4336-b47c-81f9e2799829-model-cache\") pod \"61fd91e0-c9e9-4336-b47c-81f9e2799829\" (UID: \"61fd91e0-c9e9-4336-b47c-81f9e2799829\") "
Nov 02 15:40:30.661063 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:40:30.661022    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61fd91e0-c9e9-4336-b47c-81f9e2799829-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/61fd91e0-c9e9-4336-b47c-81f9e2799829-tmp-dir\") pod \"61fd91e0-c9e9-4336-b47c-81f9e2799829\" (UID: \"61fd91e0-c9e9-4336-b47c-81f9e2799829\") "
Nov 02 15:40:30.661063 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:40:30.661052    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61fd91e0-c9e9-4336-b47c-81f9e2799829-home\" (UniqueName: \"kubernetes.io/empty-dir/61fd91e0-c9e9-4336-b47c-81f9e2799829-home\") pod \"61fd91e0-c9e9-4336-b47c-81f9e2799829\" (UID: \"61fd91e0-c9e9-4336-b47c-81f9e2799829\") "
Nov 02 15:40:30.661229 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:40:30.661090    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61fd91e0-c9e9-4336-b47c-81f9e2799829-kube-api-access-xhjwg\" (UniqueName: \"kubernetes.io/projected/61fd91e0-c9e9-4336-b47c-81f9e2799829-kube-api-access-xhjwg\") pod \"61fd91e0-c9e9-4336-b47c-81f9e2799829\" (UID: \"61fd91e0-c9e9-4336-b47c-81f9e2799829\") "
Nov 02 15:40:30.661340 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:40:30.661318    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61fd91e0-c9e9-4336-b47c-81f9e2799829-home" pod "61fd91e0-c9e9-4336-b47c-81f9e2799829" (UID: "61fd91e0-c9e9-4336-b47c-81f9e2799829"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:40:30.661435 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:40:30.661408    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61fd91e0-c9e9-4336-b47c-81f9e2799829-tmp-dir" pod "61fd91e0-c9e9-4336-b47c-81f9e2799829" (UID: "61fd91e0-c9e9-4336-b47c-81f9e2799829"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:40:30.661490 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:40:30.661415    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61fd91e0-c9e9-4336-b47c-81f9e2799829-model-cache" pod "61fd91e0-c9e9-4336-b47c-81f9e2799829" (UID: "61fd91e0-c9e9-4336-b47c-81f9e2799829"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:40:30.661639 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:40:30.661616    5304 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/61fd91e0-c9e9-4336-b47c-81f9e2799829-model-cache\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:40:30.661639 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:40:30.661635    5304 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/61fd91e0-c9e9-4336-b47c-81f9e2799829-tmp-dir\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:40:30.661729 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:40:30.661643    5304 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/61fd91e0-c9e9-4336-b47c-81f9e2799829-home\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:40:30.663146 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:40:30.663113    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61fd91e0-c9e9-4336-b47c-81f9e2799829-dshm" pod "61fd91e0-c9e9-4336-b47c-81f9e2799829" (UID: "61fd91e0-c9e9-4336-b47c-81f9e2799829"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:40:30.663254 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:40:30.663233    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61fd91e0-c9e9-4336-b47c-81f9e2799829-kube-api-access-xhjwg" pod "61fd91e0-c9e9-4336-b47c-81f9e2799829" (UID: "61fd91e0-c9e9-4336-b47c-81f9e2799829"). InnerVolumeSpecName "kube-api-access-xhjwg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:40:30.663409 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:40:30.663390    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/61fd91e0-c9e9-4336-b47c-81f9e2799829-tls-certs" pod "61fd91e0-c9e9-4336-b47c-81f9e2799829" (UID: "61fd91e0-c9e9-4336-b47c-81f9e2799829"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:40:30.762833 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:40:30.762802    5304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xhjwg\" (UniqueName: \"kubernetes.io/projected/61fd91e0-c9e9-4336-b47c-81f9e2799829-kube-api-access-xhjwg\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:40:30.762833 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:40:30.762830    5304 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/61fd91e0-c9e9-4336-b47c-81f9e2799829-dshm\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:40:30.762833 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:40:30.762841    5304 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/61fd91e0-c9e9-4336-b47c-81f9e2799829-tls-certs\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:40:30.889936 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:40:30.889842    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w6-b955c5/auth-test-model-b955c5-kserve-7965dcb68f-75hkb"]
Nov 02 15:40:30.895971 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:40:30.895942    5304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w6-b955c5/auth-test-model-b955c5-kserve-7965dcb68f-75hkb"]
Nov 02 15:40:32.118708 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:40:32.118673    5304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="61fd91e0-c9e9-4336-b47c-81f9e2799829" path="/var/lib/kubelet/pods/61fd91e0-c9e9-4336-b47c-81f9e2799829/volumes"
Nov 02 15:40:32.119196 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:40:32.119181    5304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7e84c4e2-b5e2-4ba8-a6e8-8090a06ac68c" path="/var/lib/kubelet/pods/7e84c4e2-b5e2-4ba8-a6e8-8090a06ac68c/volumes"
Nov 02 15:40:37.295199 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:40:37.295160    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w6-b55da1/auth-test-model-b55da1-kserve-9f6cd75b5-8lljz"]
Nov 02 15:40:38.392635 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:40:38.392598    5304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-f84ad8-openshift-default-779f5d9f49-mstqq"]
Nov 02 15:40:38.393369 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:40:38.393353    5304 cpu_manager.go:414] "containerMap: removing container" podUID="c4240adb-0390-4339-a752-209bdd4b9b50" containerName="main"
Nov 02 15:40:38.393423 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:40:38.393371    5304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c4240adb-0390-4339-a752-209bdd4b9b50" containerName="main"
Nov 02 15:40:38.393423 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:40:38.393385    5304 cpu_manager.go:414] "containerMap: removing container" podUID="7e84c4e2-b5e2-4ba8-a6e8-8090a06ac68c" containerName="main"
Nov 02 15:40:38.393423 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:40:38.393394    5304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e84c4e2-b5e2-4ba8-a6e8-8090a06ac68c" containerName="main"
Nov 02 15:40:38.393522 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:40:38.393427    5304 cpu_manager.go:414] "containerMap: removing container" podUID="61fd91e0-c9e9-4336-b47c-81f9e2799829" containerName="main"
Nov 02 15:40:38.393522 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:40:38.393436    5304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61fd91e0-c9e9-4336-b47c-81f9e2799829" containerName="main"
Nov 02 15:40:38.393593 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:40:38.393547    5304 memory_manager.go:365] "RemoveStaleState removing state" podUID="61fd91e0-c9e9-4336-b47c-81f9e2799829" containerName="main"
Nov 02 15:40:38.393593 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:40:38.393567    5304 memory_manager.go:365] "RemoveStaleState removing state" podUID="c4240adb-0390-4339-a752-209bdd4b9b50" containerName="main"
Nov 02 15:40:38.393593 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:40:38.393575    5304 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e84c4e2-b5e2-4ba8-a6e8-8090a06ac68c" containerName="main"
Nov 02 15:40:38.397330 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:40:38.397306    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-f84ad8-openshift-default-779f5d9f49-mstqq"
Nov 02 15:40:38.411248 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:40:38.411200    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b96fbb28-3d8d-4871-8895-ab78dfab160d-credential-socket\") pod \"e2e-ipp-a-f84ad8-openshift-default-779f5d9f49-mstqq\" (UID: \"b96fbb28-3d8d-4871-8895-ab78dfab160d\") " pod="openshift-ingress/e2e-ipp-a-f84ad8-openshift-default-779f5d9f49-mstqq"
Nov 02 15:40:38.411417 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:40:38.411350    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b96fbb28-3d8d-4871-8895-ab78dfab160d-istio-token\") pod \"e2e-ipp-a-f84ad8-openshift-default-779f5d9f49-mstqq\" (UID: \"b96fbb28-3d8d-4871-8895-ab78dfab160d\") " pod="openshift-ingress/e2e-ipp-a-f84ad8-openshift-default-779f5d9f49-mstqq"
Nov 02 15:40:38.411483 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:40:38.411454    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b96fbb28-3d8d-4871-8895-ab78dfab160d-istio-data\") pod \"e2e-ipp-a-f84ad8-openshift-default-779f5d9f49-mstqq\" (UID: \"b96fbb28-3d8d-4871-8895-ab78dfab160d\") " pod="openshift-ingress/e2e-ipp-a-f84ad8-openshift-default-779f5d9f49-mstqq"
Nov 02 15:40:38.411587 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:40:38.411481    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b96fbb28-3d8d-4871-8895-ab78dfab160d-istiod-ca-cert\") pod \"e2e-ipp-a-f84ad8-openshift-default-779f5d9f49-mstqq\" (UID: \"b96fbb28-3d8d-4871-8895-ab78dfab160d\") " pod="openshift-ingress/e2e-ipp-a-f84ad8-openshift-default-779f5d9f49-mstqq"
Nov 02 15:40:38.411587 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:40:38.411516    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b96fbb28-3d8d-4871-8895-ab78dfab160d-workload-socket\") pod \"e2e-ipp-a-f84ad8-openshift-default-779f5d9f49-mstqq\" (UID: \"b96fbb28-3d8d-4871-8895-ab78dfab160d\") " pod="openshift-ingress/e2e-ipp-a-f84ad8-openshift-default-779f5d9f49-mstqq"
Nov 02 15:40:38.411677 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:40:38.411626    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b96fbb28-3d8d-4871-8895-ab78dfab160d-istio-envoy\") pod \"e2e-ipp-a-f84ad8-openshift-default-779f5d9f49-mstqq\" (UID: \"b96fbb28-3d8d-4871-8895-ab78dfab160d\") " pod="openshift-ingress/e2e-ipp-a-f84ad8-openshift-default-779f5d9f49-mstqq"
Nov 02 15:40:38.411833 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:40:38.411803    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b96fbb28-3d8d-4871-8895-ab78dfab160d-istio-podinfo\") pod \"e2e-ipp-a-f84ad8-openshift-default-779f5d9f49-mstqq\" (UID: \"b96fbb28-3d8d-4871-8895-ab78dfab160d\") " pod="openshift-ingress/e2e-ipp-a-f84ad8-openshift-default-779f5d9f49-mstqq"
Nov 02 15:40:38.411981 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:40:38.411871    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b96fbb28-3d8d-4871-8895-ab78dfab160d-workload-certs\") pod \"e2e-ipp-a-f84ad8-openshift-default-779f5d9f49-mstqq\" (UID: \"b96fbb28-3d8d-4871-8895-ab78dfab160d\") " pod="openshift-ingress/e2e-ipp-a-f84ad8-openshift-default-779f5d9f49-mstqq"
Nov 02 15:40:38.411981 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:40:38.411940    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wx7sn\" (UniqueName: \"kubernetes.io/projected/b96fbb28-3d8d-4871-8895-ab78dfab160d-kube-api-access-wx7sn\") pod \"e2e-ipp-a-f84ad8-openshift-default-779f5d9f49-mstqq\" (UID: \"b96fbb28-3d8d-4871-8895-ab78dfab160d\") " pod="openshift-ingress/e2e-ipp-a-f84ad8-openshift-default-779f5d9f49-mstqq"
Nov 02 15:40:38.419705 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:40:38.419672    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-f84ad8-openshift-default-779f5d9f49-mstqq"]
Nov 02 15:40:38.428721 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:40:38.428690    5304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-f84ad8-openshift-default-dockercfg-hzh8x\""
Nov 02 15:40:38.523435 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:40:38.523396    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b96fbb28-3d8d-4871-8895-ab78dfab160d-istio-podinfo\") pod \"e2e-ipp-a-f84ad8-openshift-default-779f5d9f49-mstqq\" (UID: \"b96fbb28-3d8d-4871-8895-ab78dfab160d\") " pod="openshift-ingress/e2e-ipp-a-f84ad8-openshift-default-779f5d9f49-mstqq"
Nov 02 15:40:38.524342 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:40:38.524314    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b96fbb28-3d8d-4871-8895-ab78dfab160d-workload-certs\") pod \"e2e-ipp-a-f84ad8-openshift-default-779f5d9f49-mstqq\" (UID: \"b96fbb28-3d8d-4871-8895-ab78dfab160d\") " pod="openshift-ingress/e2e-ipp-a-f84ad8-openshift-default-779f5d9f49-mstqq"
Nov 02 15:40:38.524947 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:40:38.524926    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wx7sn\" (UniqueName: \"kubernetes.io/projected/b96fbb28-3d8d-4871-8895-ab78dfab160d-kube-api-access-wx7sn\") pod \"e2e-ipp-a-f84ad8-openshift-default-779f5d9f49-mstqq\" (UID: \"b96fbb28-3d8d-4871-8895-ab78dfab160d\") " pod="openshift-ingress/e2e-ipp-a-f84ad8-openshift-default-779f5d9f49-mstqq"
Nov 02 15:40:38.528090 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:40:38.528066    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b96fbb28-3d8d-4871-8895-ab78dfab160d-credential-socket\") pod \"e2e-ipp-a-f84ad8-openshift-default-779f5d9f49-mstqq\" (UID: \"b96fbb28-3d8d-4871-8895-ab78dfab160d\") " pod="openshift-ingress/e2e-ipp-a-f84ad8-openshift-default-779f5d9f49-mstqq"
Nov 02 15:40:38.528710 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:40:38.528691    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b96fbb28-3d8d-4871-8895-ab78dfab160d-istio-token\") pod \"e2e-ipp-a-f84ad8-openshift-default-779f5d9f49-mstqq\" (UID: \"b96fbb28-3d8d-4871-8895-ab78dfab160d\") " pod="openshift-ingress/e2e-ipp-a-f84ad8-openshift-default-779f5d9f49-mstqq"
Nov 02 15:40:38.529223 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:40:38.529200    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b96fbb28-3d8d-4871-8895-ab78dfab160d-istio-data\") pod \"e2e-ipp-a-f84ad8-openshift-default-779f5d9f49-mstqq\" (UID: \"b96fbb28-3d8d-4871-8895-ab78dfab160d\") " pod="openshift-ingress/e2e-ipp-a-f84ad8-openshift-default-779f5d9f49-mstqq"
Nov 02 15:40:38.529686 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:40:38.529670    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b96fbb28-3d8d-4871-8895-ab78dfab160d-istiod-ca-cert\") pod \"e2e-ipp-a-f84ad8-openshift-default-779f5d9f49-mstqq\" (UID: \"b96fbb28-3d8d-4871-8895-ab78dfab160d\") " pod="openshift-ingress/e2e-ipp-a-f84ad8-openshift-default-779f5d9f49-mstqq"
Nov 02 15:40:38.530534 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:40:38.530513    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b96fbb28-3d8d-4871-8895-ab78dfab160d-workload-socket\") pod \"e2e-ipp-a-f84ad8-openshift-default-779f5d9f49-mstqq\" (UID: \"b96fbb28-3d8d-4871-8895-ab78dfab160d\") " pod="openshift-ingress/e2e-ipp-a-f84ad8-openshift-default-779f5d9f49-mstqq"
Nov 02 15:40:38.532220 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:40:38.532199    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b96fbb28-3d8d-4871-8895-ab78dfab160d-istio-envoy\") pod \"e2e-ipp-a-f84ad8-openshift-default-779f5d9f49-mstqq\" (UID: \"b96fbb28-3d8d-4871-8895-ab78dfab160d\") " pod="openshift-ingress/e2e-ipp-a-f84ad8-openshift-default-779f5d9f49-mstqq"
Nov 02 15:40:38.532805 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:40:38.530438    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b96fbb28-3d8d-4871-8895-ab78dfab160d-istiod-ca-cert\") pod \"e2e-ipp-a-f84ad8-openshift-default-779f5d9f49-mstqq\" (UID: \"b96fbb28-3d8d-4871-8895-ab78dfab160d\") " pod="openshift-ingress/e2e-ipp-a-f84ad8-openshift-default-779f5d9f49-mstqq"
Nov 02 15:40:38.532956 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:40:38.528562    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b96fbb28-3d8d-4871-8895-ab78dfab160d-credential-socket\") pod \"e2e-ipp-a-f84ad8-openshift-default-779f5d9f49-mstqq\" (UID: \"b96fbb28-3d8d-4871-8895-ab78dfab160d\") " pod="openshift-ingress/e2e-ipp-a-f84ad8-openshift-default-779f5d9f49-mstqq"
Nov 02 15:40:38.533064 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:40:38.532137    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b96fbb28-3d8d-4871-8895-ab78dfab160d-workload-socket\") pod \"e2e-ipp-a-f84ad8-openshift-default-779f5d9f49-mstqq\" (UID: \"b96fbb28-3d8d-4871-8895-ab78dfab160d\") " pod="openshift-ingress/e2e-ipp-a-f84ad8-openshift-default-779f5d9f49-mstqq"
Nov 02 15:40:38.533193 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:40:38.529603    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b96fbb28-3d8d-4871-8895-ab78dfab160d-istio-data\") pod \"e2e-ipp-a-f84ad8-openshift-default-779f5d9f49-mstqq\" (UID: \"b96fbb28-3d8d-4871-8895-ab78dfab160d\") " pod="openshift-ingress/e2e-ipp-a-f84ad8-openshift-default-779f5d9f49-mstqq"
Nov 02 15:40:38.533324 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:40:38.524868    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b96fbb28-3d8d-4871-8895-ab78dfab160d-workload-certs\") pod \"e2e-ipp-a-f84ad8-openshift-default-779f5d9f49-mstqq\" (UID: \"b96fbb28-3d8d-4871-8895-ab78dfab160d\") " pod="openshift-ingress/e2e-ipp-a-f84ad8-openshift-default-779f5d9f49-mstqq"
Nov 02 15:40:38.570214 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:40:38.563050    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b96fbb28-3d8d-4871-8895-ab78dfab160d-istio-podinfo\") pod \"e2e-ipp-a-f84ad8-openshift-default-779f5d9f49-mstqq\" (UID: \"b96fbb28-3d8d-4871-8895-ab78dfab160d\") " pod="openshift-ingress/e2e-ipp-a-f84ad8-openshift-default-779f5d9f49-mstqq"
Nov 02 15:40:38.588297 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:40:38.584392    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b96fbb28-3d8d-4871-8895-ab78dfab160d-istio-token\") pod \"e2e-ipp-a-f84ad8-openshift-default-779f5d9f49-mstqq\" (UID: \"b96fbb28-3d8d-4871-8895-ab78dfab160d\") " pod="openshift-ingress/e2e-ipp-a-f84ad8-openshift-default-779f5d9f49-mstqq"
Nov 02 15:40:38.588297 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:40:38.584603    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wx7sn\" (UniqueName: \"kubernetes.io/projected/b96fbb28-3d8d-4871-8895-ab78dfab160d-kube-api-access-wx7sn\") pod \"e2e-ipp-a-f84ad8-openshift-default-779f5d9f49-mstqq\" (UID: \"b96fbb28-3d8d-4871-8895-ab78dfab160d\") " pod="openshift-ingress/e2e-ipp-a-f84ad8-openshift-default-779f5d9f49-mstqq"
Nov 02 15:40:38.588700 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:40:38.588639    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b96fbb28-3d8d-4871-8895-ab78dfab160d-istio-envoy\") pod \"e2e-ipp-a-f84ad8-openshift-default-779f5d9f49-mstqq\" (UID: \"b96fbb28-3d8d-4871-8895-ab78dfab160d\") " pod="openshift-ingress/e2e-ipp-a-f84ad8-openshift-default-779f5d9f49-mstqq"
Nov 02 15:40:38.712343 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:40:38.712269    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-f84ad8-openshift-default-779f5d9f49-mstqq"
Nov 02 15:40:39.050286 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:40:39.050206    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-f84ad8-openshift-default-779f5d9f49-mstqq"]
Nov 02 15:40:39.053418 ip-10-0-135-138 kubenswrapper[5304]: W1102 15:40:39.053392    5304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb96fbb28_3d8d_4871_8895_ab78dfab160d.slice/crio-89b538dae2afd2ad0fb66e7480d6afab632929aa075ff4dfbaa8c4877fb2415d WatchSource:0}: Error finding container 89b538dae2afd2ad0fb66e7480d6afab632929aa075ff4dfbaa8c4877fb2415d: Status 404 returned error can't find the container with id 89b538dae2afd2ad0fb66e7480d6afab632929aa075ff4dfbaa8c4877fb2415d
Nov 02 15:40:39.057172 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:40:39.057109    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:40:39.057427 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:40:39.057226    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:40:39.057427 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:40:39.057273    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:40:39.625422 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:40:39.625388    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f84ad8-openshift-default-779f5d9f49-mstqq" event={"ID":"b96fbb28-3d8d-4871-8895-ab78dfab160d","Type":"ContainerStarted","Data":"8dd5dd14b116e03a35e9b9218038cdacfc7c086554d20c52616e125e269e744c"}
Nov 02 15:40:39.625867 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:40:39.625446    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f84ad8-openshift-default-779f5d9f49-mstqq" event={"ID":"b96fbb28-3d8d-4871-8895-ab78dfab160d","Type":"ContainerStarted","Data":"89b538dae2afd2ad0fb66e7480d6afab632929aa075ff4dfbaa8c4877fb2415d"}
Nov 02 15:40:39.694372 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:40:39.694251    5304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-f84ad8-openshift-default-779f5d9f49-mstqq" podStartSLOduration=1.694234737 podStartE2EDuration="1.694234737s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:40:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:40:39.681918346 +0000 UTC m=+2639.675742756" watchObservedRunningTime="2026-11-02 15:40:39.694234737 +0000 UTC m=+2639.688059146"
Nov 02 15:40:39.712591 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:40:39.712566    5304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-f84ad8-openshift-default-779f5d9f49-mstqq"
Nov 02 15:40:40.719755 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:40:40.719723    5304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-f84ad8-openshift-default-779f5d9f49-mstqq"
Nov 02 15:40:41.649810 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:40:41.649778    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-f84ad8-openshift-default-779f5d9f49-mstqq"
Nov 02 15:40:41.650653 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:40:41.650636    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-f84ad8-openshift-default-779f5d9f49-mstqq"
Nov 02 15:40:44.059315 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:40:44.059275    5304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f84ad8-795988bcd7-tghkt"]
Nov 02 15:40:44.064633 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:40:44.064610    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f84ad8-795988bcd7-tghkt"
Nov 02 15:40:44.081066 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:40:44.081040    5304 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-f84ad8\""
Nov 02 15:40:44.081235 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:40:44.081156    5304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-f84ad8-dockercfg-gcg4p\""
Nov 02 15:40:44.092156 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:40:44.092111    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f84ad8-795988bcd7-tghkt"]
Nov 02 15:40:44.123114 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:40:44.123083    5304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f84ad8-565799f599-4cnmv"]
Nov 02 15:40:44.127036 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:40:44.127014    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-f84ad8-565799f599-4cnmv"
Nov 02 15:40:44.146479 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:40:44.146449    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f84ad8-565799f599-4cnmv"]
Nov 02 15:40:44.263415 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:40:44.263365    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3700f5e-e510-4aec-92cb-e223459f4d0f-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-f84ad8-565799f599-4cnmv\" (UID: \"c3700f5e-e510-4aec-92cb-e223459f4d0f\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f84ad8-565799f599-4cnmv"
Nov 02 15:40:44.263894 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:40:44.263860    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/454ca996-42ce-47d6-96c0-1c9610867def-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-f84ad8-795988bcd7-tghkt\" (UID: \"454ca996-42ce-47d6-96c0-1c9610867def\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f84ad8-795988bcd7-tghkt"
Nov 02 15:40:44.264083 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:40:44.264061    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jhftz\" (UniqueName: \"kubernetes.io/projected/c3700f5e-e510-4aec-92cb-e223459f4d0f-kube-api-access-jhftz\") pod \"payload-processing-e2e-ipp-a-f84ad8-565799f599-4cnmv\" (UID: \"c3700f5e-e510-4aec-92cb-e223459f4d0f\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f84ad8-565799f599-4cnmv"
Nov 02 15:40:44.264301 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:40:44.264272    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x7tvc\" (UniqueName: \"kubernetes.io/projected/454ca996-42ce-47d6-96c0-1c9610867def-kube-api-access-x7tvc\") pod \"payload-pre-processing-e2e-ipp-a-f84ad8-795988bcd7-tghkt\" (UID: \"454ca996-42ce-47d6-96c0-1c9610867def\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f84ad8-795988bcd7-tghkt"
Nov 02 15:40:44.365910 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:40:44.365874    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/454ca996-42ce-47d6-96c0-1c9610867def-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-f84ad8-795988bcd7-tghkt\" (UID: \"454ca996-42ce-47d6-96c0-1c9610867def\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f84ad8-795988bcd7-tghkt"
Nov 02 15:40:44.365910 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:40:44.365917    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jhftz\" (UniqueName: \"kubernetes.io/projected/c3700f5e-e510-4aec-92cb-e223459f4d0f-kube-api-access-jhftz\") pod \"payload-processing-e2e-ipp-a-f84ad8-565799f599-4cnmv\" (UID: \"c3700f5e-e510-4aec-92cb-e223459f4d0f\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f84ad8-565799f599-4cnmv"
Nov 02 15:40:44.366349 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:40:44.365965    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x7tvc\" (UniqueName: \"kubernetes.io/projected/454ca996-42ce-47d6-96c0-1c9610867def-kube-api-access-x7tvc\") pod \"payload-pre-processing-e2e-ipp-a-f84ad8-795988bcd7-tghkt\" (UID: \"454ca996-42ce-47d6-96c0-1c9610867def\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f84ad8-795988bcd7-tghkt"
Nov 02 15:40:44.366349 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:40:44.366072    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3700f5e-e510-4aec-92cb-e223459f4d0f-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-f84ad8-565799f599-4cnmv\" (UID: \"c3700f5e-e510-4aec-92cb-e223459f4d0f\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f84ad8-565799f599-4cnmv"
Nov 02 15:40:44.366753 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:40:44.366729    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/454ca996-42ce-47d6-96c0-1c9610867def-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-f84ad8-795988bcd7-tghkt\" (UID: \"454ca996-42ce-47d6-96c0-1c9610867def\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f84ad8-795988bcd7-tghkt"
Nov 02 15:40:44.366855 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:40:44.366834    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3700f5e-e510-4aec-92cb-e223459f4d0f-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-f84ad8-565799f599-4cnmv\" (UID: \"c3700f5e-e510-4aec-92cb-e223459f4d0f\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f84ad8-565799f599-4cnmv"
Nov 02 15:40:44.391487 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:40:44.391449    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x7tvc\" (UniqueName: \"kubernetes.io/projected/454ca996-42ce-47d6-96c0-1c9610867def-kube-api-access-x7tvc\") pod \"payload-pre-processing-e2e-ipp-a-f84ad8-795988bcd7-tghkt\" (UID: \"454ca996-42ce-47d6-96c0-1c9610867def\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f84ad8-795988bcd7-tghkt"
Nov 02 15:40:44.391807 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:40:44.391781    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jhftz\" (UniqueName: \"kubernetes.io/projected/c3700f5e-e510-4aec-92cb-e223459f4d0f-kube-api-access-jhftz\") pod \"payload-processing-e2e-ipp-a-f84ad8-565799f599-4cnmv\" (UID: \"c3700f5e-e510-4aec-92cb-e223459f4d0f\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f84ad8-565799f599-4cnmv"
Nov 02 15:40:44.441267 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:40:44.441231    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-f84ad8-565799f599-4cnmv"
Nov 02 15:40:44.678357 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:40:44.678266    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f84ad8-795988bcd7-tghkt"
Nov 02 15:40:44.750375 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:40:44.750346    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f84ad8-565799f599-4cnmv"]
Nov 02 15:40:44.759255 ip-10-0-135-138 kubenswrapper[5304]: W1102 15:40:44.759214    5304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3700f5e_e510_4aec_92cb_e223459f4d0f.slice/crio-9d450d412badabcad4dec72b1bbe18989ce3b12c008d50f6a088b0a345529c36 WatchSource:0}: Error finding container 9d450d412badabcad4dec72b1bbe18989ce3b12c008d50f6a088b0a345529c36: Status 404 returned error can't find the container with id 9d450d412badabcad4dec72b1bbe18989ce3b12c008d50f6a088b0a345529c36
Nov 02 15:40:44.998526 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:40:44.998355    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f84ad8-795988bcd7-tghkt"]
Nov 02 15:40:45.001672 ip-10-0-135-138 kubenswrapper[5304]: W1102 15:40:45.001640    5304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod454ca996_42ce_47d6_96c0_1c9610867def.slice/crio-8ef0907add5efc1ea93ceb32d323172846cda01ff848229500beb75e76f2384c WatchSource:0}: Error finding container 8ef0907add5efc1ea93ceb32d323172846cda01ff848229500beb75e76f2384c: Status 404 returned error can't find the container with id 8ef0907add5efc1ea93ceb32d323172846cda01ff848229500beb75e76f2384c
Nov 02 15:40:45.677052 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:40:45.677016    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f84ad8-795988bcd7-tghkt" event={"ID":"454ca996-42ce-47d6-96c0-1c9610867def","Type":"ContainerStarted","Data":"dac19558ee7d019fab660b715896aac520ad6437156a67fa751e77404e7dc4ee"}
Nov 02 15:40:45.677573 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:40:45.677079    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f84ad8-795988bcd7-tghkt" event={"ID":"454ca996-42ce-47d6-96c0-1c9610867def","Type":"ContainerStarted","Data":"8ef0907add5efc1ea93ceb32d323172846cda01ff848229500beb75e76f2384c"}
Nov 02 15:40:45.677573 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:40:45.677223    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f84ad8-795988bcd7-tghkt"
Nov 02 15:40:45.678628 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:40:45.678603    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-f84ad8-565799f599-4cnmv" event={"ID":"c3700f5e-e510-4aec-92cb-e223459f4d0f","Type":"ContainerStarted","Data":"8d556e318b0ffeea2d755f31279d27f0b5f2ff2081bb1603f392fee7bc7f995b"}
Nov 02 15:40:45.678763 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:40:45.678643    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-f84ad8-565799f599-4cnmv" event={"ID":"c3700f5e-e510-4aec-92cb-e223459f4d0f","Type":"ContainerStarted","Data":"9d450d412badabcad4dec72b1bbe18989ce3b12c008d50f6a088b0a345529c36"}
Nov 02 15:40:45.678763 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:40:45.678694    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-f84ad8-565799f599-4cnmv"
Nov 02 15:40:45.743374 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:40:45.743319    5304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f84ad8-795988bcd7-tghkt" podStartSLOduration=1.743306341 podStartE2EDuration="1.743306341s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:40:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:40:45.739813719 +0000 UTC m=+2645.733638128" watchObservedRunningTime="2026-11-02 15:40:45.743306341 +0000 UTC m=+2645.737130789"
Nov 02 15:40:45.806829 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:40:45.806767    5304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-f84ad8-565799f599-4cnmv" podStartSLOduration=1.806753029 podStartE2EDuration="1.806753029s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:40:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:40:45.801987608 +0000 UTC m=+2645.795812018" watchObservedRunningTime="2026-11-02 15:40:45.806753029 +0000 UTC m=+2645.800577489"
Nov 02 15:40:52.322291 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:40:52.322249    5304 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w6-b55da1/auth-test-model-b55da1-kserve-9f6cd75b5-8lljz" podUID="860edf42-9efd-4d01-aaf3-78bbd87cc2f9" containerName="main" containerID="cri-o://3dad92941bbe1f68a7b5b2e5994fae11ae109b4e86f6d10472a30fe77e5e1ff2" gracePeriod=45
Nov 02 15:40:52.643350 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:40:52.643323    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w6-b55da1/auth-test-model-b55da1-kserve-9f6cd75b5-8lljz"
Nov 02 15:40:52.677970 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:40:52.677939    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/860edf42-9efd-4d01-aaf3-78bbd87cc2f9-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/860edf42-9efd-4d01-aaf3-78bbd87cc2f9-tmp-dir\") pod \"860edf42-9efd-4d01-aaf3-78bbd87cc2f9\" (UID: \"860edf42-9efd-4d01-aaf3-78bbd87cc2f9\") "
Nov 02 15:40:52.678160 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:40:52.678091    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/860edf42-9efd-4d01-aaf3-78bbd87cc2f9-dshm\" (UniqueName: \"kubernetes.io/empty-dir/860edf42-9efd-4d01-aaf3-78bbd87cc2f9-dshm\") pod \"860edf42-9efd-4d01-aaf3-78bbd87cc2f9\" (UID: \"860edf42-9efd-4d01-aaf3-78bbd87cc2f9\") "
Nov 02 15:40:52.678160 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:40:52.678118    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/860edf42-9efd-4d01-aaf3-78bbd87cc2f9-home\" (UniqueName: \"kubernetes.io/empty-dir/860edf42-9efd-4d01-aaf3-78bbd87cc2f9-home\") pod \"860edf42-9efd-4d01-aaf3-78bbd87cc2f9\" (UID: \"860edf42-9efd-4d01-aaf3-78bbd87cc2f9\") "
Nov 02 15:40:52.678303 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:40:52.678177    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/860edf42-9efd-4d01-aaf3-78bbd87cc2f9-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/860edf42-9efd-4d01-aaf3-78bbd87cc2f9-model-cache\") pod \"860edf42-9efd-4d01-aaf3-78bbd87cc2f9\" (UID: \"860edf42-9efd-4d01-aaf3-78bbd87cc2f9\") "
Nov 02 15:40:52.678303 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:40:52.678274    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/860edf42-9efd-4d01-aaf3-78bbd87cc2f9-kube-api-access-84zg5\" (UniqueName: \"kubernetes.io/projected/860edf42-9efd-4d01-aaf3-78bbd87cc2f9-kube-api-access-84zg5\") pod \"860edf42-9efd-4d01-aaf3-78bbd87cc2f9\" (UID: \"860edf42-9efd-4d01-aaf3-78bbd87cc2f9\") "
Nov 02 15:40:52.678425 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:40:52.678316    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/860edf42-9efd-4d01-aaf3-78bbd87cc2f9-tls-certs\" (UniqueName: \"kubernetes.io/secret/860edf42-9efd-4d01-aaf3-78bbd87cc2f9-tls-certs\") pod \"860edf42-9efd-4d01-aaf3-78bbd87cc2f9\" (UID: \"860edf42-9efd-4d01-aaf3-78bbd87cc2f9\") "
Nov 02 15:40:52.678518 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:40:52.678466    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/860edf42-9efd-4d01-aaf3-78bbd87cc2f9-model-cache" pod "860edf42-9efd-4d01-aaf3-78bbd87cc2f9" (UID: "860edf42-9efd-4d01-aaf3-78bbd87cc2f9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:40:52.678581 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:40:52.678559    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/860edf42-9efd-4d01-aaf3-78bbd87cc2f9-tmp-dir" pod "860edf42-9efd-4d01-aaf3-78bbd87cc2f9" (UID: "860edf42-9efd-4d01-aaf3-78bbd87cc2f9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:40:52.678683 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:40:52.678657    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/860edf42-9efd-4d01-aaf3-78bbd87cc2f9-home" pod "860edf42-9efd-4d01-aaf3-78bbd87cc2f9" (UID: "860edf42-9efd-4d01-aaf3-78bbd87cc2f9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:40:52.679115 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:40:52.679093    5304 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/860edf42-9efd-4d01-aaf3-78bbd87cc2f9-home\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:40:52.679234 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:40:52.679117    5304 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/860edf42-9efd-4d01-aaf3-78bbd87cc2f9-model-cache\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:40:52.679234 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:40:52.679145    5304 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/860edf42-9efd-4d01-aaf3-78bbd87cc2f9-tmp-dir\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:40:52.680875 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:40:52.680854    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/860edf42-9efd-4d01-aaf3-78bbd87cc2f9-dshm" pod "860edf42-9efd-4d01-aaf3-78bbd87cc2f9" (UID: "860edf42-9efd-4d01-aaf3-78bbd87cc2f9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:40:52.680962 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:40:52.680901    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/860edf42-9efd-4d01-aaf3-78bbd87cc2f9-tls-certs" pod "860edf42-9efd-4d01-aaf3-78bbd87cc2f9" (UID: "860edf42-9efd-4d01-aaf3-78bbd87cc2f9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:40:52.681022 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:40:52.680986    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/860edf42-9efd-4d01-aaf3-78bbd87cc2f9-kube-api-access-84zg5" pod "860edf42-9efd-4d01-aaf3-78bbd87cc2f9" (UID: "860edf42-9efd-4d01-aaf3-78bbd87cc2f9"). InnerVolumeSpecName "kube-api-access-84zg5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:40:52.722801 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:40:52.722764    5304 generic.go:356] "Generic (PLEG): container finished" podID="860edf42-9efd-4d01-aaf3-78bbd87cc2f9" containerID="3dad92941bbe1f68a7b5b2e5994fae11ae109b4e86f6d10472a30fe77e5e1ff2" exitCode=0
Nov 02 15:40:52.722962 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:40:52.722811    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w6-b55da1/auth-test-model-b55da1-kserve-9f6cd75b5-8lljz"
Nov 02 15:40:52.722962 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:40:52.722849    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w6-b55da1/auth-test-model-b55da1-kserve-9f6cd75b5-8lljz" event={"ID":"860edf42-9efd-4d01-aaf3-78bbd87cc2f9","Type":"ContainerDied","Data":"3dad92941bbe1f68a7b5b2e5994fae11ae109b4e86f6d10472a30fe77e5e1ff2"}
Nov 02 15:40:52.722962 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:40:52.722929    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w6-b55da1/auth-test-model-b55da1-kserve-9f6cd75b5-8lljz" event={"ID":"860edf42-9efd-4d01-aaf3-78bbd87cc2f9","Type":"ContainerDied","Data":"e77c8a42554b98a42ca05f8411c0d2cc850f1a36c5baa7d341b6b3449975868c"}
Nov 02 15:40:52.722962 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:40:52.722948    5304 scope.go:122] "RemoveContainer" containerID="3dad92941bbe1f68a7b5b2e5994fae11ae109b4e86f6d10472a30fe77e5e1ff2"
Nov 02 15:40:52.735858 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:40:52.735810    5304 scope.go:122] "RemoveContainer" containerID="3dad92941bbe1f68a7b5b2e5994fae11ae109b4e86f6d10472a30fe77e5e1ff2"
Nov 02 15:40:52.736299 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:40:52.736271    5304 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3dad92941bbe1f68a7b5b2e5994fae11ae109b4e86f6d10472a30fe77e5e1ff2\": container with ID starting with 3dad92941bbe1f68a7b5b2e5994fae11ae109b4e86f6d10472a30fe77e5e1ff2 not found: ID does not exist" containerID="3dad92941bbe1f68a7b5b2e5994fae11ae109b4e86f6d10472a30fe77e5e1ff2"
Nov 02 15:40:52.736426 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:40:52.736310    5304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3dad92941bbe1f68a7b5b2e5994fae11ae109b4e86f6d10472a30fe77e5e1ff2"} err="failed to get container status \"3dad92941bbe1f68a7b5b2e5994fae11ae109b4e86f6d10472a30fe77e5e1ff2\": rpc error: code = NotFound desc = could not find container \"3dad92941bbe1f68a7b5b2e5994fae11ae109b4e86f6d10472a30fe77e5e1ff2\": container with ID starting with 3dad92941bbe1f68a7b5b2e5994fae11ae109b4e86f6d10472a30fe77e5e1ff2 not found: ID does not exist"
Nov 02 15:40:52.783892 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:40:52.775647    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w6-b55da1/auth-test-model-b55da1-kserve-9f6cd75b5-8lljz"]
Nov 02 15:40:52.783892 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:40:52.783285    5304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w6-b55da1/auth-test-model-b55da1-kserve-9f6cd75b5-8lljz"]
Nov 02 15:40:52.784396 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:40:52.784366    5304 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/860edf42-9efd-4d01-aaf3-78bbd87cc2f9-dshm\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:40:52.784396 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:40:52.784397    5304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-84zg5\" (UniqueName: \"kubernetes.io/projected/860edf42-9efd-4d01-aaf3-78bbd87cc2f9-kube-api-access-84zg5\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:40:52.784614 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:40:52.784412    5304 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/860edf42-9efd-4d01-aaf3-78bbd87cc2f9-tls-certs\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:40:54.144991 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:40:54.144957    5304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="860edf42-9efd-4d01-aaf3-78bbd87cc2f9" path="/var/lib/kubelet/pods/860edf42-9efd-4d01-aaf3-78bbd87cc2f9/volumes"
Nov 02 15:40:54.798808 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:40:54.798775    5304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k6fjk" containerName="extractor"
Nov 02 15:40:56.686853 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:40:56.686815    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f84ad8-795988bcd7-tghkt"
Nov 02 15:40:56.687336 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:40:56.686983    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-f84ad8-565799f599-4cnmv"
Nov 02 15:41:24.803518 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:41:24.803480    5304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k6fjk" containerName="extractor"
Nov 02 15:41:33.242323 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:41:33.242274    5304 patch_prober.go:29] interesting pod/e2e-ipp-a-9cfd1f-openshift-default-6fbb7b57d9-hh28t container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.159:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 15:41:33.242720 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:41:33.242360    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-9cfd1f-openshift-default-6fbb7b57d9-hh28t" podUID="d665d588-97c3-4797-be57-c426b3d905ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.159:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:41:36.027035 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:41:36.026999    5304 generic.go:356] "Generic (PLEG): container finished" podID="d665d588-97c3-4797-be57-c426b3d905ff" containerID="844d78510fb78a43c3c7c69d7bafedfd7f55e584f99d15174806ac946b6c56a3" exitCode=0
Nov 02 15:41:36.027550 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:41:36.027074    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-9cfd1f-openshift-default-6fbb7b57d9-hh28t" event={"ID":"d665d588-97c3-4797-be57-c426b3d905ff","Type":"ContainerDied","Data":"844d78510fb78a43c3c7c69d7bafedfd7f55e584f99d15174806ac946b6c56a3"}
Nov 02 15:41:36.027550 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:41:36.027469    5304 scope.go:122] "RemoveContainer" containerID="844d78510fb78a43c3c7c69d7bafedfd7f55e584f99d15174806ac946b6c56a3"
Nov 02 15:41:36.029068 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:41:36.029030    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:41:36.029227 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:41:36.029119    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:41:36.029227 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:41:36.029190    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:41:37.056331 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:41:37.056295    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-9cfd1f-openshift-default-6fbb7b57d9-hh28t" event={"ID":"d665d588-97c3-4797-be57-c426b3d905ff","Type":"ContainerStarted","Data":"c775141644658fa89c11450fa81952fd56da0b264110c7fc3655b74f6763e0dc"}
Nov 02 15:41:37.169854 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:41:37.169821    5304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-9cfd1f-openshift-default-6fbb7b57d9-hh28t"
Nov 02 15:41:37.170036 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:41:37.169980    5304 patch_prober.go:29] interesting pod/e2e-ipp-a-9cfd1f-openshift-default-6fbb7b57d9-hh28t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" start-of-body=
Nov 02 15:41:37.170082 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:41:37.170028    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-9cfd1f-openshift-default-6fbb7b57d9-hh28t" podUID="d665d588-97c3-4797-be57-c426b3d905ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused"
Nov 02 15:41:38.170659 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:41:38.170619    5304 patch_prober.go:29] interesting pod/e2e-ipp-a-9cfd1f-openshift-default-6fbb7b57d9-hh28t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" start-of-body=
Nov 02 15:41:38.171055 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:41:38.170683    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-9cfd1f-openshift-default-6fbb7b57d9-hh28t" podUID="d665d588-97c3-4797-be57-c426b3d905ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused"
Nov 02 15:41:39.170312 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:41:39.170279    5304 patch_prober.go:29] interesting pod/e2e-ipp-a-9cfd1f-openshift-default-6fbb7b57d9-hh28t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" start-of-body=
Nov 02 15:41:39.170509 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:41:39.170338    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-9cfd1f-openshift-default-6fbb7b57d9-hh28t" podUID="d665d588-97c3-4797-be57-c426b3d905ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused"
Nov 02 15:41:40.170478 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:41:40.170445    5304 patch_prober.go:29] interesting pod/e2e-ipp-a-9cfd1f-openshift-default-6fbb7b57d9-hh28t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" start-of-body=
Nov 02 15:41:40.170892 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:41:40.170499    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-9cfd1f-openshift-default-6fbb7b57d9-hh28t" podUID="d665d588-97c3-4797-be57-c426b3d905ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused"
Nov 02 15:41:41.170585 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:41:41.170552    5304 patch_prober.go:29] interesting pod/e2e-ipp-a-9cfd1f-openshift-default-6fbb7b57d9-hh28t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" start-of-body=
Nov 02 15:41:41.171092 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:41:41.170610    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-9cfd1f-openshift-default-6fbb7b57d9-hh28t" podUID="d665d588-97c3-4797-be57-c426b3d905ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused"
Nov 02 15:41:42.170216 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:41:42.170172    5304 patch_prober.go:29] interesting pod/e2e-ipp-a-9cfd1f-openshift-default-6fbb7b57d9-hh28t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" start-of-body=
Nov 02 15:41:42.170454 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:41:42.170245    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-9cfd1f-openshift-default-6fbb7b57d9-hh28t" podUID="d665d588-97c3-4797-be57-c426b3d905ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused"
Nov 02 15:41:43.174856 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:41:43.174822    5304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-9cfd1f-openshift-default-6fbb7b57d9-hh28t"
Nov 02 15:41:43.175272 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:41:43.175228    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-9cfd1f-openshift-default-6fbb7b57d9-hh28t"
Nov 02 15:41:43.175962 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:41:43.175941    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-9cfd1f-openshift-default-6fbb7b57d9-hh28t"
Nov 02 15:41:44.800149 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:41:44.799180    5304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-9cfd1f-9bf597b4c-6fbz9"]
Nov 02 15:41:44.800704 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:41:44.800319    5304 cpu_manager.go:414] "containerMap: removing container" podUID="860edf42-9efd-4d01-aaf3-78bbd87cc2f9" containerName="main"
Nov 02 15:41:44.800704 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:41:44.800340    5304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="860edf42-9efd-4d01-aaf3-78bbd87cc2f9" containerName="main"
Nov 02 15:41:44.800704 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:41:44.800619    5304 memory_manager.go:365] "RemoveStaleState removing state" podUID="860edf42-9efd-4d01-aaf3-78bbd87cc2f9" containerName="main"
Nov 02 15:41:44.807156 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:41:44.805345    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9cfd1f-9bf597b4c-6fbz9"
Nov 02 15:41:44.822094 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:41:44.822071    5304 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-9cfd1f\""
Nov 02 15:41:44.837796 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:41:44.837757    5304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-9cfd1f-dockercfg-bmqkm\""
Nov 02 15:41:44.838212 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:41:44.838182    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-9cfd1f-9bf597b4c-6fbz9"]
Nov 02 15:41:44.863979 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:41:44.863941    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vw4z6\" (UniqueName: \"kubernetes.io/projected/d8479a68-3948-4660-a797-8aca350dcd67-kube-api-access-vw4z6\") pod \"payload-pre-processing-e2e-ipp-a-9cfd1f-9bf597b4c-6fbz9\" (UID: \"d8479a68-3948-4660-a797-8aca350dcd67\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9cfd1f-9bf597b4c-6fbz9"
Nov 02 15:41:44.874452 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:41:44.874115    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d8479a68-3948-4660-a797-8aca350dcd67-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-9cfd1f-9bf597b4c-6fbz9\" (UID: \"d8479a68-3948-4660-a797-8aca350dcd67\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9cfd1f-9bf597b4c-6fbz9"
Nov 02 15:41:44.969454 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:41:44.969415    5304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-9cfd1f-77499c8864-p9ccn"]
Nov 02 15:41:44.972814 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:41:44.972789    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-9cfd1f-77499c8864-p9ccn"
Nov 02 15:41:44.982226 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:41:44.982189    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d8479a68-3948-4660-a797-8aca350dcd67-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-9cfd1f-9bf597b4c-6fbz9\" (UID: \"d8479a68-3948-4660-a797-8aca350dcd67\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9cfd1f-9bf597b4c-6fbz9"
Nov 02 15:41:44.982403 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:41:44.982318    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vw4z6\" (UniqueName: \"kubernetes.io/projected/d8479a68-3948-4660-a797-8aca350dcd67-kube-api-access-vw4z6\") pod \"payload-pre-processing-e2e-ipp-a-9cfd1f-9bf597b4c-6fbz9\" (UID: \"d8479a68-3948-4660-a797-8aca350dcd67\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9cfd1f-9bf597b4c-6fbz9"
Nov 02 15:41:44.983358 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:41:44.983331    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d8479a68-3948-4660-a797-8aca350dcd67-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-9cfd1f-9bf597b4c-6fbz9\" (UID: \"d8479a68-3948-4660-a797-8aca350dcd67\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9cfd1f-9bf597b4c-6fbz9"
Nov 02 15:41:44.993686 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:41:44.993654    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-9cfd1f-77499c8864-p9ccn"]
Nov 02 15:41:45.006978 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:41:45.006947    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vw4z6\" (UniqueName: \"kubernetes.io/projected/d8479a68-3948-4660-a797-8aca350dcd67-kube-api-access-vw4z6\") pod \"payload-pre-processing-e2e-ipp-a-9cfd1f-9bf597b4c-6fbz9\" (UID: \"d8479a68-3948-4660-a797-8aca350dcd67\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9cfd1f-9bf597b4c-6fbz9"
Nov 02 15:41:45.084856 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:41:45.084814    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7ffnt\" (UniqueName: \"kubernetes.io/projected/c85057ae-7026-4b45-8ce9-2568c3027565-kube-api-access-7ffnt\") pod \"payload-processing-e2e-ipp-a-9cfd1f-77499c8864-p9ccn\" (UID: \"c85057ae-7026-4b45-8ce9-2568c3027565\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-9cfd1f-77499c8864-p9ccn"
Nov 02 15:41:45.085028 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:41:45.085010    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c85057ae-7026-4b45-8ce9-2568c3027565-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-9cfd1f-77499c8864-p9ccn\" (UID: \"c85057ae-7026-4b45-8ce9-2568c3027565\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-9cfd1f-77499c8864-p9ccn"
Nov 02 15:41:45.125772 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:41:45.125743    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9cfd1f-9bf597b4c-6fbz9"
Nov 02 15:41:45.187161 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:41:45.187093    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7ffnt\" (UniqueName: \"kubernetes.io/projected/c85057ae-7026-4b45-8ce9-2568c3027565-kube-api-access-7ffnt\") pod \"payload-processing-e2e-ipp-a-9cfd1f-77499c8864-p9ccn\" (UID: \"c85057ae-7026-4b45-8ce9-2568c3027565\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-9cfd1f-77499c8864-p9ccn"
Nov 02 15:41:45.187315 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:41:45.187297    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c85057ae-7026-4b45-8ce9-2568c3027565-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-9cfd1f-77499c8864-p9ccn\" (UID: \"c85057ae-7026-4b45-8ce9-2568c3027565\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-9cfd1f-77499c8864-p9ccn"
Nov 02 15:41:45.191266 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:41:45.188544    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c85057ae-7026-4b45-8ce9-2568c3027565-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-9cfd1f-77499c8864-p9ccn\" (UID: \"c85057ae-7026-4b45-8ce9-2568c3027565\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-9cfd1f-77499c8864-p9ccn"
Nov 02 15:41:45.213070 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:41:45.213034    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7ffnt\" (UniqueName: \"kubernetes.io/projected/c85057ae-7026-4b45-8ce9-2568c3027565-kube-api-access-7ffnt\") pod \"payload-processing-e2e-ipp-a-9cfd1f-77499c8864-p9ccn\" (UID: \"c85057ae-7026-4b45-8ce9-2568c3027565\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-9cfd1f-77499c8864-p9ccn"
Nov 02 15:41:45.297059 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:41:45.297021    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-9cfd1f-77499c8864-p9ccn"
Nov 02 15:41:45.435024 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:41:45.434999    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-9cfd1f-9bf597b4c-6fbz9"]
Nov 02 15:41:45.437860 ip-10-0-135-138 kubenswrapper[5304]: W1102 15:41:45.437817    5304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8479a68_3948_4660_a797_8aca350dcd67.slice/crio-5536dc9304299982ecba1927285b9143b9d58102995b9a8b3f8940726a5c6b2a WatchSource:0}: Error finding container 5536dc9304299982ecba1927285b9143b9d58102995b9a8b3f8940726a5c6b2a: Status 404 returned error can't find the container with id 5536dc9304299982ecba1927285b9143b9d58102995b9a8b3f8940726a5c6b2a
Nov 02 15:41:45.439855 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:41:45.439838    5304 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 02 15:41:45.602096 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:41:45.601894    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-9cfd1f-77499c8864-p9ccn"]
Nov 02 15:41:45.605178 ip-10-0-135-138 kubenswrapper[5304]: W1102 15:41:45.605151    5304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc85057ae_7026_4b45_8ce9_2568c3027565.slice/crio-42a582dffc6f3e8af274e5901bba56b6a8ae00af6ce32b5c5ac891c5154ffa4d WatchSource:0}: Error finding container 42a582dffc6f3e8af274e5901bba56b6a8ae00af6ce32b5c5ac891c5154ffa4d: Status 404 returned error can't find the container with id 42a582dffc6f3e8af274e5901bba56b6a8ae00af6ce32b5c5ac891c5154ffa4d
Nov 02 15:41:46.131424 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:41:46.131378    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-9cfd1f-77499c8864-p9ccn" event={"ID":"c85057ae-7026-4b45-8ce9-2568c3027565","Type":"ContainerStarted","Data":"42a582dffc6f3e8af274e5901bba56b6a8ae00af6ce32b5c5ac891c5154ffa4d"}
Nov 02 15:41:46.134074 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:41:46.134045    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9cfd1f-9bf597b4c-6fbz9" event={"ID":"d8479a68-3948-4660-a797-8aca350dcd67","Type":"ContainerStarted","Data":"5536dc9304299982ecba1927285b9143b9d58102995b9a8b3f8940726a5c6b2a"}
Nov 02 15:41:48.172570 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:41:48.172534    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-9cfd1f-77499c8864-p9ccn" event={"ID":"c85057ae-7026-4b45-8ce9-2568c3027565","Type":"ContainerStarted","Data":"c5133aa6e14ffbf1455468aa960943e2b2647f981ad4a90b3f7ab890eb10b079"}
Nov 02 15:41:48.173151 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:41:48.172606    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-9cfd1f-77499c8864-p9ccn"
Nov 02 15:41:48.174339 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:41:48.174308    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9cfd1f-9bf597b4c-6fbz9" event={"ID":"d8479a68-3948-4660-a797-8aca350dcd67","Type":"ContainerStarted","Data":"e11a96e2f8769c7790258581d5fdd1625e9ad831d7762a360956df550b3b22df"}
Nov 02 15:41:48.174480 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:41:48.174367    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9cfd1f-9bf597b4c-6fbz9"
Nov 02 15:41:48.215559 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:41:48.215407    5304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-9cfd1f-77499c8864-p9ccn" podStartSLOduration=2.419781697 podStartE2EDuration="4.215387476s" totalImagesPullingTime="1.795605779s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:41:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:41:48.213765326 +0000 UTC m=+2708.207589735" watchObservedRunningTime="2026-11-02 15:41:48.215387476 +0000 UTC m=+2708.209211886"
Nov 02 15:41:48.271007 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:41:48.270950    5304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9cfd1f-9bf597b4c-6fbz9" podStartSLOduration=2.34628643 podStartE2EDuration="4.270931376s" totalImagesPullingTime="1.924644946s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:41:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:41:48.268917521 +0000 UTC m=+2708.262741927" watchObservedRunningTime="2026-11-02 15:41:48.270931376 +0000 UTC m=+2708.264755881"
Nov 02 15:41:54.800143 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:41:54.800047    5304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k6fjk" containerName="extractor"
Nov 02 15:41:57.005114 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:41:57.005067    5304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w2-797488/sub-test-model-797488-kserve-79764959f-k4scd"]
Nov 02 15:41:57.012727 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:41:57.012694    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-797488/sub-test-model-797488-kserve-79764959f-k4scd"
Nov 02 15:41:57.035783 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:41:57.035751    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-797488/sub-test-model-797488-kserve-79764959f-k4scd"]
Nov 02 15:41:57.043955 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:41:57.043924    5304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-797488\"/\"default-dockercfg-w8bnd\""
Nov 02 15:41:57.044579 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:41:57.044366    5304 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-797488\"/\"kube-root-ca.crt\""
Nov 02 15:41:57.044757 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:41:57.044413    5304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-797488\"/\"sub-test-model-797488-kserve-self-signed-certs\""
Nov 02 15:41:57.044840 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:41:57.044503    5304 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-797488\"/\"openshift-service-ca.crt\""
Nov 02 15:41:57.119039 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:41:57.119005    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d6e474d-c5d3-4c69-b12f-993ff23910c1-tls-certs\") pod \"sub-test-model-797488-kserve-79764959f-k4scd\" (UID: \"2d6e474d-c5d3-4c69-b12f-993ff23910c1\") " pod="ai-tenant-e2e-shared-b-w2-797488/sub-test-model-797488-kserve-79764959f-k4scd"
Nov 02 15:41:57.119238 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:41:57.119186    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2dj2p\" (UniqueName: \"kubernetes.io/projected/2d6e474d-c5d3-4c69-b12f-993ff23910c1-kube-api-access-2dj2p\") pod \"sub-test-model-797488-kserve-79764959f-k4scd\" (UID: \"2d6e474d-c5d3-4c69-b12f-993ff23910c1\") " pod="ai-tenant-e2e-shared-b-w2-797488/sub-test-model-797488-kserve-79764959f-k4scd"
Nov 02 15:41:57.119238 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:41:57.119226    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2d6e474d-c5d3-4c69-b12f-993ff23910c1-model-cache\") pod \"sub-test-model-797488-kserve-79764959f-k4scd\" (UID: \"2d6e474d-c5d3-4c69-b12f-993ff23910c1\") " pod="ai-tenant-e2e-shared-b-w2-797488/sub-test-model-797488-kserve-79764959f-k4scd"
Nov 02 15:41:57.119361 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:41:57.119247    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2d6e474d-c5d3-4c69-b12f-993ff23910c1-dshm\") pod \"sub-test-model-797488-kserve-79764959f-k4scd\" (UID: \"2d6e474d-c5d3-4c69-b12f-993ff23910c1\") " pod="ai-tenant-e2e-shared-b-w2-797488/sub-test-model-797488-kserve-79764959f-k4scd"
Nov 02 15:41:57.119433 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:41:57.119409    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2d6e474d-c5d3-4c69-b12f-993ff23910c1-home\") pod \"sub-test-model-797488-kserve-79764959f-k4scd\" (UID: \"2d6e474d-c5d3-4c69-b12f-993ff23910c1\") " pod="ai-tenant-e2e-shared-b-w2-797488/sub-test-model-797488-kserve-79764959f-k4scd"
Nov 02 15:41:57.119552 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:41:57.119509    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d6e474d-c5d3-4c69-b12f-993ff23910c1-tmp-dir\") pod \"sub-test-model-797488-kserve-79764959f-k4scd\" (UID: \"2d6e474d-c5d3-4c69-b12f-993ff23910c1\") " pod="ai-tenant-e2e-shared-b-w2-797488/sub-test-model-797488-kserve-79764959f-k4scd"
Nov 02 15:41:57.221353 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:41:57.221313    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d6e474d-c5d3-4c69-b12f-993ff23910c1-tls-certs\") pod \"sub-test-model-797488-kserve-79764959f-k4scd\" (UID: \"2d6e474d-c5d3-4c69-b12f-993ff23910c1\") " pod="ai-tenant-e2e-shared-b-w2-797488/sub-test-model-797488-kserve-79764959f-k4scd"
Nov 02 15:41:57.221587 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:41:57.221457    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2dj2p\" (UniqueName: \"kubernetes.io/projected/2d6e474d-c5d3-4c69-b12f-993ff23910c1-kube-api-access-2dj2p\") pod \"sub-test-model-797488-kserve-79764959f-k4scd\" (UID: \"2d6e474d-c5d3-4c69-b12f-993ff23910c1\") " pod="ai-tenant-e2e-shared-b-w2-797488/sub-test-model-797488-kserve-79764959f-k4scd"
Nov 02 15:41:57.221587 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:41:57.221486    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2d6e474d-c5d3-4c69-b12f-993ff23910c1-model-cache\") pod \"sub-test-model-797488-kserve-79764959f-k4scd\" (UID: \"2d6e474d-c5d3-4c69-b12f-993ff23910c1\") " pod="ai-tenant-e2e-shared-b-w2-797488/sub-test-model-797488-kserve-79764959f-k4scd"
Nov 02 15:41:57.221587 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:41:57.221505    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2d6e474d-c5d3-4c69-b12f-993ff23910c1-dshm\") pod \"sub-test-model-797488-kserve-79764959f-k4scd\" (UID: \"2d6e474d-c5d3-4c69-b12f-993ff23910c1\") " pod="ai-tenant-e2e-shared-b-w2-797488/sub-test-model-797488-kserve-79764959f-k4scd"
Nov 02 15:41:57.221587 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:41:57.221539    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2d6e474d-c5d3-4c69-b12f-993ff23910c1-home\") pod \"sub-test-model-797488-kserve-79764959f-k4scd\" (UID: \"2d6e474d-c5d3-4c69-b12f-993ff23910c1\") " pod="ai-tenant-e2e-shared-b-w2-797488/sub-test-model-797488-kserve-79764959f-k4scd"
Nov 02 15:41:57.221837 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:41:57.221604    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d6e474d-c5d3-4c69-b12f-993ff23910c1-tmp-dir\") pod \"sub-test-model-797488-kserve-79764959f-k4scd\" (UID: \"2d6e474d-c5d3-4c69-b12f-993ff23910c1\") " pod="ai-tenant-e2e-shared-b-w2-797488/sub-test-model-797488-kserve-79764959f-k4scd"
Nov 02 15:41:57.222053 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:41:57.221931    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2d6e474d-c5d3-4c69-b12f-993ff23910c1-model-cache\") pod \"sub-test-model-797488-kserve-79764959f-k4scd\" (UID: \"2d6e474d-c5d3-4c69-b12f-993ff23910c1\") " pod="ai-tenant-e2e-shared-b-w2-797488/sub-test-model-797488-kserve-79764959f-k4scd"
Nov 02 15:41:57.222212 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:41:57.222005    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d6e474d-c5d3-4c69-b12f-993ff23910c1-tmp-dir\") pod \"sub-test-model-797488-kserve-79764959f-k4scd\" (UID: \"2d6e474d-c5d3-4c69-b12f-993ff23910c1\") " pod="ai-tenant-e2e-shared-b-w2-797488/sub-test-model-797488-kserve-79764959f-k4scd"
Nov 02 15:41:57.222212 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:41:57.222057    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2d6e474d-c5d3-4c69-b12f-993ff23910c1-home\") pod \"sub-test-model-797488-kserve-79764959f-k4scd\" (UID: \"2d6e474d-c5d3-4c69-b12f-993ff23910c1\") " pod="ai-tenant-e2e-shared-b-w2-797488/sub-test-model-797488-kserve-79764959f-k4scd"
Nov 02 15:41:57.223910 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:41:57.223888    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2d6e474d-c5d3-4c69-b12f-993ff23910c1-dshm\") pod \"sub-test-model-797488-kserve-79764959f-k4scd\" (UID: \"2d6e474d-c5d3-4c69-b12f-993ff23910c1\") " pod="ai-tenant-e2e-shared-b-w2-797488/sub-test-model-797488-kserve-79764959f-k4scd"
Nov 02 15:41:57.224040 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:41:57.223915    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d6e474d-c5d3-4c69-b12f-993ff23910c1-tls-certs\") pod \"sub-test-model-797488-kserve-79764959f-k4scd\" (UID: \"2d6e474d-c5d3-4c69-b12f-993ff23910c1\") " pod="ai-tenant-e2e-shared-b-w2-797488/sub-test-model-797488-kserve-79764959f-k4scd"
Nov 02 15:41:57.249491 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:41:57.249457    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2dj2p\" (UniqueName: \"kubernetes.io/projected/2d6e474d-c5d3-4c69-b12f-993ff23910c1-kube-api-access-2dj2p\") pod \"sub-test-model-797488-kserve-79764959f-k4scd\" (UID: \"2d6e474d-c5d3-4c69-b12f-993ff23910c1\") " pod="ai-tenant-e2e-shared-b-w2-797488/sub-test-model-797488-kserve-79764959f-k4scd"
Nov 02 15:41:57.351770 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:41:57.351724    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-797488/sub-test-model-797488-kserve-79764959f-k4scd"
Nov 02 15:41:57.675198 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:41:57.675162    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-797488/sub-test-model-797488-kserve-79764959f-k4scd"]
Nov 02 15:41:58.250210 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:41:58.250169    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-797488/sub-test-model-797488-kserve-79764959f-k4scd" event={"ID":"2d6e474d-c5d3-4c69-b12f-993ff23910c1","Type":"ContainerStarted","Data":"ed3dfc44a4c9b27ff03ddc35712aa5e43e60362546463f53adf9ea3e0556a7eb"}
Nov 02 15:41:58.250663 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:41:58.250223    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-797488/sub-test-model-797488-kserve-79764959f-k4scd" event={"ID":"2d6e474d-c5d3-4c69-b12f-993ff23910c1","Type":"ContainerStarted","Data":"ea166f3df9e64bfb8ba6de86155dd0052e15c76c65eed7fd98aac1c3e1e7c542"}
Nov 02 15:41:58.324336 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:41:58.324282    5304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w2-797488/sub-test-model-797488-kserve-79764959f-k4scd" podStartSLOduration=2.324267004 podStartE2EDuration="2.324267004s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:41:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:41:58.318572689 +0000 UTC m=+2718.312397097" watchObservedRunningTime="2026-11-02 15:41:58.324267004 +0000 UTC m=+2718.318091453"
Nov 02 15:41:58.352055 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:41:58.352011    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w2-797488/sub-test-model-797488-kserve-79764959f-k4scd"
Nov 02 15:41:59.182897 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:41:59.182860    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-9cfd1f-77499c8864-p9ccn"
Nov 02 15:41:59.183081 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:41:59.182939    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9cfd1f-9bf597b4c-6fbz9"
Nov 02 15:42:03.521280 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:42:03.521222    5304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-cspmf"]
Nov 02 15:42:03.526371 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:42:03.526344    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-cspmf"
Nov 02 15:42:03.542572 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:42:03.542546    5304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-3805b8-openshift-default-dockercfg-h5w4z\""
Nov 02 15:42:03.555812 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:42:03.555783    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-cspmf"]
Nov 02 15:42:03.629155 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:42:03.628519    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/529d1f83-57ea-433f-aa66-ee16157950ca-istio-data\") pod \"e2e-ipp-b-3805b8-openshift-default-6c9f89869-cspmf\" (UID: \"529d1f83-57ea-433f-aa66-ee16157950ca\") " pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-cspmf"
Nov 02 15:42:03.629155 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:42:03.628636    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/529d1f83-57ea-433f-aa66-ee16157950ca-istio-podinfo\") pod \"e2e-ipp-b-3805b8-openshift-default-6c9f89869-cspmf\" (UID: \"529d1f83-57ea-433f-aa66-ee16157950ca\") " pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-cspmf"
Nov 02 15:42:03.629155 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:42:03.628683    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/529d1f83-57ea-433f-aa66-ee16157950ca-istio-envoy\") pod \"e2e-ipp-b-3805b8-openshift-default-6c9f89869-cspmf\" (UID: \"529d1f83-57ea-433f-aa66-ee16157950ca\") " pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-cspmf"
Nov 02 15:42:03.629155 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:42:03.628725    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/529d1f83-57ea-433f-aa66-ee16157950ca-istiod-ca-cert\") pod \"e2e-ipp-b-3805b8-openshift-default-6c9f89869-cspmf\" (UID: \"529d1f83-57ea-433f-aa66-ee16157950ca\") " pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-cspmf"
Nov 02 15:42:03.629155 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:42:03.628757    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/529d1f83-57ea-433f-aa66-ee16157950ca-workload-certs\") pod \"e2e-ipp-b-3805b8-openshift-default-6c9f89869-cspmf\" (UID: \"529d1f83-57ea-433f-aa66-ee16157950ca\") " pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-cspmf"
Nov 02 15:42:03.629155 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:42:03.628798    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/529d1f83-57ea-433f-aa66-ee16157950ca-workload-socket\") pod \"e2e-ipp-b-3805b8-openshift-default-6c9f89869-cspmf\" (UID: \"529d1f83-57ea-433f-aa66-ee16157950ca\") " pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-cspmf"
Nov 02 15:42:03.629155 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:42:03.628827    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lhjbd\" (UniqueName: \"kubernetes.io/projected/529d1f83-57ea-433f-aa66-ee16157950ca-kube-api-access-lhjbd\") pod \"e2e-ipp-b-3805b8-openshift-default-6c9f89869-cspmf\" (UID: \"529d1f83-57ea-433f-aa66-ee16157950ca\") " pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-cspmf"
Nov 02 15:42:03.629155 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:42:03.628858    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/529d1f83-57ea-433f-aa66-ee16157950ca-credential-socket\") pod \"e2e-ipp-b-3805b8-openshift-default-6c9f89869-cspmf\" (UID: \"529d1f83-57ea-433f-aa66-ee16157950ca\") " pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-cspmf"
Nov 02 15:42:03.629155 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:42:03.628956    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/529d1f83-57ea-433f-aa66-ee16157950ca-istio-token\") pod \"e2e-ipp-b-3805b8-openshift-default-6c9f89869-cspmf\" (UID: \"529d1f83-57ea-433f-aa66-ee16157950ca\") " pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-cspmf"
Nov 02 15:42:03.736761 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:42:03.734905    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/529d1f83-57ea-433f-aa66-ee16157950ca-istio-token\") pod \"e2e-ipp-b-3805b8-openshift-default-6c9f89869-cspmf\" (UID: \"529d1f83-57ea-433f-aa66-ee16157950ca\") " pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-cspmf"
Nov 02 15:42:03.736761 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:42:03.735015    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/529d1f83-57ea-433f-aa66-ee16157950ca-istio-data\") pod \"e2e-ipp-b-3805b8-openshift-default-6c9f89869-cspmf\" (UID: \"529d1f83-57ea-433f-aa66-ee16157950ca\") " pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-cspmf"
Nov 02 15:42:03.736761 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:42:03.735093    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/529d1f83-57ea-433f-aa66-ee16157950ca-istio-podinfo\") pod \"e2e-ipp-b-3805b8-openshift-default-6c9f89869-cspmf\" (UID: \"529d1f83-57ea-433f-aa66-ee16157950ca\") " pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-cspmf"
Nov 02 15:42:03.736761 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:42:03.736532    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/529d1f83-57ea-433f-aa66-ee16157950ca-istio-data\") pod \"e2e-ipp-b-3805b8-openshift-default-6c9f89869-cspmf\" (UID: \"529d1f83-57ea-433f-aa66-ee16157950ca\") " pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-cspmf"
Nov 02 15:42:03.738152 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:42:03.737552    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/529d1f83-57ea-433f-aa66-ee16157950ca-istio-envoy\") pod \"e2e-ipp-b-3805b8-openshift-default-6c9f89869-cspmf\" (UID: \"529d1f83-57ea-433f-aa66-ee16157950ca\") " pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-cspmf"
Nov 02 15:42:03.738152 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:42:03.737608    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/529d1f83-57ea-433f-aa66-ee16157950ca-istiod-ca-cert\") pod \"e2e-ipp-b-3805b8-openshift-default-6c9f89869-cspmf\" (UID: \"529d1f83-57ea-433f-aa66-ee16157950ca\") " pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-cspmf"
Nov 02 15:42:03.738152 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:42:03.737638    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/529d1f83-57ea-433f-aa66-ee16157950ca-workload-certs\") pod \"e2e-ipp-b-3805b8-openshift-default-6c9f89869-cspmf\" (UID: \"529d1f83-57ea-433f-aa66-ee16157950ca\") " pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-cspmf"
Nov 02 15:42:03.738152 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:42:03.737678    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/529d1f83-57ea-433f-aa66-ee16157950ca-workload-socket\") pod \"e2e-ipp-b-3805b8-openshift-default-6c9f89869-cspmf\" (UID: \"529d1f83-57ea-433f-aa66-ee16157950ca\") " pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-cspmf"
Nov 02 15:42:03.738152 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:42:03.737709    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lhjbd\" (UniqueName: \"kubernetes.io/projected/529d1f83-57ea-433f-aa66-ee16157950ca-kube-api-access-lhjbd\") pod \"e2e-ipp-b-3805b8-openshift-default-6c9f89869-cspmf\" (UID: \"529d1f83-57ea-433f-aa66-ee16157950ca\") " pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-cspmf"
Nov 02 15:42:03.738152 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:42:03.737739    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/529d1f83-57ea-433f-aa66-ee16157950ca-credential-socket\") pod \"e2e-ipp-b-3805b8-openshift-default-6c9f89869-cspmf\" (UID: \"529d1f83-57ea-433f-aa66-ee16157950ca\") " pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-cspmf"
Nov 02 15:42:03.738152 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:42:03.738080    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/529d1f83-57ea-433f-aa66-ee16157950ca-credential-socket\") pod \"e2e-ipp-b-3805b8-openshift-default-6c9f89869-cspmf\" (UID: \"529d1f83-57ea-433f-aa66-ee16157950ca\") " pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-cspmf"
Nov 02 15:42:03.740909 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:42:03.739577    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/529d1f83-57ea-433f-aa66-ee16157950ca-workload-certs\") pod \"e2e-ipp-b-3805b8-openshift-default-6c9f89869-cspmf\" (UID: \"529d1f83-57ea-433f-aa66-ee16157950ca\") " pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-cspmf"
Nov 02 15:42:03.740909 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:42:03.740326    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/529d1f83-57ea-433f-aa66-ee16157950ca-istiod-ca-cert\") pod \"e2e-ipp-b-3805b8-openshift-default-6c9f89869-cspmf\" (UID: \"529d1f83-57ea-433f-aa66-ee16157950ca\") " pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-cspmf"
Nov 02 15:42:03.742487 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:42:03.742445    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/529d1f83-57ea-433f-aa66-ee16157950ca-workload-socket\") pod \"e2e-ipp-b-3805b8-openshift-default-6c9f89869-cspmf\" (UID: \"529d1f83-57ea-433f-aa66-ee16157950ca\") " pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-cspmf"
Nov 02 15:42:03.748568 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:42:03.748542    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/529d1f83-57ea-433f-aa66-ee16157950ca-istio-envoy\") pod \"e2e-ipp-b-3805b8-openshift-default-6c9f89869-cspmf\" (UID: \"529d1f83-57ea-433f-aa66-ee16157950ca\") " pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-cspmf"
Nov 02 15:42:03.750739 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:42:03.750715    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/529d1f83-57ea-433f-aa66-ee16157950ca-istio-podinfo\") pod \"e2e-ipp-b-3805b8-openshift-default-6c9f89869-cspmf\" (UID: \"529d1f83-57ea-433f-aa66-ee16157950ca\") " pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-cspmf"
Nov 02 15:42:03.760832 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:42:03.760807    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/529d1f83-57ea-433f-aa66-ee16157950ca-istio-token\") pod \"e2e-ipp-b-3805b8-openshift-default-6c9f89869-cspmf\" (UID: \"529d1f83-57ea-433f-aa66-ee16157950ca\") " pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-cspmf"
Nov 02 15:42:03.764027 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:42:03.763996    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lhjbd\" (UniqueName: \"kubernetes.io/projected/529d1f83-57ea-433f-aa66-ee16157950ca-kube-api-access-lhjbd\") pod \"e2e-ipp-b-3805b8-openshift-default-6c9f89869-cspmf\" (UID: \"529d1f83-57ea-433f-aa66-ee16157950ca\") " pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-cspmf"
Nov 02 15:42:03.842783 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:42:03.842741    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-cspmf"
Nov 02 15:42:04.183639 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:42:04.183613    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-cspmf"]
Nov 02 15:42:04.186299 ip-10-0-135-138 kubenswrapper[5304]: W1102 15:42:04.186256    5304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod529d1f83_57ea_433f_aa66_ee16157950ca.slice/crio-a42710044e5eb273532556517341d653192000c01cfa9e890d30b92096aac8fb WatchSource:0}: Error finding container a42710044e5eb273532556517341d653192000c01cfa9e890d30b92096aac8fb: Status 404 returned error can't find the container with id a42710044e5eb273532556517341d653192000c01cfa9e890d30b92096aac8fb
Nov 02 15:42:04.189044 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:42:04.189007    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:42:04.189164 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:42:04.189077    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:42:04.189164 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:42:04.189106    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:42:04.299452 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:42:04.299415    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-cspmf" event={"ID":"529d1f83-57ea-433f-aa66-ee16157950ca","Type":"ContainerStarted","Data":"a42710044e5eb273532556517341d653192000c01cfa9e890d30b92096aac8fb"}
Nov 02 15:42:05.307516 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:42:05.307477    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-cspmf" event={"ID":"529d1f83-57ea-433f-aa66-ee16157950ca","Type":"ContainerStarted","Data":"6774500a1af6c9e7e9c101c43bd4da5839fc192191732fd0aff2eba2c0b62600"}
Nov 02 15:42:05.380349 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:42:05.380276    5304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-cspmf" podStartSLOduration=2.380256596 podStartE2EDuration="2.380256596s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:42:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:42:05.378196654 +0000 UTC m=+2725.372021062" watchObservedRunningTime="2026-11-02 15:42:05.380256596 +0000 UTC m=+2725.374081011"
Nov 02 15:42:05.854159 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:42:05.853525    5304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-cspmf"
Nov 02 15:42:05.860269 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:42:05.860237    5304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-cspmf"
Nov 02 15:42:06.313719 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:42:06.313632    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-cspmf"
Nov 02 15:42:06.314702 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:42:06.314682    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-cspmf"
Nov 02 15:42:07.354188 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:42:07.354159    5304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w2-797488/sub-test-model-797488-kserve-79764959f-k4scd"
Nov 02 15:42:07.367814 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:42:07.367787    5304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w2-797488/sub-test-model-797488-kserve-79764959f-k4scd"
Nov 02 15:42:08.368828 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:42:08.368801    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w2-797488/sub-test-model-797488-kserve-79764959f-k4scd"
Nov 02 15:42:10.174027 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:42:10.173996    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f84ad8-565799f599-4cnmv"]
Nov 02 15:42:10.241689 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:42:10.241646    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f84ad8-795988bcd7-tghkt"]
Nov 02 15:42:10.722016 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:42:10.721979    5304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-2wwx7"]
Nov 02 15:42:10.727359 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:42:10.727332    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-2wwx7"
Nov 02 15:42:10.752318 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:42:10.752288    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-2wwx7"]
Nov 02 15:42:10.755475 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:42:10.755434    5304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-e80c77-openshift-default-dockercfg-b7nkf\""
Nov 02 15:42:10.793617 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:42:10.793584    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e11f9d23-10db-4bad-a457-7c5db3056979-workload-socket\") pod \"e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-2wwx7\" (UID: \"e11f9d23-10db-4bad-a457-7c5db3056979\") " pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-2wwx7"
Nov 02 15:42:10.793845 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:42:10.793628    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e11f9d23-10db-4bad-a457-7c5db3056979-istio-token\") pod \"e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-2wwx7\" (UID: \"e11f9d23-10db-4bad-a457-7c5db3056979\") " pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-2wwx7"
Nov 02 15:42:10.793845 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:42:10.793657    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e11f9d23-10db-4bad-a457-7c5db3056979-credential-socket\") pod \"e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-2wwx7\" (UID: \"e11f9d23-10db-4bad-a457-7c5db3056979\") " pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-2wwx7"
Nov 02 15:42:10.793845 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:42:10.793700    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e11f9d23-10db-4bad-a457-7c5db3056979-workload-certs\") pod \"e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-2wwx7\" (UID: \"e11f9d23-10db-4bad-a457-7c5db3056979\") " pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-2wwx7"
Nov 02 15:42:10.793845 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:42:10.793770    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e11f9d23-10db-4bad-a457-7c5db3056979-istio-data\") pod \"e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-2wwx7\" (UID: \"e11f9d23-10db-4bad-a457-7c5db3056979\") " pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-2wwx7"
Nov 02 15:42:10.793845 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:42:10.793797    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x4p6m\" (UniqueName: \"kubernetes.io/projected/e11f9d23-10db-4bad-a457-7c5db3056979-kube-api-access-x4p6m\") pod \"e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-2wwx7\" (UID: \"e11f9d23-10db-4bad-a457-7c5db3056979\") " pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-2wwx7"
Nov 02 15:42:10.793845 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:42:10.793837    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e11f9d23-10db-4bad-a457-7c5db3056979-istiod-ca-cert\") pod \"e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-2wwx7\" (UID: \"e11f9d23-10db-4bad-a457-7c5db3056979\") " pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-2wwx7"
Nov 02 15:42:10.794173 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:42:10.793880    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e11f9d23-10db-4bad-a457-7c5db3056979-istio-envoy\") pod \"e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-2wwx7\" (UID: \"e11f9d23-10db-4bad-a457-7c5db3056979\") " pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-2wwx7"
Nov 02 15:42:10.794173 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:42:10.793902    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e11f9d23-10db-4bad-a457-7c5db3056979-istio-podinfo\") pod \"e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-2wwx7\" (UID: \"e11f9d23-10db-4bad-a457-7c5db3056979\") " pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-2wwx7"
Nov 02 15:42:10.895437 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:42:10.895391    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e11f9d23-10db-4bad-a457-7c5db3056979-workload-socket\") pod \"e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-2wwx7\" (UID: \"e11f9d23-10db-4bad-a457-7c5db3056979\") " pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-2wwx7"
Nov 02 15:42:10.895643 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:42:10.895449    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e11f9d23-10db-4bad-a457-7c5db3056979-istio-token\") pod \"e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-2wwx7\" (UID: \"e11f9d23-10db-4bad-a457-7c5db3056979\") " pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-2wwx7"
Nov 02 15:42:10.895643 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:42:10.895478    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e11f9d23-10db-4bad-a457-7c5db3056979-credential-socket\") pod \"e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-2wwx7\" (UID: \"e11f9d23-10db-4bad-a457-7c5db3056979\") " pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-2wwx7"
Nov 02 15:42:10.895643 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:42:10.895527    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e11f9d23-10db-4bad-a457-7c5db3056979-workload-certs\") pod \"e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-2wwx7\" (UID: \"e11f9d23-10db-4bad-a457-7c5db3056979\") " pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-2wwx7"
Nov 02 15:42:10.895837 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:42:10.895651    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e11f9d23-10db-4bad-a457-7c5db3056979-istio-data\") pod \"e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-2wwx7\" (UID: \"e11f9d23-10db-4bad-a457-7c5db3056979\") " pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-2wwx7"
Nov 02 15:42:10.895837 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:42:10.895700    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x4p6m\" (UniqueName: \"kubernetes.io/projected/e11f9d23-10db-4bad-a457-7c5db3056979-kube-api-access-x4p6m\") pod \"e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-2wwx7\" (UID: \"e11f9d23-10db-4bad-a457-7c5db3056979\") " pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-2wwx7"
Nov 02 15:42:10.895837 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:42:10.895761    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e11f9d23-10db-4bad-a457-7c5db3056979-istiod-ca-cert\") pod \"e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-2wwx7\" (UID: \"e11f9d23-10db-4bad-a457-7c5db3056979\") " pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-2wwx7"
Nov 02 15:42:10.896011 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:42:10.895841    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e11f9d23-10db-4bad-a457-7c5db3056979-istio-envoy\") pod \"e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-2wwx7\" (UID: \"e11f9d23-10db-4bad-a457-7c5db3056979\") " pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-2wwx7"
Nov 02 15:42:10.896011 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:42:10.895866    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e11f9d23-10db-4bad-a457-7c5db3056979-istio-podinfo\") pod \"e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-2wwx7\" (UID: \"e11f9d23-10db-4bad-a457-7c5db3056979\") " pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-2wwx7"
Nov 02 15:42:10.896011 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:42:10.895902    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e11f9d23-10db-4bad-a457-7c5db3056979-workload-socket\") pod \"e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-2wwx7\" (UID: \"e11f9d23-10db-4bad-a457-7c5db3056979\") " pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-2wwx7"
Nov 02 15:42:10.896274 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:42:10.896247    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e11f9d23-10db-4bad-a457-7c5db3056979-credential-socket\") pod \"e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-2wwx7\" (UID: \"e11f9d23-10db-4bad-a457-7c5db3056979\") " pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-2wwx7"
Nov 02 15:42:10.896412 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:42:10.896312    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e11f9d23-10db-4bad-a457-7c5db3056979-istio-data\") pod \"e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-2wwx7\" (UID: \"e11f9d23-10db-4bad-a457-7c5db3056979\") " pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-2wwx7"
Nov 02 15:42:10.896491 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:42:10.896466    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e11f9d23-10db-4bad-a457-7c5db3056979-workload-certs\") pod \"e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-2wwx7\" (UID: \"e11f9d23-10db-4bad-a457-7c5db3056979\") " pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-2wwx7"
Nov 02 15:42:10.896556 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:42:10.896540    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e11f9d23-10db-4bad-a457-7c5db3056979-istiod-ca-cert\") pod \"e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-2wwx7\" (UID: \"e11f9d23-10db-4bad-a457-7c5db3056979\") " pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-2wwx7"
Nov 02 15:42:10.898631 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:42:10.898600    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e11f9d23-10db-4bad-a457-7c5db3056979-istio-envoy\") pod \"e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-2wwx7\" (UID: \"e11f9d23-10db-4bad-a457-7c5db3056979\") " pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-2wwx7"
Nov 02 15:42:10.898930 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:42:10.898907    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e11f9d23-10db-4bad-a457-7c5db3056979-istio-podinfo\") pod \"e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-2wwx7\" (UID: \"e11f9d23-10db-4bad-a457-7c5db3056979\") " pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-2wwx7"
Nov 02 15:42:10.928017 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:42:10.927977    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e11f9d23-10db-4bad-a457-7c5db3056979-istio-token\") pod \"e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-2wwx7\" (UID: \"e11f9d23-10db-4bad-a457-7c5db3056979\") " pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-2wwx7"
Nov 02 15:42:10.928600 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:42:10.928578    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x4p6m\" (UniqueName: \"kubernetes.io/projected/e11f9d23-10db-4bad-a457-7c5db3056979-kube-api-access-x4p6m\") pod \"e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-2wwx7\" (UID: \"e11f9d23-10db-4bad-a457-7c5db3056979\") " pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-2wwx7"
Nov 02 15:42:11.058724 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:42:11.058370    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-2wwx7"
Nov 02 15:42:11.479654 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:42:11.479627    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-2wwx7"]
Nov 02 15:42:11.482218 ip-10-0-135-138 kubenswrapper[5304]: W1102 15:42:11.482165    5304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode11f9d23_10db_4bad_a457_7c5db3056979.slice/crio-db6e887cf31577dc937784515fa0fd0f8d8be227246059d5a9e25b7e41acc35a WatchSource:0}: Error finding container db6e887cf31577dc937784515fa0fd0f8d8be227246059d5a9e25b7e41acc35a: Status 404 returned error can't find the container with id db6e887cf31577dc937784515fa0fd0f8d8be227246059d5a9e25b7e41acc35a
Nov 02 15:42:11.485323 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:42:11.485285    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:42:11.485432 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:42:11.485356    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:42:11.485432 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:42:11.485386    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:42:12.383427 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:42:12.383384    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-2wwx7" event={"ID":"e11f9d23-10db-4bad-a457-7c5db3056979","Type":"ContainerStarted","Data":"82afe6e52444f7b0e63d73c694b765f1e4a10e6f6969c22dddf3cbbce5aedf44"}
Nov 02 15:42:12.383666 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:42:12.383453    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-2wwx7" event={"ID":"e11f9d23-10db-4bad-a457-7c5db3056979","Type":"ContainerStarted","Data":"db6e887cf31577dc937784515fa0fd0f8d8be227246059d5a9e25b7e41acc35a"}
Nov 02 15:42:12.432775 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:42:12.432723    5304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-2wwx7" podStartSLOduration=2.432708915 podStartE2EDuration="2.432708915s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:42:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:42:12.429163147 +0000 UTC m=+2732.422987553" watchObservedRunningTime="2026-11-02 15:42:12.432708915 +0000 UTC m=+2732.426533388"
Nov 02 15:42:12.691314 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:42:12.691234    5304 patch_prober.go:29] interesting pod/e2e-ipp-a-f84ad8-openshift-default-779f5d9f49-mstqq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.160:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 15:42:12.691680 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:42:12.691300    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-f84ad8-openshift-default-779f5d9f49-mstqq" podUID="b96fbb28-3d8d-4871-8895-ab78dfab160d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.160:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:42:13.059000 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:42:13.058917    5304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-2wwx7"
Nov 02 15:42:13.065895 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:42:13.065867    5304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-2wwx7"
Nov 02 15:42:13.390569 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:42:13.390352    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-2wwx7"
Nov 02 15:42:13.391359 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:42:13.391335    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-2wwx7"
Nov 02 15:42:14.397546 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:42:14.397464    5304 generic.go:356] "Generic (PLEG): container finished" podID="b96fbb28-3d8d-4871-8895-ab78dfab160d" containerID="8dd5dd14b116e03a35e9b9218038cdacfc7c086554d20c52616e125e269e744c" exitCode=0
Nov 02 15:42:14.397546 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:42:14.397536    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f84ad8-openshift-default-779f5d9f49-mstqq" event={"ID":"b96fbb28-3d8d-4871-8895-ab78dfab160d","Type":"ContainerDied","Data":"8dd5dd14b116e03a35e9b9218038cdacfc7c086554d20c52616e125e269e744c"}
Nov 02 15:42:14.398105 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:42:14.398092    5304 scope.go:122] "RemoveContainer" containerID="8dd5dd14b116e03a35e9b9218038cdacfc7c086554d20c52616e125e269e744c"
Nov 02 15:42:14.399761 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:42:14.399731    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:42:14.399862 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:42:14.399800    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:42:14.399862 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:42:14.399833    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:42:15.174569 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:42:15.174510    5304 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-f84ad8-565799f599-4cnmv" podUID="c3700f5e-e510-4aec-92cb-e223459f4d0f" containerName="payload-processing" containerID="cri-o://8d556e318b0ffeea2d755f31279d27f0b5f2ff2081bb1603f392fee7bc7f995b" gracePeriod=25
Nov 02 15:42:15.246227 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:42:15.242208    5304 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f84ad8-795988bcd7-tghkt" podUID="454ca996-42ce-47d6-96c0-1c9610867def" containerName="payload-pre-processing" containerID="cri-o://dac19558ee7d019fab660b715896aac520ad6437156a67fa751e77404e7dc4ee" gracePeriod=25
Nov 02 15:42:15.411225 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:42:15.409189    5304 generic.go:356] "Generic (PLEG): container finished" podID="454ca996-42ce-47d6-96c0-1c9610867def" containerID="dac19558ee7d019fab660b715896aac520ad6437156a67fa751e77404e7dc4ee" exitCode=0
Nov 02 15:42:15.411225 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:42:15.409380    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f84ad8-795988bcd7-tghkt" event={"ID":"454ca996-42ce-47d6-96c0-1c9610867def","Type":"ContainerDied","Data":"dac19558ee7d019fab660b715896aac520ad6437156a67fa751e77404e7dc4ee"}
Nov 02 15:42:15.411714 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:42:15.411391    5304 generic.go:356] "Generic (PLEG): container finished" podID="c3700f5e-e510-4aec-92cb-e223459f4d0f" containerID="8d556e318b0ffeea2d755f31279d27f0b5f2ff2081bb1603f392fee7bc7f995b" exitCode=0
Nov 02 15:42:15.411714 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:42:15.411497    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-f84ad8-565799f599-4cnmv" event={"ID":"c3700f5e-e510-4aec-92cb-e223459f4d0f","Type":"ContainerDied","Data":"8d556e318b0ffeea2d755f31279d27f0b5f2ff2081bb1603f392fee7bc7f995b"}
Nov 02 15:42:15.419004 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:42:15.414512    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f84ad8-openshift-default-779f5d9f49-mstqq" event={"ID":"b96fbb28-3d8d-4871-8895-ab78dfab160d","Type":"ContainerStarted","Data":"016be369cc2fd35f4dad3d3ed3dba1cc3085d1b59257ffaeb55db4a43fd57c90"}
Nov 02 15:42:15.617323 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:42:15.617291    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-f84ad8-565799f599-4cnmv"
Nov 02 15:42:15.690214 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:42:15.690185    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f84ad8-795988bcd7-tghkt"
Nov 02 15:42:15.709675 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:42:15.709638    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/454ca996-42ce-47d6-96c0-1c9610867def-kube-api-access-x7tvc\" (UniqueName: \"kubernetes.io/projected/454ca996-42ce-47d6-96c0-1c9610867def-kube-api-access-x7tvc\") pod \"454ca996-42ce-47d6-96c0-1c9610867def\" (UID: \"454ca996-42ce-47d6-96c0-1c9610867def\") "
Nov 02 15:42:15.709854 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:42:15.709720    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c3700f5e-e510-4aec-92cb-e223459f4d0f-kube-api-access-jhftz\" (UniqueName: \"kubernetes.io/projected/c3700f5e-e510-4aec-92cb-e223459f4d0f-kube-api-access-jhftz\") pod \"c3700f5e-e510-4aec-92cb-e223459f4d0f\" (UID: \"c3700f5e-e510-4aec-92cb-e223459f4d0f\") "
Nov 02 15:42:15.709854 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:42:15.709799    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c3700f5e-e510-4aec-92cb-e223459f4d0f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3700f5e-e510-4aec-92cb-e223459f4d0f-plugins-config-volume\") pod \"c3700f5e-e510-4aec-92cb-e223459f4d0f\" (UID: \"c3700f5e-e510-4aec-92cb-e223459f4d0f\") "
Nov 02 15:42:15.710007 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:42:15.709984    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/454ca996-42ce-47d6-96c0-1c9610867def-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/454ca996-42ce-47d6-96c0-1c9610867def-plugins-config-volume\") pod \"454ca996-42ce-47d6-96c0-1c9610867def\" (UID: \"454ca996-42ce-47d6-96c0-1c9610867def\") "
Nov 02 15:42:15.710412 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:42:15.710380    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3700f5e-e510-4aec-92cb-e223459f4d0f-plugins-config-volume" pod "c3700f5e-e510-4aec-92cb-e223459f4d0f" (UID: "c3700f5e-e510-4aec-92cb-e223459f4d0f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:42:15.710550 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:42:15.710410    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/454ca996-42ce-47d6-96c0-1c9610867def-plugins-config-volume" pod "454ca996-42ce-47d6-96c0-1c9610867def" (UID: "454ca996-42ce-47d6-96c0-1c9610867def"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:42:15.710626 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:42:15.710596    5304 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/454ca996-42ce-47d6-96c0-1c9610867def-plugins-config-volume\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:42:15.710626 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:42:15.710617    5304 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3700f5e-e510-4aec-92cb-e223459f4d0f-plugins-config-volume\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:42:15.712411 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:42:15.712382    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/454ca996-42ce-47d6-96c0-1c9610867def-kube-api-access-x7tvc" pod "454ca996-42ce-47d6-96c0-1c9610867def" (UID: "454ca996-42ce-47d6-96c0-1c9610867def"). InnerVolumeSpecName "kube-api-access-x7tvc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:42:15.712551 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:42:15.712532    5304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-f84ad8-openshift-default-779f5d9f49-mstqq"
Nov 02 15:42:15.712626 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:42:15.712563    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3700f5e-e510-4aec-92cb-e223459f4d0f-kube-api-access-jhftz" pod "c3700f5e-e510-4aec-92cb-e223459f4d0f" (UID: "c3700f5e-e510-4aec-92cb-e223459f4d0f"). InnerVolumeSpecName "kube-api-access-jhftz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:42:15.712774 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:42:15.712750    5304 patch_prober.go:29] interesting pod/e2e-ipp-a-f84ad8-openshift-default-779f5d9f49-mstqq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" start-of-body=
Nov 02 15:42:15.712848 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:42:15.712801    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f84ad8-openshift-default-779f5d9f49-mstqq" podUID="b96fbb28-3d8d-4871-8895-ab78dfab160d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused"
Nov 02 15:42:15.811545 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:42:15.811462    5304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x7tvc\" (UniqueName: \"kubernetes.io/projected/454ca996-42ce-47d6-96c0-1c9610867def-kube-api-access-x7tvc\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:42:15.811545 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:42:15.811492    5304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jhftz\" (UniqueName: \"kubernetes.io/projected/c3700f5e-e510-4aec-92cb-e223459f4d0f-kube-api-access-jhftz\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:42:16.422585 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:42:16.422502    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f84ad8-795988bcd7-tghkt"
Nov 02 15:42:16.422585 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:42:16.422500    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f84ad8-795988bcd7-tghkt" event={"ID":"454ca996-42ce-47d6-96c0-1c9610867def","Type":"ContainerDied","Data":"8ef0907add5efc1ea93ceb32d323172846cda01ff848229500beb75e76f2384c"}
Nov 02 15:42:16.423153 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:42:16.422633    5304 scope.go:122] "RemoveContainer" containerID="dac19558ee7d019fab660b715896aac520ad6437156a67fa751e77404e7dc4ee"
Nov 02 15:42:16.424089 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:42:16.424060    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-f84ad8-565799f599-4cnmv" event={"ID":"c3700f5e-e510-4aec-92cb-e223459f4d0f","Type":"ContainerDied","Data":"9d450d412badabcad4dec72b1bbe18989ce3b12c008d50f6a088b0a345529c36"}
Nov 02 15:42:16.424324 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:42:16.424306    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-f84ad8-565799f599-4cnmv"
Nov 02 15:42:16.458561 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:42:16.458539    5304 scope.go:122] "RemoveContainer" containerID="8d556e318b0ffeea2d755f31279d27f0b5f2ff2081bb1603f392fee7bc7f995b"
Nov 02 15:42:16.483988 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:42:16.483951    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f84ad8-795988bcd7-tghkt"]
Nov 02 15:42:16.488710 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:42:16.488684    5304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f84ad8-795988bcd7-tghkt"]
Nov 02 15:42:16.523168 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:42:16.523135    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f84ad8-565799f599-4cnmv"]
Nov 02 15:42:16.528816 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:42:16.528789    5304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f84ad8-565799f599-4cnmv"]
Nov 02 15:42:16.713495 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:42:16.713416    5304 patch_prober.go:29] interesting pod/e2e-ipp-a-f84ad8-openshift-default-779f5d9f49-mstqq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" start-of-body=
Nov 02 15:42:16.713495 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:42:16.713476    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f84ad8-openshift-default-779f5d9f49-mstqq" podUID="b96fbb28-3d8d-4871-8895-ab78dfab160d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused"
Nov 02 15:42:16.978750 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:42:16.970819    5304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-e80c77-9dc7f56b7-mtbzn"]
Nov 02 15:42:16.978750 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:42:16.971745    5304 cpu_manager.go:414] "containerMap: removing container" podUID="454ca996-42ce-47d6-96c0-1c9610867def" containerName="payload-pre-processing"
Nov 02 15:42:16.978750 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:42:16.971772    5304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="454ca996-42ce-47d6-96c0-1c9610867def" containerName="payload-pre-processing"
Nov 02 15:42:16.978750 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:42:16.971843    5304 cpu_manager.go:414] "containerMap: removing container" podUID="c3700f5e-e510-4aec-92cb-e223459f4d0f" containerName="payload-processing"
Nov 02 15:42:16.978750 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:42:16.971852    5304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3700f5e-e510-4aec-92cb-e223459f4d0f" containerName="payload-processing"
Nov 02 15:42:16.978750 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:42:16.971999    5304 memory_manager.go:365] "RemoveStaleState removing state" podUID="454ca996-42ce-47d6-96c0-1c9610867def" containerName="payload-pre-processing"
Nov 02 15:42:16.978750 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:42:16.972017    5304 memory_manager.go:365] "RemoveStaleState removing state" podUID="c3700f5e-e510-4aec-92cb-e223459f4d0f" containerName="payload-processing"
Nov 02 15:42:17.004696 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:42:17.004664    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-e80c77-9dc7f56b7-mtbzn"]
Nov 02 15:42:17.005064 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:42:17.004992    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-e80c77-9dc7f56b7-mtbzn"
Nov 02 15:42:17.031555 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:42:17.031522    5304 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-e80c77\""
Nov 02 15:42:17.032662 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:42:17.032642    5304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-e80c77-dockercfg-w5tv4\""
Nov 02 15:42:17.033812 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:42:17.033776    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/109405a5-0ac0-4964-918e-95870cfbbbc7-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-e80c77-9dc7f56b7-mtbzn\" (UID: \"109405a5-0ac0-4964-918e-95870cfbbbc7\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-e80c77-9dc7f56b7-mtbzn"
Nov 02 15:42:17.034160 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:42:17.034117    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vnshk\" (UniqueName: \"kubernetes.io/projected/109405a5-0ac0-4964-918e-95870cfbbbc7-kube-api-access-vnshk\") pod \"payload-processing-e2e-ipp-b-e80c77-9dc7f56b7-mtbzn\" (UID: \"109405a5-0ac0-4964-918e-95870cfbbbc7\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-e80c77-9dc7f56b7-mtbzn"
Nov 02 15:42:17.142012 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:42:17.141958    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/109405a5-0ac0-4964-918e-95870cfbbbc7-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-e80c77-9dc7f56b7-mtbzn\" (UID: \"109405a5-0ac0-4964-918e-95870cfbbbc7\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-e80c77-9dc7f56b7-mtbzn"
Nov 02 15:42:17.143486 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:42:17.143460    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vnshk\" (UniqueName: \"kubernetes.io/projected/109405a5-0ac0-4964-918e-95870cfbbbc7-kube-api-access-vnshk\") pod \"payload-processing-e2e-ipp-b-e80c77-9dc7f56b7-mtbzn\" (UID: \"109405a5-0ac0-4964-918e-95870cfbbbc7\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-e80c77-9dc7f56b7-mtbzn"
Nov 02 15:42:17.163189 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:42:17.143143    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/109405a5-0ac0-4964-918e-95870cfbbbc7-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-e80c77-9dc7f56b7-mtbzn\" (UID: \"109405a5-0ac0-4964-918e-95870cfbbbc7\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-e80c77-9dc7f56b7-mtbzn"
Nov 02 15:42:17.184182 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:42:17.184039    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vnshk\" (UniqueName: \"kubernetes.io/projected/109405a5-0ac0-4964-918e-95870cfbbbc7-kube-api-access-vnshk\") pod \"payload-processing-e2e-ipp-b-e80c77-9dc7f56b7-mtbzn\" (UID: \"109405a5-0ac0-4964-918e-95870cfbbbc7\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-e80c77-9dc7f56b7-mtbzn"
Nov 02 15:42:17.414185 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:42:17.412266    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-e80c77-9dc7f56b7-mtbzn"
Nov 02 15:42:17.713332 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:42:17.713245    5304 patch_prober.go:29] interesting pod/e2e-ipp-a-f84ad8-openshift-default-779f5d9f49-mstqq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" start-of-body=
Nov 02 15:42:17.713332 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:42:17.713300    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f84ad8-openshift-default-779f5d9f49-mstqq" podUID="b96fbb28-3d8d-4871-8895-ab78dfab160d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused"
Nov 02 15:42:17.769353 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:42:17.769045    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-e80c77-9dc7f56b7-mtbzn"]
Nov 02 15:42:18.119342 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:42:18.119309    5304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="454ca996-42ce-47d6-96c0-1c9610867def" path="/var/lib/kubelet/pods/454ca996-42ce-47d6-96c0-1c9610867def/volumes"
Nov 02 15:42:18.119737 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:42:18.119724    5304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c3700f5e-e510-4aec-92cb-e223459f4d0f" path="/var/lib/kubelet/pods/c3700f5e-e510-4aec-92cb-e223459f4d0f/volumes"
Nov 02 15:42:18.473567 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:42:18.473466    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-e80c77-9dc7f56b7-mtbzn" event={"ID":"109405a5-0ac0-4964-918e-95870cfbbbc7","Type":"ContainerStarted","Data":"4f13fd63e93223c481a56d354e3c049dba1033fa216cf32356ce223bc33dff07"}
Nov 02 15:42:18.473567 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:42:18.473532    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-e80c77-9dc7f56b7-mtbzn" event={"ID":"109405a5-0ac0-4964-918e-95870cfbbbc7","Type":"ContainerStarted","Data":"26541f41a48e07ba2d85e5870a5101bc103aaba6b59edc5c3c1f84606bb90497"}
Nov 02 15:42:18.473567 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:42:18.473563    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-e80c77-9dc7f56b7-mtbzn"
Nov 02 15:42:18.516349 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:42:18.516292    5304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-e80c77-9dc7f56b7-mtbzn" podStartSLOduration=2.5162793949999998 podStartE2EDuration="2.516279395s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:42:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:42:18.51551296 +0000 UTC m=+2738.509337381" watchObservedRunningTime="2026-11-02 15:42:18.516279395 +0000 UTC m=+2738.510103803"
Nov 02 15:42:18.713874 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:42:18.713831    5304 patch_prober.go:29] interesting pod/e2e-ipp-a-f84ad8-openshift-default-779f5d9f49-mstqq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" start-of-body=
Nov 02 15:42:18.714544 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:42:18.714488    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f84ad8-openshift-default-779f5d9f49-mstqq" podUID="b96fbb28-3d8d-4871-8895-ab78dfab160d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused"
Nov 02 15:42:19.712633 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:42:19.712600    5304 patch_prober.go:29] interesting pod/e2e-ipp-a-f84ad8-openshift-default-779f5d9f49-mstqq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" start-of-body=
Nov 02 15:42:19.712817 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:42:19.712652    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f84ad8-openshift-default-779f5d9f49-mstqq" podUID="b96fbb28-3d8d-4871-8895-ab78dfab160d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused"
Nov 02 15:42:20.721073 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:42:20.719441    5304 patch_prober.go:29] interesting pod/e2e-ipp-a-f84ad8-openshift-default-779f5d9f49-mstqq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" start-of-body=
Nov 02 15:42:20.721073 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:42:20.719508    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f84ad8-openshift-default-779f5d9f49-mstqq" podUID="b96fbb28-3d8d-4871-8895-ab78dfab160d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused"
Nov 02 15:42:21.018653 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:42:21.018565    5304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w1-cce791-openshift-default-c54fc95f9-n5z7h"]
Nov 02 15:42:21.022227 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:42:21.022192    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-cce791-openshift-default-c54fc95f9-n5z7h"
Nov 02 15:42:21.037471 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:42:21.037443    5304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w1-cce791-openshift-default-dockercfg-w7mhr\""
Nov 02 15:42:21.052886 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:42:21.052858    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-cce791-openshift-default-c54fc95f9-n5z7h"]
Nov 02 15:42:21.127102 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:42:21.127066    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7bafb51c-17bf-4c9a-8e43-cfdf16c614d5-istio-podinfo\") pod \"e2e-shared-b-w1-cce791-openshift-default-c54fc95f9-n5z7h\" (UID: \"7bafb51c-17bf-4c9a-8e43-cfdf16c614d5\") " pod="openshift-ingress/e2e-shared-b-w1-cce791-openshift-default-c54fc95f9-n5z7h"
Nov 02 15:42:21.127337 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:42:21.127120    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bafb51c-17bf-4c9a-8e43-cfdf16c614d5-workload-socket\") pod \"e2e-shared-b-w1-cce791-openshift-default-c54fc95f9-n5z7h\" (UID: \"7bafb51c-17bf-4c9a-8e43-cfdf16c614d5\") " pod="openshift-ingress/e2e-shared-b-w1-cce791-openshift-default-c54fc95f9-n5z7h"
Nov 02 15:42:21.127337 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:42:21.127252    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7bafb51c-17bf-4c9a-8e43-cfdf16c614d5-istio-data\") pod \"e2e-shared-b-w1-cce791-openshift-default-c54fc95f9-n5z7h\" (UID: \"7bafb51c-17bf-4c9a-8e43-cfdf16c614d5\") " pod="openshift-ingress/e2e-shared-b-w1-cce791-openshift-default-c54fc95f9-n5z7h"
Nov 02 15:42:21.127337 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:42:21.127292    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9n57j\" (UniqueName: \"kubernetes.io/projected/7bafb51c-17bf-4c9a-8e43-cfdf16c614d5-kube-api-access-9n57j\") pod \"e2e-shared-b-w1-cce791-openshift-default-c54fc95f9-n5z7h\" (UID: \"7bafb51c-17bf-4c9a-8e43-cfdf16c614d5\") " pod="openshift-ingress/e2e-shared-b-w1-cce791-openshift-default-c54fc95f9-n5z7h"
Nov 02 15:42:21.127337 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:42:21.127325    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7bafb51c-17bf-4c9a-8e43-cfdf16c614d5-workload-certs\") pod \"e2e-shared-b-w1-cce791-openshift-default-c54fc95f9-n5z7h\" (UID: \"7bafb51c-17bf-4c9a-8e43-cfdf16c614d5\") " pod="openshift-ingress/e2e-shared-b-w1-cce791-openshift-default-c54fc95f9-n5z7h"
Nov 02 15:42:21.127580 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:42:21.127440    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7bafb51c-17bf-4c9a-8e43-cfdf16c614d5-istio-envoy\") pod \"e2e-shared-b-w1-cce791-openshift-default-c54fc95f9-n5z7h\" (UID: \"7bafb51c-17bf-4c9a-8e43-cfdf16c614d5\") " pod="openshift-ingress/e2e-shared-b-w1-cce791-openshift-default-c54fc95f9-n5z7h"
Nov 02 15:42:21.127580 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:42:21.127472    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7bafb51c-17bf-4c9a-8e43-cfdf16c614d5-istio-token\") pod \"e2e-shared-b-w1-cce791-openshift-default-c54fc95f9-n5z7h\" (UID: \"7bafb51c-17bf-4c9a-8e43-cfdf16c614d5\") " pod="openshift-ingress/e2e-shared-b-w1-cce791-openshift-default-c54fc95f9-n5z7h"
Nov 02 15:42:21.127580 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:42:21.127502    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bafb51c-17bf-4c9a-8e43-cfdf16c614d5-credential-socket\") pod \"e2e-shared-b-w1-cce791-openshift-default-c54fc95f9-n5z7h\" (UID: \"7bafb51c-17bf-4c9a-8e43-cfdf16c614d5\") " pod="openshift-ingress/e2e-shared-b-w1-cce791-openshift-default-c54fc95f9-n5z7h"
Nov 02 15:42:21.127580 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:42:21.127549    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7bafb51c-17bf-4c9a-8e43-cfdf16c614d5-istiod-ca-cert\") pod \"e2e-shared-b-w1-cce791-openshift-default-c54fc95f9-n5z7h\" (UID: \"7bafb51c-17bf-4c9a-8e43-cfdf16c614d5\") " pod="openshift-ingress/e2e-shared-b-w1-cce791-openshift-default-c54fc95f9-n5z7h"
Nov 02 15:42:21.228849 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:42:21.228813    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7bafb51c-17bf-4c9a-8e43-cfdf16c614d5-istio-podinfo\") pod \"e2e-shared-b-w1-cce791-openshift-default-c54fc95f9-n5z7h\" (UID: \"7bafb51c-17bf-4c9a-8e43-cfdf16c614d5\") " pod="openshift-ingress/e2e-shared-b-w1-cce791-openshift-default-c54fc95f9-n5z7h"
Nov 02 15:42:21.229049 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:42:21.228882    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bafb51c-17bf-4c9a-8e43-cfdf16c614d5-workload-socket\") pod \"e2e-shared-b-w1-cce791-openshift-default-c54fc95f9-n5z7h\" (UID: \"7bafb51c-17bf-4c9a-8e43-cfdf16c614d5\") " pod="openshift-ingress/e2e-shared-b-w1-cce791-openshift-default-c54fc95f9-n5z7h"
Nov 02 15:42:21.229193 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:42:21.229169    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7bafb51c-17bf-4c9a-8e43-cfdf16c614d5-istio-data\") pod \"e2e-shared-b-w1-cce791-openshift-default-c54fc95f9-n5z7h\" (UID: \"7bafb51c-17bf-4c9a-8e43-cfdf16c614d5\") " pod="openshift-ingress/e2e-shared-b-w1-cce791-openshift-default-c54fc95f9-n5z7h"
Nov 02 15:42:21.229280 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:42:21.229212    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9n57j\" (UniqueName: \"kubernetes.io/projected/7bafb51c-17bf-4c9a-8e43-cfdf16c614d5-kube-api-access-9n57j\") pod \"e2e-shared-b-w1-cce791-openshift-default-c54fc95f9-n5z7h\" (UID: \"7bafb51c-17bf-4c9a-8e43-cfdf16c614d5\") " pod="openshift-ingress/e2e-shared-b-w1-cce791-openshift-default-c54fc95f9-n5z7h"
Nov 02 15:42:21.229280 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:42:21.229258    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7bafb51c-17bf-4c9a-8e43-cfdf16c614d5-workload-certs\") pod \"e2e-shared-b-w1-cce791-openshift-default-c54fc95f9-n5z7h\" (UID: \"7bafb51c-17bf-4c9a-8e43-cfdf16c614d5\") " pod="openshift-ingress/e2e-shared-b-w1-cce791-openshift-default-c54fc95f9-n5z7h"
Nov 02 15:42:21.229406 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:42:21.229365    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bafb51c-17bf-4c9a-8e43-cfdf16c614d5-workload-socket\") pod \"e2e-shared-b-w1-cce791-openshift-default-c54fc95f9-n5z7h\" (UID: \"7bafb51c-17bf-4c9a-8e43-cfdf16c614d5\") " pod="openshift-ingress/e2e-shared-b-w1-cce791-openshift-default-c54fc95f9-n5z7h"
Nov 02 15:42:21.229480 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:42:21.229459    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7bafb51c-17bf-4c9a-8e43-cfdf16c614d5-istio-envoy\") pod \"e2e-shared-b-w1-cce791-openshift-default-c54fc95f9-n5z7h\" (UID: \"7bafb51c-17bf-4c9a-8e43-cfdf16c614d5\") " pod="openshift-ingress/e2e-shared-b-w1-cce791-openshift-default-c54fc95f9-n5z7h"
Nov 02 15:42:21.229553 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:42:21.229503    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7bafb51c-17bf-4c9a-8e43-cfdf16c614d5-istio-token\") pod \"e2e-shared-b-w1-cce791-openshift-default-c54fc95f9-n5z7h\" (UID: \"7bafb51c-17bf-4c9a-8e43-cfdf16c614d5\") " pod="openshift-ingress/e2e-shared-b-w1-cce791-openshift-default-c54fc95f9-n5z7h"
Nov 02 15:42:21.229553 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:42:21.229535    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bafb51c-17bf-4c9a-8e43-cfdf16c614d5-credential-socket\") pod \"e2e-shared-b-w1-cce791-openshift-default-c54fc95f9-n5z7h\" (UID: \"7bafb51c-17bf-4c9a-8e43-cfdf16c614d5\") " pod="openshift-ingress/e2e-shared-b-w1-cce791-openshift-default-c54fc95f9-n5z7h"
Nov 02 15:42:21.229664 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:42:21.229561    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7bafb51c-17bf-4c9a-8e43-cfdf16c614d5-istiod-ca-cert\") pod \"e2e-shared-b-w1-cce791-openshift-default-c54fc95f9-n5z7h\" (UID: \"7bafb51c-17bf-4c9a-8e43-cfdf16c614d5\") " pod="openshift-ingress/e2e-shared-b-w1-cce791-openshift-default-c54fc95f9-n5z7h"
Nov 02 15:42:21.229664 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:42:21.229468    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7bafb51c-17bf-4c9a-8e43-cfdf16c614d5-istio-data\") pod \"e2e-shared-b-w1-cce791-openshift-default-c54fc95f9-n5z7h\" (UID: \"7bafb51c-17bf-4c9a-8e43-cfdf16c614d5\") " pod="openshift-ingress/e2e-shared-b-w1-cce791-openshift-default-c54fc95f9-n5z7h"
Nov 02 15:42:21.229664 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:42:21.229579    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7bafb51c-17bf-4c9a-8e43-cfdf16c614d5-workload-certs\") pod \"e2e-shared-b-w1-cce791-openshift-default-c54fc95f9-n5z7h\" (UID: \"7bafb51c-17bf-4c9a-8e43-cfdf16c614d5\") " pod="openshift-ingress/e2e-shared-b-w1-cce791-openshift-default-c54fc95f9-n5z7h"
Nov 02 15:42:21.230325 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:42:21.230304    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7bafb51c-17bf-4c9a-8e43-cfdf16c614d5-istiod-ca-cert\") pod \"e2e-shared-b-w1-cce791-openshift-default-c54fc95f9-n5z7h\" (UID: \"7bafb51c-17bf-4c9a-8e43-cfdf16c614d5\") " pod="openshift-ingress/e2e-shared-b-w1-cce791-openshift-default-c54fc95f9-n5z7h"
Nov 02 15:42:21.230432 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:42:21.230378    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bafb51c-17bf-4c9a-8e43-cfdf16c614d5-credential-socket\") pod \"e2e-shared-b-w1-cce791-openshift-default-c54fc95f9-n5z7h\" (UID: \"7bafb51c-17bf-4c9a-8e43-cfdf16c614d5\") " pod="openshift-ingress/e2e-shared-b-w1-cce791-openshift-default-c54fc95f9-n5z7h"
Nov 02 15:42:21.231533 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:42:21.231509    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7bafb51c-17bf-4c9a-8e43-cfdf16c614d5-istio-podinfo\") pod \"e2e-shared-b-w1-cce791-openshift-default-c54fc95f9-n5z7h\" (UID: \"7bafb51c-17bf-4c9a-8e43-cfdf16c614d5\") " pod="openshift-ingress/e2e-shared-b-w1-cce791-openshift-default-c54fc95f9-n5z7h"
Nov 02 15:42:21.231699 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:42:21.231683    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7bafb51c-17bf-4c9a-8e43-cfdf16c614d5-istio-envoy\") pod \"e2e-shared-b-w1-cce791-openshift-default-c54fc95f9-n5z7h\" (UID: \"7bafb51c-17bf-4c9a-8e43-cfdf16c614d5\") " pod="openshift-ingress/e2e-shared-b-w1-cce791-openshift-default-c54fc95f9-n5z7h"
Nov 02 15:42:21.255417 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:42:21.255387    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7bafb51c-17bf-4c9a-8e43-cfdf16c614d5-istio-token\") pod \"e2e-shared-b-w1-cce791-openshift-default-c54fc95f9-n5z7h\" (UID: \"7bafb51c-17bf-4c9a-8e43-cfdf16c614d5\") " pod="openshift-ingress/e2e-shared-b-w1-cce791-openshift-default-c54fc95f9-n5z7h"
Nov 02 15:42:21.255572 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:42:21.255534    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9n57j\" (UniqueName: \"kubernetes.io/projected/7bafb51c-17bf-4c9a-8e43-cfdf16c614d5-kube-api-access-9n57j\") pod \"e2e-shared-b-w1-cce791-openshift-default-c54fc95f9-n5z7h\" (UID: \"7bafb51c-17bf-4c9a-8e43-cfdf16c614d5\") " pod="openshift-ingress/e2e-shared-b-w1-cce791-openshift-default-c54fc95f9-n5z7h"
Nov 02 15:42:21.337277 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:42:21.337248    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-cce791-openshift-default-c54fc95f9-n5z7h"
Nov 02 15:42:21.661816 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:42:21.661790    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-cce791-openshift-default-c54fc95f9-n5z7h"]
Nov 02 15:42:21.664578 ip-10-0-135-138 kubenswrapper[5304]: W1102 15:42:21.664550    5304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7bafb51c_17bf_4c9a_8e43_cfdf16c614d5.slice/crio-412cf8eefe3ae791c211b2925c174b01bd44a120553c221b8e70b297e7bddc3f WatchSource:0}: Error finding container 412cf8eefe3ae791c211b2925c174b01bd44a120553c221b8e70b297e7bddc3f: Status 404 returned error can't find the container with id 412cf8eefe3ae791c211b2925c174b01bd44a120553c221b8e70b297e7bddc3f
Nov 02 15:42:21.667533 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:42:21.667495    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:42:21.667631 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:42:21.667584    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:42:21.667703 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:42:21.667663    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:42:21.717511 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:42:21.717489    5304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-f84ad8-openshift-default-779f5d9f49-mstqq"
Nov 02 15:42:21.717874 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:42:21.717852    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-f84ad8-openshift-default-779f5d9f49-mstqq"
Nov 02 15:42:21.718595 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:42:21.718577    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-f84ad8-openshift-default-779f5d9f49-mstqq"
Nov 02 15:42:22.506545 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:42:22.506507    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-cce791-openshift-default-c54fc95f9-n5z7h" event={"ID":"7bafb51c-17bf-4c9a-8e43-cfdf16c614d5","Type":"ContainerStarted","Data":"0a7c1bdd6e9168536c04211525a73596221c25a083a8ccdfefe4e72d055e358e"}
Nov 02 15:42:22.507012 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:42:22.506564    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-cce791-openshift-default-c54fc95f9-n5z7h" event={"ID":"7bafb51c-17bf-4c9a-8e43-cfdf16c614d5","Type":"ContainerStarted","Data":"412cf8eefe3ae791c211b2925c174b01bd44a120553c221b8e70b297e7bddc3f"}
Nov 02 15:42:22.560487 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:42:22.560421    5304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w1-cce791-openshift-default-c54fc95f9-n5z7h" podStartSLOduration=2.560402426 podStartE2EDuration="2.560402426s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:42:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:42:22.555615076 +0000 UTC m=+2742.549439486" watchObservedRunningTime="2026-11-02 15:42:22.560402426 +0000 UTC m=+2742.554226836"
Nov 02 15:42:23.353229 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:42:23.353192    5304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-cce791-openshift-default-c54fc95f9-n5z7h"
Nov 02 15:42:23.355262 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:42:23.355230    5304 patch_prober.go:29] interesting pod/e2e-shared-b-w1-cce791-openshift-default-c54fc95f9-n5z7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused" start-of-body=
Nov 02 15:42:23.355441 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:42:23.355288    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-cce791-openshift-default-c54fc95f9-n5z7h" podUID="7bafb51c-17bf-4c9a-8e43-cfdf16c614d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused"
Nov 02 15:42:24.356297 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:42:24.356256    5304 patch_prober.go:29] interesting pod/e2e-shared-b-w1-cce791-openshift-default-c54fc95f9-n5z7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused" start-of-body=
Nov 02 15:42:24.356848 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:42:24.356328    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-cce791-openshift-default-c54fc95f9-n5z7h" podUID="7bafb51c-17bf-4c9a-8e43-cfdf16c614d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused"
Nov 02 15:42:24.799262 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:42:24.799163    5304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k6fjk" containerName="extractor"
Nov 02 15:42:25.339231 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:42:25.339192    5304 patch_prober.go:29] interesting pod/e2e-shared-b-w1-cce791-openshift-default-c54fc95f9-n5z7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused" start-of-body=
Nov 02 15:42:25.339527 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:42:25.339498    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-cce791-openshift-default-c54fc95f9-n5z7h" podUID="7bafb51c-17bf-4c9a-8e43-cfdf16c614d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused"
Nov 02 15:42:26.338946 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:42:26.338508    5304 patch_prober.go:29] interesting pod/e2e-shared-b-w1-cce791-openshift-default-c54fc95f9-n5z7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused" start-of-body=
Nov 02 15:42:26.338946 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:42:26.338570    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-cce791-openshift-default-c54fc95f9-n5z7h" podUID="7bafb51c-17bf-4c9a-8e43-cfdf16c614d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused"
Nov 02 15:42:27.350150 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:42:27.338465    5304 patch_prober.go:29] interesting pod/e2e-shared-b-w1-cce791-openshift-default-c54fc95f9-n5z7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused" start-of-body=
Nov 02 15:42:27.350150 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:42:27.338525    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-cce791-openshift-default-c54fc95f9-n5z7h" podUID="7bafb51c-17bf-4c9a-8e43-cfdf16c614d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused"
Nov 02 15:42:28.471611 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:42:28.471574    5304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-cce791-openshift-default-c54fc95f9-n5z7h"
Nov 02 15:42:28.472909 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:42:28.472878    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-cce791-openshift-default-c54fc95f9-n5z7h"
Nov 02 15:42:28.474447 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:42:28.474422    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-cce791-openshift-default-c54fc95f9-n5z7h"
Nov 02 15:42:29.483033 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:42:29.482997    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-e80c77-9dc7f56b7-mtbzn"
Nov 02 15:42:46.850194 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:42:46.850150    5304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f84ad8-789796656c-g97z7"]
Nov 02 15:42:46.875193 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:42:46.871487    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f84ad8-789796656c-g97z7"
Nov 02 15:42:46.875193 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:42:46.874698    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f84ad8-789796656c-g97z7"]
Nov 02 15:42:46.895879 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:42:46.895184    5304 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-f84ad8\""
Nov 02 15:42:46.908945 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:42:46.908890    5304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-f84ad8-dockercfg-crw4v\""
Nov 02 15:42:46.984457 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:42:46.984416    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l5sh9\" (UniqueName: \"kubernetes.io/projected/fc34fbb9-6804-4ad8-a007-25a6551a0960-kube-api-access-l5sh9\") pod \"payload-pre-processing-e2e-ipp-a-f84ad8-789796656c-g97z7\" (UID: \"fc34fbb9-6804-4ad8-a007-25a6551a0960\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f84ad8-789796656c-g97z7"
Nov 02 15:42:46.984664 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:42:46.984602    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc34fbb9-6804-4ad8-a007-25a6551a0960-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-f84ad8-789796656c-g97z7\" (UID: \"fc34fbb9-6804-4ad8-a007-25a6551a0960\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f84ad8-789796656c-g97z7"
Nov 02 15:42:47.013370 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:42:47.013332    5304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f84ad8-64f97947fb-zppnf"]
Nov 02 15:42:47.017095 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:42:47.017076    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-f84ad8-64f97947fb-zppnf"
Nov 02 15:42:47.040594 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:42:47.040562    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f84ad8-64f97947fb-zppnf"]
Nov 02 15:42:47.086044 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:42:47.086004    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5386e95b-8eef-49a6-a84e-80c8119c27fa-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-f84ad8-64f97947fb-zppnf\" (UID: \"5386e95b-8eef-49a6-a84e-80c8119c27fa\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f84ad8-64f97947fb-zppnf"
Nov 02 15:42:47.086800 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:42:47.086769    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jwstw\" (UniqueName: \"kubernetes.io/projected/5386e95b-8eef-49a6-a84e-80c8119c27fa-kube-api-access-jwstw\") pod \"payload-processing-e2e-ipp-a-f84ad8-64f97947fb-zppnf\" (UID: \"5386e95b-8eef-49a6-a84e-80c8119c27fa\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f84ad8-64f97947fb-zppnf"
Nov 02 15:42:47.087080 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:42:47.087048    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc34fbb9-6804-4ad8-a007-25a6551a0960-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-f84ad8-789796656c-g97z7\" (UID: \"fc34fbb9-6804-4ad8-a007-25a6551a0960\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f84ad8-789796656c-g97z7"
Nov 02 15:42:47.087342 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:42:47.087318    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l5sh9\" (UniqueName: \"kubernetes.io/projected/fc34fbb9-6804-4ad8-a007-25a6551a0960-kube-api-access-l5sh9\") pod \"payload-pre-processing-e2e-ipp-a-f84ad8-789796656c-g97z7\" (UID: \"fc34fbb9-6804-4ad8-a007-25a6551a0960\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f84ad8-789796656c-g97z7"
Nov 02 15:42:47.088201 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:42:47.088174    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc34fbb9-6804-4ad8-a007-25a6551a0960-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-f84ad8-789796656c-g97z7\" (UID: \"fc34fbb9-6804-4ad8-a007-25a6551a0960\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f84ad8-789796656c-g97z7"
Nov 02 15:42:47.111737 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:42:47.111673    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l5sh9\" (UniqueName: \"kubernetes.io/projected/fc34fbb9-6804-4ad8-a007-25a6551a0960-kube-api-access-l5sh9\") pod \"payload-pre-processing-e2e-ipp-a-f84ad8-789796656c-g97z7\" (UID: \"fc34fbb9-6804-4ad8-a007-25a6551a0960\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f84ad8-789796656c-g97z7"
Nov 02 15:42:47.189440 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:42:47.189408    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5386e95b-8eef-49a6-a84e-80c8119c27fa-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-f84ad8-64f97947fb-zppnf\" (UID: \"5386e95b-8eef-49a6-a84e-80c8119c27fa\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f84ad8-64f97947fb-zppnf"
Nov 02 15:42:47.189440 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:42:47.189443    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jwstw\" (UniqueName: \"kubernetes.io/projected/5386e95b-8eef-49a6-a84e-80c8119c27fa-kube-api-access-jwstw\") pod \"payload-processing-e2e-ipp-a-f84ad8-64f97947fb-zppnf\" (UID: \"5386e95b-8eef-49a6-a84e-80c8119c27fa\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f84ad8-64f97947fb-zppnf"
Nov 02 15:42:47.190195 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:42:47.190153    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5386e95b-8eef-49a6-a84e-80c8119c27fa-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-f84ad8-64f97947fb-zppnf\" (UID: \"5386e95b-8eef-49a6-a84e-80c8119c27fa\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f84ad8-64f97947fb-zppnf"
Nov 02 15:42:47.213210 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:42:47.213185    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jwstw\" (UniqueName: \"kubernetes.io/projected/5386e95b-8eef-49a6-a84e-80c8119c27fa-kube-api-access-jwstw\") pod \"payload-processing-e2e-ipp-a-f84ad8-64f97947fb-zppnf\" (UID: \"5386e95b-8eef-49a6-a84e-80c8119c27fa\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f84ad8-64f97947fb-zppnf"
Nov 02 15:42:47.217088 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:42:47.217057    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f84ad8-789796656c-g97z7"
Nov 02 15:42:47.330371 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:42:47.330341    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-f84ad8-64f97947fb-zppnf"
Nov 02 15:42:47.540359 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:42:47.540331    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f84ad8-789796656c-g97z7"]
Nov 02 15:42:47.542939 ip-10-0-135-138 kubenswrapper[5304]: W1102 15:42:47.542914    5304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc34fbb9_6804_4ad8_a007_25a6551a0960.slice/crio-1b7f6022b164e933c06198063b4a6841724dd32514f10008831710d42db8ae98 WatchSource:0}: Error finding container 1b7f6022b164e933c06198063b4a6841724dd32514f10008831710d42db8ae98: Status 404 returned error can't find the container with id 1b7f6022b164e933c06198063b4a6841724dd32514f10008831710d42db8ae98
Nov 02 15:42:47.662170 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:42:47.662101    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f84ad8-64f97947fb-zppnf"]
Nov 02 15:42:47.663034 ip-10-0-135-138 kubenswrapper[5304]: W1102 15:42:47.663008    5304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5386e95b_8eef_49a6_a84e_80c8119c27fa.slice/crio-34e392631a5f0fda92fbb33f52c4bba19b7a2199ad34d80dd7ccd6f992dc3862 WatchSource:0}: Error finding container 34e392631a5f0fda92fbb33f52c4bba19b7a2199ad34d80dd7ccd6f992dc3862: Status 404 returned error can't find the container with id 34e392631a5f0fda92fbb33f52c4bba19b7a2199ad34d80dd7ccd6f992dc3862
Nov 02 15:42:47.737119 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:42:47.737087    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f84ad8-789796656c-g97z7" event={"ID":"fc34fbb9-6804-4ad8-a007-25a6551a0960","Type":"ContainerStarted","Data":"eabbaf604a20bcbdc58b08d2331287047ee01c0e0f8c516efdb2c97625677a87"}
Nov 02 15:42:47.737264 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:42:47.737202    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f84ad8-789796656c-g97z7" event={"ID":"fc34fbb9-6804-4ad8-a007-25a6551a0960","Type":"ContainerStarted","Data":"1b7f6022b164e933c06198063b4a6841724dd32514f10008831710d42db8ae98"}
Nov 02 15:42:47.738069 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:42:47.738047    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f84ad8-789796656c-g97z7"
Nov 02 15:42:47.744481 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:42:47.744446    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-f84ad8-64f97947fb-zppnf" event={"ID":"5386e95b-8eef-49a6-a84e-80c8119c27fa","Type":"ContainerStarted","Data":"34e392631a5f0fda92fbb33f52c4bba19b7a2199ad34d80dd7ccd6f992dc3862"}
Nov 02 15:42:48.753470 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:42:48.753432    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-f84ad8-64f97947fb-zppnf" event={"ID":"5386e95b-8eef-49a6-a84e-80c8119c27fa","Type":"ContainerStarted","Data":"25f8e82eec34ba10a101448d30de932c80726a73af3327c7b2d2f39069f290f3"}
Nov 02 15:42:48.753874 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:42:48.753667    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-f84ad8-64f97947fb-zppnf"
Nov 02 15:42:48.798235 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:42:48.798178    5304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f84ad8-789796656c-g97z7" podStartSLOduration=2.798164119 podStartE2EDuration="2.798164119s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:42:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:42:47.78453701 +0000 UTC m=+2767.778361420" watchObservedRunningTime="2026-11-02 15:42:48.798164119 +0000 UTC m=+2768.791988524"
Nov 02 15:42:48.798553 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:42:48.798534    5304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-f84ad8-64f97947fb-zppnf" podStartSLOduration=2.798528734 podStartE2EDuration="2.798528734s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:42:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:42:48.795484126 +0000 UTC m=+2768.789308535" watchObservedRunningTime="2026-11-02 15:42:48.798528734 +0000 UTC m=+2768.792353142"
Nov 02 15:42:54.804574 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:42:54.804534    5304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k6fjk" containerName="extractor"
Nov 02 15:42:59.762316 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:42:59.762284    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f84ad8-789796656c-g97z7"
Nov 02 15:42:59.762959 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:42:59.762362    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-f84ad8-64f97947fb-zppnf"
Nov 02 15:43:24.799370 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:43:24.799159    5304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k6fjk" containerName="extractor"
Nov 02 15:43:47.279783 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:43:47.279749    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-e80c77-9dc7f56b7-mtbzn"]
Nov 02 15:43:52.285314 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:43:52.285231    5304 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-e80c77-9dc7f56b7-mtbzn" podUID="109405a5-0ac0-4964-918e-95870cfbbbc7" containerName="payload-processing" containerID="cri-o://4f13fd63e93223c481a56d354e3c049dba1033fa216cf32356ce223bc33dff07" gracePeriod=25
Nov 02 15:43:52.807498 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:43:52.807462    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-e80c77-9dc7f56b7-mtbzn"
Nov 02 15:43:52.913615 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:43:52.913534    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/109405a5-0ac0-4964-918e-95870cfbbbc7-plugins-config-volume" pod "109405a5-0ac0-4964-918e-95870cfbbbc7" (UID: "109405a5-0ac0-4964-918e-95870cfbbbc7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:43:52.913615 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:43:52.912058    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/109405a5-0ac0-4964-918e-95870cfbbbc7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/109405a5-0ac0-4964-918e-95870cfbbbc7-plugins-config-volume\") pod \"109405a5-0ac0-4964-918e-95870cfbbbc7\" (UID: \"109405a5-0ac0-4964-918e-95870cfbbbc7\") "
Nov 02 15:43:52.919572 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:43:52.919537    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/109405a5-0ac0-4964-918e-95870cfbbbc7-kube-api-access-vnshk\" (UniqueName: \"kubernetes.io/projected/109405a5-0ac0-4964-918e-95870cfbbbc7-kube-api-access-vnshk\") pod \"109405a5-0ac0-4964-918e-95870cfbbbc7\" (UID: \"109405a5-0ac0-4964-918e-95870cfbbbc7\") "
Nov 02 15:43:52.920077 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:43:52.920059    5304 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/109405a5-0ac0-4964-918e-95870cfbbbc7-plugins-config-volume\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:43:52.921917 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:43:52.921888    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/109405a5-0ac0-4964-918e-95870cfbbbc7-kube-api-access-vnshk" pod "109405a5-0ac0-4964-918e-95870cfbbbc7" (UID: "109405a5-0ac0-4964-918e-95870cfbbbc7"). InnerVolumeSpecName "kube-api-access-vnshk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:43:53.020957 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:43:53.020921    5304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vnshk\" (UniqueName: \"kubernetes.io/projected/109405a5-0ac0-4964-918e-95870cfbbbc7-kube-api-access-vnshk\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:43:53.193627 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:43:53.193529    5304 generic.go:356] "Generic (PLEG): container finished" podID="109405a5-0ac0-4964-918e-95870cfbbbc7" containerID="4f13fd63e93223c481a56d354e3c049dba1033fa216cf32356ce223bc33dff07" exitCode=0
Nov 02 15:43:53.193627 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:43:53.193582    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-e80c77-9dc7f56b7-mtbzn"
Nov 02 15:43:53.193627 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:43:53.193607    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-e80c77-9dc7f56b7-mtbzn" event={"ID":"109405a5-0ac0-4964-918e-95870cfbbbc7","Type":"ContainerDied","Data":"4f13fd63e93223c481a56d354e3c049dba1033fa216cf32356ce223bc33dff07"}
Nov 02 15:43:53.193892 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:43:53.193688    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-e80c77-9dc7f56b7-mtbzn" event={"ID":"109405a5-0ac0-4964-918e-95870cfbbbc7","Type":"ContainerDied","Data":"26541f41a48e07ba2d85e5870a5101bc103aaba6b59edc5c3c1f84606bb90497"}
Nov 02 15:43:53.193892 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:43:53.193713    5304 scope.go:122] "RemoveContainer" containerID="4f13fd63e93223c481a56d354e3c049dba1033fa216cf32356ce223bc33dff07"
Nov 02 15:43:53.205066 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:43:53.205043    5304 scope.go:122] "RemoveContainer" containerID="4f13fd63e93223c481a56d354e3c049dba1033fa216cf32356ce223bc33dff07"
Nov 02 15:43:53.205391 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:43:53.205369    5304 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f13fd63e93223c481a56d354e3c049dba1033fa216cf32356ce223bc33dff07\": container with ID starting with 4f13fd63e93223c481a56d354e3c049dba1033fa216cf32356ce223bc33dff07 not found: ID does not exist" containerID="4f13fd63e93223c481a56d354e3c049dba1033fa216cf32356ce223bc33dff07"
Nov 02 15:43:53.205449 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:43:53.205401    5304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f13fd63e93223c481a56d354e3c049dba1033fa216cf32356ce223bc33dff07"} err="failed to get container status \"4f13fd63e93223c481a56d354e3c049dba1033fa216cf32356ce223bc33dff07\": rpc error: code = NotFound desc = could not find container \"4f13fd63e93223c481a56d354e3c049dba1033fa216cf32356ce223bc33dff07\": container with ID starting with 4f13fd63e93223c481a56d354e3c049dba1033fa216cf32356ce223bc33dff07 not found: ID does not exist"
Nov 02 15:43:53.325049 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:43:53.325008    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-e80c77-9dc7f56b7-mtbzn"]
Nov 02 15:43:53.354153 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:43:53.353464    5304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-e80c77-9dc7f56b7-mtbzn"]
Nov 02 15:43:54.168494 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:43:54.168448    5304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="109405a5-0ac0-4964-918e-95870cfbbbc7" path="/var/lib/kubelet/pods/109405a5-0ac0-4964-918e-95870cfbbbc7/volumes"
Nov 02 15:43:54.817189 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:43:54.817156    5304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k6fjk" containerName="extractor"
Nov 02 15:43:59.432299 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:43:59.432255    5304 patch_prober.go:29] interesting pod/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-2wwx7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.167:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 15:43:59.432811 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:43:59.432336    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-2wwx7" podUID="e11f9d23-10db-4bad-a457-7c5db3056979" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.167:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:44:07.356324 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:44:07.356270    5304 patch_prober.go:29] interesting pod/e2e-ipp-b-3805b8-openshift-default-6c9f89869-cspmf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.166:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 15:44:07.356981 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:44:07.356367    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-cspmf" podUID="529d1f83-57ea-433f-aa66-ee16157950ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.166:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:44:14.433281 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:44:14.433235    5304 patch_prober.go:29] interesting pod/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-2wwx7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.167:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 15:44:14.433826 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:44:14.433324    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-2wwx7" podUID="e11f9d23-10db-4bad-a457-7c5db3056979" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.167:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:44:19.019779 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:44:19.019740    5304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-kgbpj"]
Nov 02 15:44:19.020422 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:44:19.020406    5304 cpu_manager.go:414] "containerMap: removing container" podUID="109405a5-0ac0-4964-918e-95870cfbbbc7" containerName="payload-processing"
Nov 02 15:44:19.020492 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:44:19.020425    5304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="109405a5-0ac0-4964-918e-95870cfbbbc7" containerName="payload-processing"
Nov 02 15:44:19.020575 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:44:19.020565    5304 memory_manager.go:365] "RemoveStaleState removing state" podUID="109405a5-0ac0-4964-918e-95870cfbbbc7" containerName="payload-processing"
Nov 02 15:44:19.069022 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:44:19.068983    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-kgbpj"
Nov 02 15:44:19.127453 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:44:19.127410    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/24182bc4-1f65-4ddf-8e20-0d5e7bb2bc29-istio-token\") pod \"e2e-ipp-b-3805b8-openshift-default-6c9f89869-kgbpj\" (UID: \"24182bc4-1f65-4ddf-8e20-0d5e7bb2bc29\") " pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-kgbpj"
Nov 02 15:44:19.156804 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:44:19.156763    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24182bc4-1f65-4ddf-8e20-0d5e7bb2bc29-istiod-ca-cert\") pod \"e2e-ipp-b-3805b8-openshift-default-6c9f89869-kgbpj\" (UID: \"24182bc4-1f65-4ddf-8e20-0d5e7bb2bc29\") " pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-kgbpj"
Nov 02 15:44:19.156977 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:44:19.156823    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24182bc4-1f65-4ddf-8e20-0d5e7bb2bc29-credential-socket\") pod \"e2e-ipp-b-3805b8-openshift-default-6c9f89869-kgbpj\" (UID: \"24182bc4-1f65-4ddf-8e20-0d5e7bb2bc29\") " pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-kgbpj"
Nov 02 15:44:19.156977 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:44:19.156921    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24182bc4-1f65-4ddf-8e20-0d5e7bb2bc29-workload-certs\") pod \"e2e-ipp-b-3805b8-openshift-default-6c9f89869-kgbpj\" (UID: \"24182bc4-1f65-4ddf-8e20-0d5e7bb2bc29\") " pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-kgbpj"
Nov 02 15:44:19.157137 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:44:19.156993    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24182bc4-1f65-4ddf-8e20-0d5e7bb2bc29-istio-envoy\") pod \"e2e-ipp-b-3805b8-openshift-default-6c9f89869-kgbpj\" (UID: \"24182bc4-1f65-4ddf-8e20-0d5e7bb2bc29\") " pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-kgbpj"
Nov 02 15:44:19.157137 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:44:19.157093    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24182bc4-1f65-4ddf-8e20-0d5e7bb2bc29-istio-podinfo\") pod \"e2e-ipp-b-3805b8-openshift-default-6c9f89869-kgbpj\" (UID: \"24182bc4-1f65-4ddf-8e20-0d5e7bb2bc29\") " pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-kgbpj"
Nov 02 15:44:19.157254 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:44:19.157224    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24182bc4-1f65-4ddf-8e20-0d5e7bb2bc29-workload-socket\") pod \"e2e-ipp-b-3805b8-openshift-default-6c9f89869-kgbpj\" (UID: \"24182bc4-1f65-4ddf-8e20-0d5e7bb2bc29\") " pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-kgbpj"
Nov 02 15:44:19.157341 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:44:19.157325    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rqjxh\" (UniqueName: \"kubernetes.io/projected/24182bc4-1f65-4ddf-8e20-0d5e7bb2bc29-kube-api-access-rqjxh\") pod \"e2e-ipp-b-3805b8-openshift-default-6c9f89869-kgbpj\" (UID: \"24182bc4-1f65-4ddf-8e20-0d5e7bb2bc29\") " pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-kgbpj"
Nov 02 15:44:19.157406 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:44:19.157392    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24182bc4-1f65-4ddf-8e20-0d5e7bb2bc29-istio-data\") pod \"e2e-ipp-b-3805b8-openshift-default-6c9f89869-kgbpj\" (UID: \"24182bc4-1f65-4ddf-8e20-0d5e7bb2bc29\") " pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-kgbpj"
Nov 02 15:44:19.258941 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:44:19.258901    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rqjxh\" (UniqueName: \"kubernetes.io/projected/24182bc4-1f65-4ddf-8e20-0d5e7bb2bc29-kube-api-access-rqjxh\") pod \"e2e-ipp-b-3805b8-openshift-default-6c9f89869-kgbpj\" (UID: \"24182bc4-1f65-4ddf-8e20-0d5e7bb2bc29\") " pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-kgbpj"
Nov 02 15:44:19.259198 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:44:19.259002    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24182bc4-1f65-4ddf-8e20-0d5e7bb2bc29-istio-data\") pod \"e2e-ipp-b-3805b8-openshift-default-6c9f89869-kgbpj\" (UID: \"24182bc4-1f65-4ddf-8e20-0d5e7bb2bc29\") " pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-kgbpj"
Nov 02 15:44:19.259198 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:44:19.259082    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/24182bc4-1f65-4ddf-8e20-0d5e7bb2bc29-istio-token\") pod \"e2e-ipp-b-3805b8-openshift-default-6c9f89869-kgbpj\" (UID: \"24182bc4-1f65-4ddf-8e20-0d5e7bb2bc29\") " pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-kgbpj"
Nov 02 15:44:19.259198 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:44:19.259117    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24182bc4-1f65-4ddf-8e20-0d5e7bb2bc29-istiod-ca-cert\") pod \"e2e-ipp-b-3805b8-openshift-default-6c9f89869-kgbpj\" (UID: \"24182bc4-1f65-4ddf-8e20-0d5e7bb2bc29\") " pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-kgbpj"
Nov 02 15:44:19.259198 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:44:19.259197    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24182bc4-1f65-4ddf-8e20-0d5e7bb2bc29-credential-socket\") pod \"e2e-ipp-b-3805b8-openshift-default-6c9f89869-kgbpj\" (UID: \"24182bc4-1f65-4ddf-8e20-0d5e7bb2bc29\") " pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-kgbpj"
Nov 02 15:44:19.259455 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:44:19.259269    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24182bc4-1f65-4ddf-8e20-0d5e7bb2bc29-workload-certs\") pod \"e2e-ipp-b-3805b8-openshift-default-6c9f89869-kgbpj\" (UID: \"24182bc4-1f65-4ddf-8e20-0d5e7bb2bc29\") " pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-kgbpj"
Nov 02 15:44:19.259455 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:44:19.259394    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24182bc4-1f65-4ddf-8e20-0d5e7bb2bc29-istio-envoy\") pod \"e2e-ipp-b-3805b8-openshift-default-6c9f89869-kgbpj\" (UID: \"24182bc4-1f65-4ddf-8e20-0d5e7bb2bc29\") " pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-kgbpj"
Nov 02 15:44:19.259455 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:44:19.259425    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24182bc4-1f65-4ddf-8e20-0d5e7bb2bc29-istio-podinfo\") pod \"e2e-ipp-b-3805b8-openshift-default-6c9f89869-kgbpj\" (UID: \"24182bc4-1f65-4ddf-8e20-0d5e7bb2bc29\") " pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-kgbpj"
Nov 02 15:44:19.259635 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:44:19.259484    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24182bc4-1f65-4ddf-8e20-0d5e7bb2bc29-workload-socket\") pod \"e2e-ipp-b-3805b8-openshift-default-6c9f89869-kgbpj\" (UID: \"24182bc4-1f65-4ddf-8e20-0d5e7bb2bc29\") " pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-kgbpj"
Nov 02 15:44:19.259635 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:44:19.259517    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24182bc4-1f65-4ddf-8e20-0d5e7bb2bc29-istio-data\") pod \"e2e-ipp-b-3805b8-openshift-default-6c9f89869-kgbpj\" (UID: \"24182bc4-1f65-4ddf-8e20-0d5e7bb2bc29\") " pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-kgbpj"
Nov 02 15:44:19.259635 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:44:19.259613    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24182bc4-1f65-4ddf-8e20-0d5e7bb2bc29-workload-certs\") pod \"e2e-ipp-b-3805b8-openshift-default-6c9f89869-kgbpj\" (UID: \"24182bc4-1f65-4ddf-8e20-0d5e7bb2bc29\") " pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-kgbpj"
Nov 02 15:44:19.259880 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:44:19.259859    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24182bc4-1f65-4ddf-8e20-0d5e7bb2bc29-workload-socket\") pod \"e2e-ipp-b-3805b8-openshift-default-6c9f89869-kgbpj\" (UID: \"24182bc4-1f65-4ddf-8e20-0d5e7bb2bc29\") " pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-kgbpj"
Nov 02 15:44:19.260038 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:44:19.260015    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24182bc4-1f65-4ddf-8e20-0d5e7bb2bc29-istiod-ca-cert\") pod \"e2e-ipp-b-3805b8-openshift-default-6c9f89869-kgbpj\" (UID: \"24182bc4-1f65-4ddf-8e20-0d5e7bb2bc29\") " pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-kgbpj"
Nov 02 15:44:19.260113 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:44:19.260015    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24182bc4-1f65-4ddf-8e20-0d5e7bb2bc29-credential-socket\") pod \"e2e-ipp-b-3805b8-openshift-default-6c9f89869-kgbpj\" (UID: \"24182bc4-1f65-4ddf-8e20-0d5e7bb2bc29\") " pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-kgbpj"
Nov 02 15:44:19.354115 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:44:19.354076    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24182bc4-1f65-4ddf-8e20-0d5e7bb2bc29-istio-envoy\") pod \"e2e-ipp-b-3805b8-openshift-default-6c9f89869-kgbpj\" (UID: \"24182bc4-1f65-4ddf-8e20-0d5e7bb2bc29\") " pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-kgbpj"
Nov 02 15:44:19.354433 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:44:19.354410    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24182bc4-1f65-4ddf-8e20-0d5e7bb2bc29-istio-podinfo\") pod \"e2e-ipp-b-3805b8-openshift-default-6c9f89869-kgbpj\" (UID: \"24182bc4-1f65-4ddf-8e20-0d5e7bb2bc29\") " pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-kgbpj"
Nov 02 15:44:19.354522 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:44:19.354496    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rqjxh\" (UniqueName: \"kubernetes.io/projected/24182bc4-1f65-4ddf-8e20-0d5e7bb2bc29-kube-api-access-rqjxh\") pod \"e2e-ipp-b-3805b8-openshift-default-6c9f89869-kgbpj\" (UID: \"24182bc4-1f65-4ddf-8e20-0d5e7bb2bc29\") " pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-kgbpj"
Nov 02 15:44:19.354670 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:44:19.354647    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/24182bc4-1f65-4ddf-8e20-0d5e7bb2bc29-istio-token\") pod \"e2e-ipp-b-3805b8-openshift-default-6c9f89869-kgbpj\" (UID: \"24182bc4-1f65-4ddf-8e20-0d5e7bb2bc29\") " pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-kgbpj"
Nov 02 15:44:19.470648 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:44:19.470610    5304 generic.go:356] "Generic (PLEG): container finished" podID="529d1f83-57ea-433f-aa66-ee16157950ca" containerID="6774500a1af6c9e7e9c101c43bd4da5839fc192191732fd0aff2eba2c0b62600" exitCode=0
Nov 02 15:44:19.470822 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:44:19.470674    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-cspmf" event={"ID":"529d1f83-57ea-433f-aa66-ee16157950ca","Type":"ContainerDied","Data":"6774500a1af6c9e7e9c101c43bd4da5839fc192191732fd0aff2eba2c0b62600"}
Nov 02 15:44:19.471207 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:44:19.471186    5304 scope.go:122] "RemoveContainer" containerID="6774500a1af6c9e7e9c101c43bd4da5839fc192191732fd0aff2eba2c0b62600"
Nov 02 15:44:19.505898 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:44:19.505854    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:44:19.506044 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:44:19.505935    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:44:19.506044 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:44:19.505969    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:44:19.564350 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:44:19.564314    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-kgbpj"
Nov 02 15:44:19.741949 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:44:19.741920    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-kgbpj"]
Nov 02 15:44:20.479469 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:44:20.479429    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-cspmf" event={"ID":"529d1f83-57ea-433f-aa66-ee16157950ca","Type":"ContainerStarted","Data":"0f9dd4cca6d1f8bfba9d564ba424fd7c53f2bbc5d2e6bfcdbed7eeafd8ec2ce0"}
Nov 02 15:44:20.843287 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:44:20.843258    5304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-cspmf"
Nov 02 15:44:21.284324 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:44:21.283731    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-kgbpj"]
Nov 02 15:44:21.300916 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:44:21.300871    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:44:21.301086 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:44:21.300952    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:44:21.301086 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:44:21.300988    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:44:21.314495 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:44:21.314471    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-cspmf"
Nov 02 15:44:21.487272 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:44:21.487232    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-kgbpj" event={"ID":"24182bc4-1f65-4ddf-8e20-0d5e7bb2bc29","Type":"ContainerStarted","Data":"cb277df466b41db1518a1cc36631b2c07ddd7dd2343aff7300f8040eacc82218"}
Nov 02 15:44:21.858332 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:44:21.858258    5304 patch_prober.go:29] interesting pod/e2e-ipp-b-3805b8-openshift-default-6c9f89869-cspmf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused" start-of-body=
Nov 02 15:44:21.858732 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:44:21.858510    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-cspmf" podUID="529d1f83-57ea-433f-aa66-ee16157950ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused"
Nov 02 15:44:22.494895 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:44:22.494861    5304 generic.go:356] "Generic (PLEG): container finished" podID="e11f9d23-10db-4bad-a457-7c5db3056979" containerID="82afe6e52444f7b0e63d73c694b765f1e4a10e6f6969c22dddf3cbbce5aedf44" exitCode=0
Nov 02 15:44:22.495322 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:44:22.495043    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-2wwx7" event={"ID":"e11f9d23-10db-4bad-a457-7c5db3056979","Type":"ContainerDied","Data":"82afe6e52444f7b0e63d73c694b765f1e4a10e6f6969c22dddf3cbbce5aedf44"}
Nov 02 15:44:22.495481 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:44:22.495463    5304 scope.go:122] "RemoveContainer" containerID="82afe6e52444f7b0e63d73c694b765f1e4a10e6f6969c22dddf3cbbce5aedf44"
Nov 02 15:44:22.497271 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:44:22.497244    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-kgbpj" event={"ID":"24182bc4-1f65-4ddf-8e20-0d5e7bb2bc29","Type":"ContainerStarted","Data":"77d58210cbb297d5ba13193f55f76b17a9e94243126efd9051a88b97200745f9"}
Nov 02 15:44:22.497750 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:44:22.497717    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:44:22.497845 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:44:22.497828    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:44:22.497892 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:44:22.497878    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:44:22.567295 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:44:22.567214    5304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-kgbpj"
Nov 02 15:44:22.586151 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:44:22.576773    5304 patch_prober.go:29] interesting pod/e2e-ipp-b-3805b8-openshift-default-6c9f89869-kgbpj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused" start-of-body=
Nov 02 15:44:22.586151 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:44:22.576861    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-kgbpj" podUID="24182bc4-1f65-4ddf-8e20-0d5e7bb2bc29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused"
Nov 02 15:44:22.612587 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:44:22.612428    5304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-kgbpj" podStartSLOduration=4.612407384 podStartE2EDuration="4.612407384s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:44:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:44:22.610763998 +0000 UTC m=+2862.604588407" watchObservedRunningTime="2026-11-02 15:44:22.612407384 +0000 UTC m=+2862.606231798"
Nov 02 15:44:22.863823 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:44:22.849915    5304 patch_prober.go:29] interesting pod/e2e-ipp-b-3805b8-openshift-default-6c9f89869-cspmf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused" start-of-body=
Nov 02 15:44:22.863823 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:44:22.849979    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-cspmf" podUID="529d1f83-57ea-433f-aa66-ee16157950ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused"
Nov 02 15:44:23.508460 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:44:23.508415    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-2wwx7" event={"ID":"e11f9d23-10db-4bad-a457-7c5db3056979","Type":"ContainerStarted","Data":"b49da12e9c1eaf625d860f22d16960d3837c0f77d80f408d1da9d5f6c30a3835"}
Nov 02 15:44:23.575148 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:44:23.571519    5304 patch_prober.go:29] interesting pod/e2e-ipp-b-3805b8-openshift-default-6c9f89869-kgbpj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused" start-of-body=
Nov 02 15:44:23.575148 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:44:23.571582    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-kgbpj" podUID="24182bc4-1f65-4ddf-8e20-0d5e7bb2bc29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused"
Nov 02 15:44:23.851150 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:44:23.850578    5304 patch_prober.go:29] interesting pod/e2e-ipp-b-3805b8-openshift-default-6c9f89869-cspmf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused" start-of-body=
Nov 02 15:44:23.851150 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:44:23.850642    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-cspmf" podUID="529d1f83-57ea-433f-aa66-ee16157950ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused"
Nov 02 15:44:24.060530 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:44:24.060493    5304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-2wwx7"
Nov 02 15:44:24.065151 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:44:24.062251    5304 patch_prober.go:29] interesting pod/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-2wwx7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" start-of-body=
Nov 02 15:44:24.065151 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:44:24.062312    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-2wwx7" podUID="e11f9d23-10db-4bad-a457-7c5db3056979" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused"
Nov 02 15:44:24.566625 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:44:24.566582    5304 patch_prober.go:29] interesting pod/e2e-ipp-b-3805b8-openshift-default-6c9f89869-kgbpj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused" start-of-body=
Nov 02 15:44:24.567288 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:44:24.567251    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-kgbpj" podUID="24182bc4-1f65-4ddf-8e20-0d5e7bb2bc29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused"
Nov 02 15:44:24.853148 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:44:24.850702    5304 patch_prober.go:29] interesting pod/e2e-ipp-b-3805b8-openshift-default-6c9f89869-cspmf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused" start-of-body=
Nov 02 15:44:24.853148 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:44:24.850776    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-cspmf" podUID="529d1f83-57ea-433f-aa66-ee16157950ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused"
Nov 02 15:44:24.927317 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:44:24.926777    5304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k6fjk" containerName="extractor"
Nov 02 15:44:25.069292 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:44:25.069249    5304 patch_prober.go:29] interesting pod/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-2wwx7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" start-of-body=
Nov 02 15:44:25.069610 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:44:25.069556    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-2wwx7" podUID="e11f9d23-10db-4bad-a457-7c5db3056979" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused"
Nov 02 15:44:25.566149 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:44:25.565474    5304 patch_prober.go:29] interesting pod/e2e-ipp-b-3805b8-openshift-default-6c9f89869-kgbpj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused" start-of-body=
Nov 02 15:44:25.566149 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:44:25.565548    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-kgbpj" podUID="24182bc4-1f65-4ddf-8e20-0d5e7bb2bc29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused"
Nov 02 15:44:25.845579 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:44:25.845541    5304 patch_prober.go:29] interesting pod/e2e-ipp-b-3805b8-openshift-default-6c9f89869-cspmf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused" start-of-body=
Nov 02 15:44:25.846241 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:44:25.846206    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-cspmf" podUID="529d1f83-57ea-433f-aa66-ee16157950ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused"
Nov 02 15:44:26.080565 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:44:26.080525    5304 patch_prober.go:29] interesting pod/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-2wwx7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" start-of-body=
Nov 02 15:44:26.080852 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:44:26.080822    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-2wwx7" podUID="e11f9d23-10db-4bad-a457-7c5db3056979" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused"
Nov 02 15:44:26.456151 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:44:26.455562    5304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-3805b8-68df779859-spzxb"]
Nov 02 15:44:26.475152 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:44:26.472419    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3805b8-68df779859-spzxb"
Nov 02 15:44:26.514113 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:44:26.514062    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-3805b8-68df779859-spzxb"]
Nov 02 15:44:26.529620 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:44:26.515595    5304 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-3805b8\""
Nov 02 15:44:26.541193 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:44:26.530349    5304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-3805b8-dockercfg-822kz\""
Nov 02 15:44:26.644297 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:44:26.587470    5304 patch_prober.go:29] interesting pod/e2e-ipp-b-3805b8-openshift-default-6c9f89869-kgbpj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused" start-of-body=
Nov 02 15:44:26.644297 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:44:26.587527    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-kgbpj" podUID="24182bc4-1f65-4ddf-8e20-0d5e7bb2bc29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused"
Nov 02 15:44:26.644297 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:44:26.589528    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7a7782c-d47c-4ee8-b352-9fe2c6b2ee45-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-3805b8-68df779859-spzxb\" (UID: \"a7a7782c-d47c-4ee8-b352-9fe2c6b2ee45\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3805b8-68df779859-spzxb"
Nov 02 15:44:26.644297 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:44:26.589584    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tk9mf\" (UniqueName: \"kubernetes.io/projected/a7a7782c-d47c-4ee8-b352-9fe2c6b2ee45-kube-api-access-tk9mf\") pod \"payload-pre-processing-e2e-ipp-b-3805b8-68df779859-spzxb\" (UID: \"a7a7782c-d47c-4ee8-b352-9fe2c6b2ee45\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3805b8-68df779859-spzxb"
Nov 02 15:44:26.722975 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:44:26.721019    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7a7782c-d47c-4ee8-b352-9fe2c6b2ee45-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-3805b8-68df779859-spzxb\" (UID: \"a7a7782c-d47c-4ee8-b352-9fe2c6b2ee45\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3805b8-68df779859-spzxb"
Nov 02 15:44:26.722975 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:44:26.721092    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tk9mf\" (UniqueName: \"kubernetes.io/projected/a7a7782c-d47c-4ee8-b352-9fe2c6b2ee45-kube-api-access-tk9mf\") pod \"payload-pre-processing-e2e-ipp-b-3805b8-68df779859-spzxb\" (UID: \"a7a7782c-d47c-4ee8-b352-9fe2c6b2ee45\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3805b8-68df779859-spzxb"
Nov 02 15:44:26.722975 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:44:26.722571    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7a7782c-d47c-4ee8-b352-9fe2c6b2ee45-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-3805b8-68df779859-spzxb\" (UID: \"a7a7782c-d47c-4ee8-b352-9fe2c6b2ee45\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3805b8-68df779859-spzxb"
Nov 02 15:44:26.881696 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:44:26.881215    5304 patch_prober.go:29] interesting pod/e2e-ipp-b-3805b8-openshift-default-6c9f89869-cspmf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused" start-of-body=
Nov 02 15:44:26.881696 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:44:26.881275    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-cspmf" podUID="529d1f83-57ea-433f-aa66-ee16157950ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused"
Nov 02 15:44:26.961152 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:44:26.946945    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tk9mf\" (UniqueName: \"kubernetes.io/projected/a7a7782c-d47c-4ee8-b352-9fe2c6b2ee45-kube-api-access-tk9mf\") pod \"payload-pre-processing-e2e-ipp-b-3805b8-68df779859-spzxb\" (UID: \"a7a7782c-d47c-4ee8-b352-9fe2c6b2ee45\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3805b8-68df779859-spzxb"
Nov 02 15:44:27.061210 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:44:27.060610    5304 patch_prober.go:29] interesting pod/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-2wwx7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" start-of-body=
Nov 02 15:44:27.061210 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:44:27.060673    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-2wwx7" podUID="e11f9d23-10db-4bad-a457-7c5db3056979" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused"
Nov 02 15:44:27.102167 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:44:27.091466    5304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-3805b8-6c6d4cd55b-fbstm"]
Nov 02 15:44:27.185150 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:44:27.184926    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3805b8-68df779859-spzxb"
Nov 02 15:44:27.187158 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:44:27.186365    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-3805b8-6c6d4cd55b-fbstm"]
Nov 02 15:44:27.187158 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:44:27.186508    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-3805b8-6c6d4cd55b-fbstm"
Nov 02 15:44:27.313221 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:44:27.313103    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xxqh5\" (UniqueName: \"kubernetes.io/projected/53a0141a-7572-475e-adac-56ed64e21bcc-kube-api-access-xxqh5\") pod \"payload-processing-e2e-ipp-b-3805b8-6c6d4cd55b-fbstm\" (UID: \"53a0141a-7572-475e-adac-56ed64e21bcc\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-3805b8-6c6d4cd55b-fbstm"
Nov 02 15:44:27.313221 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:44:27.313185    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53a0141a-7572-475e-adac-56ed64e21bcc-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-3805b8-6c6d4cd55b-fbstm\" (UID: \"53a0141a-7572-475e-adac-56ed64e21bcc\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-3805b8-6c6d4cd55b-fbstm"
Nov 02 15:44:27.416149 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:44:27.414958    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53a0141a-7572-475e-adac-56ed64e21bcc-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-3805b8-6c6d4cd55b-fbstm\" (UID: \"53a0141a-7572-475e-adac-56ed64e21bcc\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-3805b8-6c6d4cd55b-fbstm"
Nov 02 15:44:27.416149 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:44:27.415308    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xxqh5\" (UniqueName: \"kubernetes.io/projected/53a0141a-7572-475e-adac-56ed64e21bcc-kube-api-access-xxqh5\") pod \"payload-processing-e2e-ipp-b-3805b8-6c6d4cd55b-fbstm\" (UID: \"53a0141a-7572-475e-adac-56ed64e21bcc\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-3805b8-6c6d4cd55b-fbstm"
Nov 02 15:44:27.423164 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:44:27.416594    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53a0141a-7572-475e-adac-56ed64e21bcc-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-3805b8-6c6d4cd55b-fbstm\" (UID: \"53a0141a-7572-475e-adac-56ed64e21bcc\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-3805b8-6c6d4cd55b-fbstm"
Nov 02 15:44:27.476865 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:44:27.474905    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xxqh5\" (UniqueName: \"kubernetes.io/projected/53a0141a-7572-475e-adac-56ed64e21bcc-kube-api-access-xxqh5\") pod \"payload-processing-e2e-ipp-b-3805b8-6c6d4cd55b-fbstm\" (UID: \"53a0141a-7572-475e-adac-56ed64e21bcc\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-3805b8-6c6d4cd55b-fbstm"
Nov 02 15:44:27.520976 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:44:27.520716    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-3805b8-6c6d4cd55b-fbstm"
Nov 02 15:44:27.606165 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:44:27.592700    5304 patch_prober.go:29] interesting pod/e2e-ipp-b-3805b8-openshift-default-6c9f89869-kgbpj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused" start-of-body=
Nov 02 15:44:27.606165 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:44:27.592761    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-kgbpj" podUID="24182bc4-1f65-4ddf-8e20-0d5e7bb2bc29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused"
Nov 02 15:44:27.804882 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:44:27.804829    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-3805b8-68df779859-spzxb"]
Nov 02 15:44:28.006267 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:44:28.005375    5304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-cspmf"
Nov 02 15:44:28.010555 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:44:28.009795    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-cspmf"
Nov 02 15:44:28.061566 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:44:28.061268    5304 patch_prober.go:29] interesting pod/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-2wwx7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" start-of-body=
Nov 02 15:44:28.061566 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:44:28.061329    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-2wwx7" podUID="e11f9d23-10db-4bad-a457-7c5db3056979" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused"
Nov 02 15:44:28.097625 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:44:28.097594    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-3805b8-6c6d4cd55b-fbstm"]
Nov 02 15:44:28.390511 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:44:28.390409    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-2wwx7"
Nov 02 15:44:28.689269 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:44:28.689180    5304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-kgbpj"
Nov 02 15:44:28.690357 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:44:28.690327    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-kgbpj"
Nov 02 15:44:28.691834 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:44:28.691813    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-kgbpj"
Nov 02 15:44:28.918166 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:44:28.914576    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-3805b8-6c6d4cd55b-fbstm" event={"ID":"53a0141a-7572-475e-adac-56ed64e21bcc","Type":"ContainerStarted","Data":"dfe92a495a2297d5e256eb7e3f4ff821895bca20955e66434b3ecd2582b4ceba"}
Nov 02 15:44:28.918166 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:44:28.914655    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-3805b8-6c6d4cd55b-fbstm" event={"ID":"53a0141a-7572-475e-adac-56ed64e21bcc","Type":"ContainerStarted","Data":"3d86a46009c32d20d7f4edef81f2cffd73ca12545c906ecd12f51b1878fa4ee3"}
Nov 02 15:44:28.918166 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:44:28.915231    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-3805b8-6c6d4cd55b-fbstm"
Nov 02 15:44:28.918166 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:44:28.917016    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3805b8-68df779859-spzxb" event={"ID":"a7a7782c-d47c-4ee8-b352-9fe2c6b2ee45","Type":"ContainerStarted","Data":"361cad4c2af48e0b49b06a72fe2c4445417d03263acf67aefd55ec05c54f04d3"}
Nov 02 15:44:28.918166 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:44:28.917059    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3805b8-68df779859-spzxb" event={"ID":"a7a7782c-d47c-4ee8-b352-9fe2c6b2ee45","Type":"ContainerStarted","Data":"dbb82366148a06cdaac0b5e218638357c284a2405ec94134d3eb6a41acb1da2a"}
Nov 02 15:44:28.918166 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:44:28.917163    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3805b8-68df779859-spzxb"
Nov 02 15:44:28.989099 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:44:28.988986    5304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-3805b8-6c6d4cd55b-fbstm" podStartSLOduration=2.988964775 podStartE2EDuration="2.988964775s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:44:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:44:28.959519472 +0000 UTC m=+2868.953343882" watchObservedRunningTime="2026-11-02 15:44:28.988964775 +0000 UTC m=+2868.982789186"
Nov 02 15:44:29.032154 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:44:29.031648    5304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3805b8-68df779859-spzxb" podStartSLOduration=3.031626706 podStartE2EDuration="3.031626706s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:44:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:44:29.025747716 +0000 UTC m=+2869.019572126" watchObservedRunningTime="2026-11-02 15:44:29.031626706 +0000 UTC m=+2869.025451117"
Nov 02 15:44:29.062551 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:44:29.062500    5304 patch_prober.go:29] interesting pod/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-2wwx7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" start-of-body=
Nov 02 15:44:29.062777 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:44:29.062565    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-2wwx7" podUID="e11f9d23-10db-4bad-a457-7c5db3056979" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused"
Nov 02 15:44:30.192306 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:44:30.192261    5304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-2wwx7"
Nov 02 15:44:30.194651 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:44:30.194618    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-2wwx7"
Nov 02 15:44:36.024148 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:44:36.018907    5304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-e80c77-dc5cd4d99-plkbt"]
Nov 02 15:44:36.053744 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:44:36.053712    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-e80c77-dc5cd4d99-plkbt"]
Nov 02 15:44:36.054144 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:44:36.054057    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e80c77-dc5cd4d99-plkbt"
Nov 02 15:44:36.090586 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:44:36.090560    5304 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-e80c77\""
Nov 02 15:44:36.113614 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:44:36.113568    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/36430ec8-da3a-4166-b04c-3671a1f3de3f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-e80c77-dc5cd4d99-plkbt\" (UID: \"36430ec8-da3a-4166-b04c-3671a1f3de3f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e80c77-dc5cd4d99-plkbt"
Nov 02 15:44:36.113898 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:44:36.113879    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7z9g\" (UniqueName: \"kubernetes.io/projected/36430ec8-da3a-4166-b04c-3671a1f3de3f-kube-api-access-s7z9g\") pod \"payload-pre-processing-e2e-ipp-b-e80c77-dc5cd4d99-plkbt\" (UID: \"36430ec8-da3a-4166-b04c-3671a1f3de3f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e80c77-dc5cd4d99-plkbt"
Nov 02 15:44:36.125645 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:44:36.125619    5304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-e80c77-dockercfg-pjkdp\""
Nov 02 15:44:36.215997 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:44:36.215950    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/36430ec8-da3a-4166-b04c-3671a1f3de3f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-e80c77-dc5cd4d99-plkbt\" (UID: \"36430ec8-da3a-4166-b04c-3671a1f3de3f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e80c77-dc5cd4d99-plkbt"
Nov 02 15:44:36.216218 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:44:36.216027    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7z9g\" (UniqueName: \"kubernetes.io/projected/36430ec8-da3a-4166-b04c-3671a1f3de3f-kube-api-access-s7z9g\") pod \"payload-pre-processing-e2e-ipp-b-e80c77-dc5cd4d99-plkbt\" (UID: \"36430ec8-da3a-4166-b04c-3671a1f3de3f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e80c77-dc5cd4d99-plkbt"
Nov 02 15:44:36.216942 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:44:36.216911    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/36430ec8-da3a-4166-b04c-3671a1f3de3f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-e80c77-dc5cd4d99-plkbt\" (UID: \"36430ec8-da3a-4166-b04c-3671a1f3de3f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e80c77-dc5cd4d99-plkbt"
Nov 02 15:44:36.247084 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:44:36.247050    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7z9g\" (UniqueName: \"kubernetes.io/projected/36430ec8-da3a-4166-b04c-3671a1f3de3f-kube-api-access-s7z9g\") pod \"payload-pre-processing-e2e-ipp-b-e80c77-dc5cd4d99-plkbt\" (UID: \"36430ec8-da3a-4166-b04c-3671a1f3de3f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e80c77-dc5cd4d99-plkbt"
Nov 02 15:44:36.425634 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:44:36.425595    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e80c77-dc5cd4d99-plkbt"
Nov 02 15:44:36.742223 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:44:36.742135    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-e80c77-dc5cd4d99-plkbt"]
Nov 02 15:44:36.745491 ip-10-0-135-138 kubenswrapper[5304]: W1102 15:44:36.745458    5304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36430ec8_da3a_4166_b04c_3671a1f3de3f.slice/crio-a55adb2dd1f93b197953cee12e348f10596fb74b41a2d70a58597a70cf3f3567 WatchSource:0}: Error finding container a55adb2dd1f93b197953cee12e348f10596fb74b41a2d70a58597a70cf3f3567: Status 404 returned error can't find the container with id a55adb2dd1f93b197953cee12e348f10596fb74b41a2d70a58597a70cf3f3567
Nov 02 15:44:36.991868 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:44:36.991824    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e80c77-dc5cd4d99-plkbt" event={"ID":"36430ec8-da3a-4166-b04c-3671a1f3de3f","Type":"ContainerStarted","Data":"632eef1c1f0ceb32dfd7180dc268d4f3f47714f519a1efde73200187082d863c"}
Nov 02 15:44:36.992056 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:44:36.991885    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e80c77-dc5cd4d99-plkbt" event={"ID":"36430ec8-da3a-4166-b04c-3671a1f3de3f","Type":"ContainerStarted","Data":"a55adb2dd1f93b197953cee12e348f10596fb74b41a2d70a58597a70cf3f3567"}
Nov 02 15:44:36.992056 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:44:36.991914    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e80c77-dc5cd4d99-plkbt"
Nov 02 15:44:37.036176 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:44:37.036031    5304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e80c77-dc5cd4d99-plkbt" podStartSLOduration=2.036011362 podStartE2EDuration="2.036011362s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:44:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:44:37.033556528 +0000 UTC m=+2877.027380937" watchObservedRunningTime="2026-11-02 15:44:37.036011362 +0000 UTC m=+2877.029835773"
Nov 02 15:44:39.931982 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:44:39.931949    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3805b8-68df779859-spzxb"
Nov 02 15:44:40.939854 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:44:40.939822    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-3805b8-6c6d4cd55b-fbstm"
Nov 02 15:44:48.000748 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:44:48.000715    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e80c77-dc5cd4d99-plkbt"
Nov 02 15:44:54.799875 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:44:54.799787    5304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k6fjk" containerName="extractor"
Nov 02 15:44:56.329864 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:44:56.328164    5304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-lvrfc"]
Nov 02 15:44:56.348452 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:44:56.348423    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-lvrfc"
Nov 02 15:44:56.368962 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:44:56.368796    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-lvrfc"]
Nov 02 15:44:56.381051 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:44:56.381003    5304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-smtdr"]
Nov 02 15:44:56.385197 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:44:56.384911    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-smtdr"
Nov 02 15:44:56.420274 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:44:56.420241    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-smtdr"]
Nov 02 15:44:56.521033 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:44:56.521002    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sv2vr\" (UniqueName: \"kubernetes.io/projected/93e559b7-2a4c-4533-90d5-5f3fd3956bf0-kube-api-access-sv2vr\") pod \"e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-lvrfc\" (UID: \"93e559b7-2a4c-4533-90d5-5f3fd3956bf0\") " pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-lvrfc"
Nov 02 15:44:56.521033 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:44:56.521038    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93e559b7-2a4c-4533-90d5-5f3fd3956bf0-workload-certs\") pod \"e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-lvrfc\" (UID: \"93e559b7-2a4c-4533-90d5-5f3fd3956bf0\") " pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-lvrfc"
Nov 02 15:44:56.521315 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:44:56.521057    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93e559b7-2a4c-4533-90d5-5f3fd3956bf0-istio-envoy\") pod \"e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-lvrfc\" (UID: \"93e559b7-2a4c-4533-90d5-5f3fd3956bf0\") " pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-lvrfc"
Nov 02 15:44:56.521315 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:44:56.521146    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5086f7a8-e10d-4aca-bc70-950a2f0cd585-credential-socket\") pod \"e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-smtdr\" (UID: \"5086f7a8-e10d-4aca-bc70-950a2f0cd585\") " pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-smtdr"
Nov 02 15:44:56.521315 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:44:56.521191    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93e559b7-2a4c-4533-90d5-5f3fd3956bf0-workload-socket\") pod \"e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-lvrfc\" (UID: \"93e559b7-2a4c-4533-90d5-5f3fd3956bf0\") " pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-lvrfc"
Nov 02 15:44:56.521315 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:44:56.521259    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93e559b7-2a4c-4533-90d5-5f3fd3956bf0-istiod-ca-cert\") pod \"e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-lvrfc\" (UID: \"93e559b7-2a4c-4533-90d5-5f3fd3956bf0\") " pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-lvrfc"
Nov 02 15:44:56.521958 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:44:56.521341    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5086f7a8-e10d-4aca-bc70-950a2f0cd585-workload-socket\") pod \"e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-smtdr\" (UID: \"5086f7a8-e10d-4aca-bc70-950a2f0cd585\") " pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-smtdr"
Nov 02 15:44:56.521958 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:44:56.521397    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/93e559b7-2a4c-4533-90d5-5f3fd3956bf0-istio-token\") pod \"e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-lvrfc\" (UID: \"93e559b7-2a4c-4533-90d5-5f3fd3956bf0\") " pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-lvrfc"
Nov 02 15:44:56.521958 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:44:56.521461    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93e559b7-2a4c-4533-90d5-5f3fd3956bf0-credential-socket\") pod \"e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-lvrfc\" (UID: \"93e559b7-2a4c-4533-90d5-5f3fd3956bf0\") " pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-lvrfc"
Nov 02 15:44:56.521958 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:44:56.521516    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5086f7a8-e10d-4aca-bc70-950a2f0cd585-istio-data\") pod \"e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-smtdr\" (UID: \"5086f7a8-e10d-4aca-bc70-950a2f0cd585\") " pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-smtdr"
Nov 02 15:44:56.521958 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:44:56.521643    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93e559b7-2a4c-4533-90d5-5f3fd3956bf0-istio-podinfo\") pod \"e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-lvrfc\" (UID: \"93e559b7-2a4c-4533-90d5-5f3fd3956bf0\") " pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-lvrfc"
Nov 02 15:44:56.521958 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:44:56.521674    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5086f7a8-e10d-4aca-bc70-950a2f0cd585-istio-envoy\") pod \"e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-smtdr\" (UID: \"5086f7a8-e10d-4aca-bc70-950a2f0cd585\") " pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-smtdr"
Nov 02 15:44:56.521958 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:44:56.521733    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vrvm8\" (UniqueName: \"kubernetes.io/projected/5086f7a8-e10d-4aca-bc70-950a2f0cd585-kube-api-access-vrvm8\") pod \"e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-smtdr\" (UID: \"5086f7a8-e10d-4aca-bc70-950a2f0cd585\") " pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-smtdr"
Nov 02 15:44:56.521958 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:44:56.521830    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93e559b7-2a4c-4533-90d5-5f3fd3956bf0-istio-data\") pod \"e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-lvrfc\" (UID: \"93e559b7-2a4c-4533-90d5-5f3fd3956bf0\") " pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-lvrfc"
Nov 02 15:44:56.521958 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:44:56.521867    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5086f7a8-e10d-4aca-bc70-950a2f0cd585-istio-token\") pod \"e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-smtdr\" (UID: \"5086f7a8-e10d-4aca-bc70-950a2f0cd585\") " pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-smtdr"
Nov 02 15:44:56.522369 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:44:56.521965    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5086f7a8-e10d-4aca-bc70-950a2f0cd585-istiod-ca-cert\") pod \"e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-smtdr\" (UID: \"5086f7a8-e10d-4aca-bc70-950a2f0cd585\") " pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-smtdr"
Nov 02 15:44:56.522369 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:44:56.521997    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5086f7a8-e10d-4aca-bc70-950a2f0cd585-istio-podinfo\") pod \"e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-smtdr\" (UID: \"5086f7a8-e10d-4aca-bc70-950a2f0cd585\") " pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-smtdr"
Nov 02 15:44:56.522369 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:44:56.522099    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5086f7a8-e10d-4aca-bc70-950a2f0cd585-workload-certs\") pod \"e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-smtdr\" (UID: \"5086f7a8-e10d-4aca-bc70-950a2f0cd585\") " pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-smtdr"
Nov 02 15:44:56.635815 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:44:56.635732    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93e559b7-2a4c-4533-90d5-5f3fd3956bf0-istio-podinfo\") pod \"e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-lvrfc\" (UID: \"93e559b7-2a4c-4533-90d5-5f3fd3956bf0\") " pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-lvrfc"
Nov 02 15:44:56.635815 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:44:56.635792    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5086f7a8-e10d-4aca-bc70-950a2f0cd585-istio-envoy\") pod \"e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-smtdr\" (UID: \"5086f7a8-e10d-4aca-bc70-950a2f0cd585\") " pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-smtdr"
Nov 02 15:44:56.636047 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:44:56.635843    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vrvm8\" (UniqueName: \"kubernetes.io/projected/5086f7a8-e10d-4aca-bc70-950a2f0cd585-kube-api-access-vrvm8\") pod \"e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-smtdr\" (UID: \"5086f7a8-e10d-4aca-bc70-950a2f0cd585\") " pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-smtdr"
Nov 02 15:44:56.636047 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:44:56.635868    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93e559b7-2a4c-4533-90d5-5f3fd3956bf0-istio-data\") pod \"e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-lvrfc\" (UID: \"93e559b7-2a4c-4533-90d5-5f3fd3956bf0\") " pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-lvrfc"
Nov 02 15:44:56.636047 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:44:56.635959    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5086f7a8-e10d-4aca-bc70-950a2f0cd585-istio-token\") pod \"e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-smtdr\" (UID: \"5086f7a8-e10d-4aca-bc70-950a2f0cd585\") " pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-smtdr"
Nov 02 15:44:56.636275 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:44:56.636113    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5086f7a8-e10d-4aca-bc70-950a2f0cd585-istiod-ca-cert\") pod \"e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-smtdr\" (UID: \"5086f7a8-e10d-4aca-bc70-950a2f0cd585\") " pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-smtdr"
Nov 02 15:44:56.636802 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:44:56.636777    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93e559b7-2a4c-4533-90d5-5f3fd3956bf0-istio-data\") pod \"e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-lvrfc\" (UID: \"93e559b7-2a4c-4533-90d5-5f3fd3956bf0\") " pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-lvrfc"
Nov 02 15:44:56.636989 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:44:56.636204    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5086f7a8-e10d-4aca-bc70-950a2f0cd585-istio-podinfo\") pod \"e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-smtdr\" (UID: \"5086f7a8-e10d-4aca-bc70-950a2f0cd585\") " pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-smtdr"
Nov 02 15:44:56.637256 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:44:56.637229    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5086f7a8-e10d-4aca-bc70-950a2f0cd585-workload-certs\") pod \"e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-smtdr\" (UID: \"5086f7a8-e10d-4aca-bc70-950a2f0cd585\") " pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-smtdr"
Nov 02 15:44:56.637345 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:44:56.637332    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sv2vr\" (UniqueName: \"kubernetes.io/projected/93e559b7-2a4c-4533-90d5-5f3fd3956bf0-kube-api-access-sv2vr\") pod \"e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-lvrfc\" (UID: \"93e559b7-2a4c-4533-90d5-5f3fd3956bf0\") " pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-lvrfc"
Nov 02 15:44:56.637424 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:44:56.637360    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93e559b7-2a4c-4533-90d5-5f3fd3956bf0-workload-certs\") pod \"e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-lvrfc\" (UID: \"93e559b7-2a4c-4533-90d5-5f3fd3956bf0\") " pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-lvrfc"
Nov 02 15:44:56.637424 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:44:56.637388    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93e559b7-2a4c-4533-90d5-5f3fd3956bf0-istio-envoy\") pod \"e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-lvrfc\" (UID: \"93e559b7-2a4c-4533-90d5-5f3fd3956bf0\") " pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-lvrfc"
Nov 02 15:44:56.637546 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:44:56.637453    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5086f7a8-e10d-4aca-bc70-950a2f0cd585-credential-socket\") pod \"e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-smtdr\" (UID: \"5086f7a8-e10d-4aca-bc70-950a2f0cd585\") " pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-smtdr"
Nov 02 15:44:56.637546 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:44:56.637528    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93e559b7-2a4c-4533-90d5-5f3fd3956bf0-workload-socket\") pod \"e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-lvrfc\" (UID: \"93e559b7-2a4c-4533-90d5-5f3fd3956bf0\") " pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-lvrfc"
Nov 02 15:44:56.637691 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:44:56.637656    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5086f7a8-e10d-4aca-bc70-950a2f0cd585-workload-certs\") pod \"e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-smtdr\" (UID: \"5086f7a8-e10d-4aca-bc70-950a2f0cd585\") " pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-smtdr"
Nov 02 15:44:56.637759 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:44:56.637723    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5086f7a8-e10d-4aca-bc70-950a2f0cd585-credential-socket\") pod \"e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-smtdr\" (UID: \"5086f7a8-e10d-4aca-bc70-950a2f0cd585\") " pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-smtdr"
Nov 02 15:44:56.637857 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:44:56.637832    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5086f7a8-e10d-4aca-bc70-950a2f0cd585-istiod-ca-cert\") pod \"e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-smtdr\" (UID: \"5086f7a8-e10d-4aca-bc70-950a2f0cd585\") " pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-smtdr"
Nov 02 15:44:56.637996 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:44:56.637968    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93e559b7-2a4c-4533-90d5-5f3fd3956bf0-istiod-ca-cert\") pod \"e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-lvrfc\" (UID: \"93e559b7-2a4c-4533-90d5-5f3fd3956bf0\") " pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-lvrfc"
Nov 02 15:44:56.638162 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:44:56.638022    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93e559b7-2a4c-4533-90d5-5f3fd3956bf0-workload-socket\") pod \"e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-lvrfc\" (UID: \"93e559b7-2a4c-4533-90d5-5f3fd3956bf0\") " pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-lvrfc"
Nov 02 15:44:56.638162 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:44:56.638028    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5086f7a8-e10d-4aca-bc70-950a2f0cd585-workload-socket\") pod \"e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-smtdr\" (UID: \"5086f7a8-e10d-4aca-bc70-950a2f0cd585\") " pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-smtdr"
Nov 02 15:44:56.638162 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:44:56.638051    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93e559b7-2a4c-4533-90d5-5f3fd3956bf0-workload-certs\") pod \"e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-lvrfc\" (UID: \"93e559b7-2a4c-4533-90d5-5f3fd3956bf0\") " pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-lvrfc"
Nov 02 15:44:56.638366 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:44:56.638177    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/93e559b7-2a4c-4533-90d5-5f3fd3956bf0-istio-token\") pod \"e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-lvrfc\" (UID: \"93e559b7-2a4c-4533-90d5-5f3fd3956bf0\") " pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-lvrfc"
Nov 02 15:44:56.638366 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:44:56.638213    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93e559b7-2a4c-4533-90d5-5f3fd3956bf0-credential-socket\") pod \"e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-lvrfc\" (UID: \"93e559b7-2a4c-4533-90d5-5f3fd3956bf0\") " pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-lvrfc"
Nov 02 15:44:56.638366 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:44:56.638292    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5086f7a8-e10d-4aca-bc70-950a2f0cd585-workload-socket\") pod \"e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-smtdr\" (UID: \"5086f7a8-e10d-4aca-bc70-950a2f0cd585\") " pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-smtdr"
Nov 02 15:44:56.638366 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:44:56.638318    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5086f7a8-e10d-4aca-bc70-950a2f0cd585-istio-data\") pod \"e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-smtdr\" (UID: \"5086f7a8-e10d-4aca-bc70-950a2f0cd585\") " pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-smtdr"
Nov 02 15:44:56.638707 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:44:56.638683    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93e559b7-2a4c-4533-90d5-5f3fd3956bf0-credential-socket\") pod \"e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-lvrfc\" (UID: \"93e559b7-2a4c-4533-90d5-5f3fd3956bf0\") " pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-lvrfc"
Nov 02 15:44:56.638707 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:44:56.638695    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5086f7a8-e10d-4aca-bc70-950a2f0cd585-istio-data\") pod \"e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-smtdr\" (UID: \"5086f7a8-e10d-4aca-bc70-950a2f0cd585\") " pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-smtdr"
Nov 02 15:44:56.639104 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:44:56.639080    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93e559b7-2a4c-4533-90d5-5f3fd3956bf0-istiod-ca-cert\") pod \"e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-lvrfc\" (UID: \"93e559b7-2a4c-4533-90d5-5f3fd3956bf0\") " pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-lvrfc"
Nov 02 15:44:56.639465 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:44:56.639432    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93e559b7-2a4c-4533-90d5-5f3fd3956bf0-istio-podinfo\") pod \"e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-lvrfc\" (UID: \"93e559b7-2a4c-4533-90d5-5f3fd3956bf0\") " pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-lvrfc"
Nov 02 15:44:56.642327 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:44:56.642304    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93e559b7-2a4c-4533-90d5-5f3fd3956bf0-istio-envoy\") pod \"e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-lvrfc\" (UID: \"93e559b7-2a4c-4533-90d5-5f3fd3956bf0\") " pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-lvrfc"
Nov 02 15:44:56.642406 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:44:56.642308    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5086f7a8-e10d-4aca-bc70-950a2f0cd585-istio-envoy\") pod \"e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-smtdr\" (UID: \"5086f7a8-e10d-4aca-bc70-950a2f0cd585\") " pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-smtdr"
Nov 02 15:44:56.642468 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:44:56.642439    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5086f7a8-e10d-4aca-bc70-950a2f0cd585-istio-podinfo\") pod \"e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-smtdr\" (UID: \"5086f7a8-e10d-4aca-bc70-950a2f0cd585\") " pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-smtdr"
Nov 02 15:44:56.661140 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:44:56.661098    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5086f7a8-e10d-4aca-bc70-950a2f0cd585-istio-token\") pod \"e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-smtdr\" (UID: \"5086f7a8-e10d-4aca-bc70-950a2f0cd585\") " pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-smtdr"
Nov 02 15:44:56.661292 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:44:56.661274    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vrvm8\" (UniqueName: \"kubernetes.io/projected/5086f7a8-e10d-4aca-bc70-950a2f0cd585-kube-api-access-vrvm8\") pod \"e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-smtdr\" (UID: \"5086f7a8-e10d-4aca-bc70-950a2f0cd585\") " pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-smtdr"
Nov 02 15:44:56.661360 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:44:56.661342    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sv2vr\" (UniqueName: \"kubernetes.io/projected/93e559b7-2a4c-4533-90d5-5f3fd3956bf0-kube-api-access-sv2vr\") pod \"e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-lvrfc\" (UID: \"93e559b7-2a4c-4533-90d5-5f3fd3956bf0\") " pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-lvrfc"
Nov 02 15:44:56.662259 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:44:56.662240    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/93e559b7-2a4c-4533-90d5-5f3fd3956bf0-istio-token\") pod \"e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-lvrfc\" (UID: \"93e559b7-2a4c-4533-90d5-5f3fd3956bf0\") " pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-lvrfc"
Nov 02 15:44:56.682326 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:44:56.682299    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-lvrfc"
Nov 02 15:44:56.700624 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:44:56.700597    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-smtdr"
Nov 02 15:44:57.009658 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:44:57.009593    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-lvrfc"]
Nov 02 15:44:57.012192 ip-10-0-135-138 kubenswrapper[5304]: W1102 15:44:57.012161    5304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93e559b7_2a4c_4533_90d5_5f3fd3956bf0.slice/crio-624678ae56f61e393423e502c4620825666ba829d18033e1df0baaf2b64d3778 WatchSource:0}: Error finding container 624678ae56f61e393423e502c4620825666ba829d18033e1df0baaf2b64d3778: Status 404 returned error can't find the container with id 624678ae56f61e393423e502c4620825666ba829d18033e1df0baaf2b64d3778
Nov 02 15:44:57.014927 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:44:57.014894    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:44:57.014993 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:44:57.014959    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:44:57.014993 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:44:57.014988    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:44:57.050118 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:44:57.050055    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-smtdr"]
Nov 02 15:44:57.062508 ip-10-0-135-138 kubenswrapper[5304]: W1102 15:44:57.062476    5304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5086f7a8_e10d_4aca_bc70_950a2f0cd585.slice/crio-a3e7ea72059348bb25b850e69731bc2f278ecda4a7024812ec9749740a4cb7ce WatchSource:0}: Error finding container a3e7ea72059348bb25b850e69731bc2f278ecda4a7024812ec9749740a4cb7ce: Status 404 returned error can't find the container with id a3e7ea72059348bb25b850e69731bc2f278ecda4a7024812ec9749740a4cb7ce
Nov 02 15:44:57.065580 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:44:57.065544    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:44:57.065677 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:44:57.065620    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:44:57.065677 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:44:57.065650    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:44:57.146180 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:44:57.146144    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-smtdr" event={"ID":"5086f7a8-e10d-4aca-bc70-950a2f0cd585","Type":"ContainerStarted","Data":"d169223ffff4b48d9ac2bb350d36d9b24a10a9228d6f22abad89d8b26d735c56"}
Nov 02 15:44:57.146435 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:44:57.146414    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-smtdr" event={"ID":"5086f7a8-e10d-4aca-bc70-950a2f0cd585","Type":"ContainerStarted","Data":"a3e7ea72059348bb25b850e69731bc2f278ecda4a7024812ec9749740a4cb7ce"}
Nov 02 15:44:57.148248 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:44:57.148220    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-lvrfc" event={"ID":"93e559b7-2a4c-4533-90d5-5f3fd3956bf0","Type":"ContainerStarted","Data":"6796c86fab92cf70995a4feb5e28881cf73c8de1f4623d22c1dd07a5c4fa9bb4"}
Nov 02 15:44:57.148353 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:44:57.148273    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-lvrfc" event={"ID":"93e559b7-2a4c-4533-90d5-5f3fd3956bf0","Type":"ContainerStarted","Data":"624678ae56f61e393423e502c4620825666ba829d18033e1df0baaf2b64d3778"}
Nov 02 15:44:57.198865 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:44:57.197870    5304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-smtdr" podStartSLOduration=1.197850199 podStartE2EDuration="1.197850199s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:44:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:44:57.19499087 +0000 UTC m=+2897.188815280" watchObservedRunningTime="2026-11-02 15:44:57.197850199 +0000 UTC m=+2897.191674608"
Nov 02 15:44:57.250626 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:44:57.250558    5304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-lvrfc" podStartSLOduration=1.250537729 podStartE2EDuration="1.250537729s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:44:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:44:57.247207278 +0000 UTC m=+2897.241031691" watchObservedRunningTime="2026-11-02 15:44:57.250537729 +0000 UTC m=+2897.244362138"
Nov 02 15:44:57.682508 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:44:57.682478    5304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-lvrfc"
Nov 02 15:44:57.701609 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:44:57.701582    5304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-smtdr"
Nov 02 15:44:58.684611 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:44:58.684575    5304 patch_prober.go:29] interesting pod/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-lvrfc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused" start-of-body=
Nov 02 15:44:58.685170 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:44:58.684747    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-lvrfc" podUID="93e559b7-2a4c-4533-90d5-5f3fd3956bf0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused"
Nov 02 15:44:58.702585 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:44:58.702553    5304 patch_prober.go:29] interesting pod/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-smtdr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused" start-of-body=
Nov 02 15:44:58.702750 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:44:58.702603    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-smtdr" podUID="5086f7a8-e10d-4aca-bc70-950a2f0cd585" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused"
Nov 02 15:44:59.688015 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:44:59.687974    5304 patch_prober.go:29] interesting pod/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-lvrfc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused" start-of-body=
Nov 02 15:44:59.688538 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:44:59.688036    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-lvrfc" podUID="93e559b7-2a4c-4533-90d5-5f3fd3956bf0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused"
Nov 02 15:44:59.701289 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:44:59.701252    5304 patch_prober.go:29] interesting pod/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-smtdr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused" start-of-body=
Nov 02 15:44:59.701415 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:44:59.701319    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-smtdr" podUID="5086f7a8-e10d-4aca-bc70-950a2f0cd585" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused"
Nov 02 15:45:00.181547 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:00.181508    5304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-b55da1-29893905-mv5qt"]
Nov 02 15:45:00.190230 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:00.190196    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-b55da1-29893905-mv5qt"
Nov 02 15:45:00.207236 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:00.207199    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-b55da1-29893905-mv5qt"]
Nov 02 15:45:00.226091 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:00.226045    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/647ad5be-64be-4aff-bff6-d0ac2943671c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-b55da1-29893905-mv5qt\" (UID: \"647ad5be-64be-4aff-bff6-d0ac2943671c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-b55da1-29893905-mv5qt"
Nov 02 15:45:00.226545 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:00.226519    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qh7nb\" (UniqueName: \"kubernetes.io/projected/647ad5be-64be-4aff-bff6-d0ac2943671c-kube-api-access-qh7nb\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-b55da1-29893905-mv5qt\" (UID: \"647ad5be-64be-4aff-bff6-d0ac2943671c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-b55da1-29893905-mv5qt"
Nov 02 15:45:00.317155 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:00.304624    5304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-m5926\""
Nov 02 15:45:00.355587 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:00.355544    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/647ad5be-64be-4aff-bff6-d0ac2943671c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-b55da1-29893905-mv5qt\" (UID: \"647ad5be-64be-4aff-bff6-d0ac2943671c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-b55da1-29893905-mv5qt"
Nov 02 15:45:00.364291 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:00.364250    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qh7nb\" (UniqueName: \"kubernetes.io/projected/647ad5be-64be-4aff-bff6-d0ac2943671c-kube-api-access-qh7nb\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-b55da1-29893905-mv5qt\" (UID: \"647ad5be-64be-4aff-bff6-d0ac2943671c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-b55da1-29893905-mv5qt"
Nov 02 15:45:00.370150 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:00.365995    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/647ad5be-64be-4aff-bff6-d0ac2943671c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-b55da1-29893905-mv5qt\" (UID: \"647ad5be-64be-4aff-bff6-d0ac2943671c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-b55da1-29893905-mv5qt"
Nov 02 15:45:00.432212 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:00.426667    5304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-78d075-29893905-sznt8"]
Nov 02 15:45:00.502152 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:00.495850    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qh7nb\" (UniqueName: \"kubernetes.io/projected/647ad5be-64be-4aff-bff6-d0ac2943671c-kube-api-access-qh7nb\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-b55da1-29893905-mv5qt\" (UID: \"647ad5be-64be-4aff-bff6-d0ac2943671c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-b55da1-29893905-mv5qt"
Nov 02 15:45:00.563460 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:00.563428    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-78d075-29893905-sznt8"
Nov 02 15:45:00.701666 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:00.701584    5304 patch_prober.go:29] interesting pod/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-smtdr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused" start-of-body=
Nov 02 15:45:00.702246 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:00.702214    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-smtdr" podUID="5086f7a8-e10d-4aca-bc70-950a2f0cd585" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused"
Nov 02 15:45:00.735152 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:00.709611    5304 patch_prober.go:29] interesting pod/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-lvrfc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused" start-of-body=
Nov 02 15:45:00.735492 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:00.735444    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-lvrfc" podUID="93e559b7-2a4c-4533-90d5-5f3fd3956bf0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused"
Nov 02 15:45:00.756149 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:00.737691    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4cce806c-5164-485a-a790-519176d79e26-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-78d075-29893905-sznt8\" (UID: \"4cce806c-5164-485a-a790-519176d79e26\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-78d075-29893905-sznt8"
Nov 02 15:45:00.829368 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:00.829323    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kk24l\" (UniqueName: \"kubernetes.io/projected/4cce806c-5164-485a-a790-519176d79e26-kube-api-access-kk24l\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-78d075-29893905-sznt8\" (UID: \"4cce806c-5164-485a-a790-519176d79e26\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-78d075-29893905-sznt8"
Nov 02 15:45:00.878302 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:00.751663    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-b55da1-29893905-mv5qt"
Nov 02 15:45:00.893235 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:00.893191    5304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893905-4d75m"]
Nov 02 15:45:00.974210 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:00.972394    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kk24l\" (UniqueName: \"kubernetes.io/projected/4cce806c-5164-485a-a790-519176d79e26-kube-api-access-kk24l\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-78d075-29893905-sznt8\" (UID: \"4cce806c-5164-485a-a790-519176d79e26\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-78d075-29893905-sznt8"
Nov 02 15:45:00.974210 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:00.972690    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4cce806c-5164-485a-a790-519176d79e26-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-78d075-29893905-sznt8\" (UID: \"4cce806c-5164-485a-a790-519176d79e26\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-78d075-29893905-sznt8"
Nov 02 15:45:00.974210 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:00.973615    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4cce806c-5164-485a-a790-519176d79e26-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-78d075-29893905-sznt8\" (UID: \"4cce806c-5164-485a-a790-519176d79e26\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-78d075-29893905-sznt8"
Nov 02 15:45:01.099154 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:01.088911    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kk24l\" (UniqueName: \"kubernetes.io/projected/4cce806c-5164-485a-a790-519176d79e26-kube-api-access-kk24l\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-78d075-29893905-sznt8\" (UID: \"4cce806c-5164-485a-a790-519176d79e26\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-78d075-29893905-sznt8"
Nov 02 15:45:01.099154 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:01.095739    5304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-0a7163-29893905-rkqjw"]
Nov 02 15:45:01.112300 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:01.112258    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893905-4d75m"
Nov 02 15:45:01.131534 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:01.131499    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-78d075-29893905-sznt8"
Nov 02 15:45:01.224268 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:01.209939    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9846a62a-c4aa-4937-b1dd-2d9a342dd72a-maas-api-service-ca\") pod \"maas-api-key-cleanup-29893905-4d75m\" (UID: \"9846a62a-c4aa-4937-b1dd-2d9a342dd72a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893905-4d75m"
Nov 02 15:45:01.224268 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:01.209999    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9dsnf\" (UniqueName: \"kubernetes.io/projected/9846a62a-c4aa-4937-b1dd-2d9a342dd72a-kube-api-access-9dsnf\") pod \"maas-api-key-cleanup-29893905-4d75m\" (UID: \"9846a62a-c4aa-4937-b1dd-2d9a342dd72a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893905-4d75m"
Nov 02 15:45:01.244183 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:01.234343    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-78d075-29893905-sznt8"]
Nov 02 15:45:01.244183 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:01.234477    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-0a7163-29893905-rkqjw"
Nov 02 15:45:01.263218 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:01.263183    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893905-4d75m"]
Nov 02 15:45:01.316047 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:01.316008    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9846a62a-c4aa-4937-b1dd-2d9a342dd72a-maas-api-service-ca\") pod \"maas-api-key-cleanup-29893905-4d75m\" (UID: \"9846a62a-c4aa-4937-b1dd-2d9a342dd72a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893905-4d75m"
Nov 02 15:45:01.340181 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:01.340138    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9dsnf\" (UniqueName: \"kubernetes.io/projected/9846a62a-c4aa-4937-b1dd-2d9a342dd72a-kube-api-access-9dsnf\") pod \"maas-api-key-cleanup-29893905-4d75m\" (UID: \"9846a62a-c4aa-4937-b1dd-2d9a342dd72a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893905-4d75m"
Nov 02 15:45:01.340698 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:01.322026    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9846a62a-c4aa-4937-b1dd-2d9a342dd72a-maas-api-service-ca\") pod \"maas-api-key-cleanup-29893905-4d75m\" (UID: \"9846a62a-c4aa-4937-b1dd-2d9a342dd72a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893905-4d75m"
Nov 02 15:45:01.393374 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:01.393290    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-624254/distinct-w3-624254-kserve-5465dc4765-dmq67" podUID="1a5aa910-a6de-417b-8638-81e4ec5c1825" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:45:01.440352 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:01.397841    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w2-797488/sub-test-model-797488-kserve-79764959f-k4scd" podUID="2d6e474d-c5d3-4c69-b12f-993ff23910c1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.165:8000/ready\": context deadline exceeded"
Nov 02 15:45:01.440352 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:01.398200    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6af50f/distinct-w0-6af50f-kserve-c6d994647-hbj9z" podUID="3dca5bf3-8e08-47a0-936e-205670d59ede" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded"
Nov 02 15:45:01.440352 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:01.423241    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1cc1b2/distinct-w5-1cc1b2-kserve-5b7459bc6d-zl8xr" podUID="211ad15d-9234-4f57-b1b9-181dd2d74208" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:45:01.440352 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:01.438614    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-0a7163-29893905-rkqjw"]
Nov 02 15:45:01.458657 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:01.458627    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0819318-3392-48ed-a0e4-4d7961144393-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-0a7163-29893905-rkqjw\" (UID: \"e0819318-3392-48ed-a0e4-4d7961144393\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-0a7163-29893905-rkqjw"
Nov 02 15:45:01.459329 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:01.459306    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-44zq4\" (UniqueName: \"kubernetes.io/projected/e0819318-3392-48ed-a0e4-4d7961144393-kube-api-access-44zq4\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-0a7163-29893905-rkqjw\" (UID: \"e0819318-3392-48ed-a0e4-4d7961144393\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-0a7163-29893905-rkqjw"
Nov 02 15:45:01.476451 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:01.476377    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9dsnf\" (UniqueName: \"kubernetes.io/projected/9846a62a-c4aa-4937-b1dd-2d9a342dd72a-kube-api-access-9dsnf\") pod \"maas-api-key-cleanup-29893905-4d75m\" (UID: \"9846a62a-c4aa-4937-b1dd-2d9a342dd72a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893905-4d75m"
Nov 02 15:45:01.486358 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:01.486326    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893905-4d75m"
Nov 02 15:45:01.540978 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:01.519850    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5dd8e9/simulator-w1-5dd8e9-kserve-6f9f755d6f-gnw5d" podUID="66b75e64-cb24-40be-9076-4984c132e208" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:45:01.540978 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:01.523499    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-739bf1/distinct-2-w2-739bf1-kserve-5c58479c48-qgfkh" podUID="9746bf8a-91bd-4715-a02e-ae2e651ca54d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded"
Nov 02 15:45:01.577228 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:01.577187    5304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-cce791-29893905-qlp7q"]
Nov 02 15:45:01.596322 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:01.578600    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-44zq4\" (UniqueName: \"kubernetes.io/projected/e0819318-3392-48ed-a0e4-4d7961144393-kube-api-access-44zq4\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-0a7163-29893905-rkqjw\" (UID: \"e0819318-3392-48ed-a0e4-4d7961144393\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-0a7163-29893905-rkqjw"
Nov 02 15:45:01.811262 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:01.811220    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0819318-3392-48ed-a0e4-4d7961144393-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-0a7163-29893905-rkqjw\" (UID: \"e0819318-3392-48ed-a0e4-4d7961144393\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-0a7163-29893905-rkqjw"
Nov 02 15:45:01.812844 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:01.706653    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7ecfe1/distinct-w4-7ecfe1-kserve-6c847c647d-d2jvv" podUID="f1f1ca5e-cca7-4c54-939f-06f095defcdd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded"
Nov 02 15:45:01.834089 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:01.706685    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1cc1b2/premium-w5-1cc1b2-kserve-58db46f97c-cgdqg" podUID="210f620a-8ae7-4e4c-acc3-46f75ddc6105" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded"
Nov 02 15:45:01.834089 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:01.706709    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-624254/simulator-w3-624254-kserve-56866b7699-z47sd" podUID="345c710f-b414-4e62-86c7-d51ff4e5be15" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:45:01.834089 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:01.709230    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-739bf1/simulator-w2-739bf1-kserve-7c74f85465-ltk4t" podUID="cf6a1ded-28a3-49eb-878b-b4ad77a0bb5c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:45:01.834089 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:01.720309    5304 patch_prober.go:29] interesting pod/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-lvrfc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused" start-of-body=
Nov 02 15:45:01.834089 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:01.824661    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-lvrfc" podUID="93e559b7-2a4c-4533-90d5-5f3fd3956bf0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused"
Nov 02 15:45:01.834089 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:01.738363    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-cce791-29893905-qlp7q"
Nov 02 15:45:01.834089 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:01.738255    5304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-246137-29893905-gxb7t"]
Nov 02 15:45:01.841214 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:01.812605    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0819318-3392-48ed-a0e4-4d7961144393-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-0a7163-29893905-rkqjw\" (UID: \"e0819318-3392-48ed-a0e4-4d7961144393\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-0a7163-29893905-rkqjw"
Nov 02 15:45:01.841214 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:01.721106    5304 patch_prober.go:29] interesting pod/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-smtdr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused" start-of-body=
Nov 02 15:45:01.841214 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:01.837258    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-smtdr" podUID="5086f7a8-e10d-4aca-bc70-950a2f0cd585" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused"
Nov 02 15:45:01.849151 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:01.584361    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nkbrk" podUID="1f68d837-fff2-418b-84d7-c2d7eb21a57e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.24:8000/ready\": context deadline exceeded"
Nov 02 15:45:01.884677 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:01.853237    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-44zq4\" (UniqueName: \"kubernetes.io/projected/e0819318-3392-48ed-a0e4-4d7961144393-kube-api-access-44zq4\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-0a7163-29893905-rkqjw\" (UID: \"e0819318-3392-48ed-a0e4-4d7961144393\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-0a7163-29893905-rkqjw"
Nov 02 15:45:01.964878 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:01.964844    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a07ca06a-c083-4fe9-bca4-1614646e80fb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-cce791-29893905-qlp7q\" (UID: \"a07ca06a-c083-4fe9-bca4-1614646e80fb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-cce791-29893905-qlp7q"
Nov 02 15:45:01.965613 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:01.965583    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dhz7b\" (UniqueName: \"kubernetes.io/projected/a07ca06a-c083-4fe9-bca4-1614646e80fb-kube-api-access-dhz7b\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-cce791-29893905-qlp7q\" (UID: \"a07ca06a-c083-4fe9-bca4-1614646e80fb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-cce791-29893905-qlp7q"
Nov 02 15:45:02.000374 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:02.000334    5304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-6af50f-29893905-85m66"]
Nov 02 15:45:02.025445 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:02.025324    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-246137-29893905-gxb7t"
Nov 02 15:45:02.043852 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:02.043814    5304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-1cc1b2-29893905-whfr9"]
Nov 02 15:45:02.046082 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:02.044239    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-6af50f-29893905-85m66"
Nov 02 15:45:02.151160 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:02.147680    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4tcl2\" (UniqueName: \"kubernetes.io/projected/1d05aa2a-2dfb-47be-9573-a8f32cf05ab7-kube-api-access-4tcl2\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-246137-29893905-gxb7t\" (UID: \"1d05aa2a-2dfb-47be-9573-a8f32cf05ab7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-246137-29893905-gxb7t"
Nov 02 15:45:02.151160 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:02.147754    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d05aa2a-2dfb-47be-9573-a8f32cf05ab7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-246137-29893905-gxb7t\" (UID: \"1d05aa2a-2dfb-47be-9573-a8f32cf05ab7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-246137-29893905-gxb7t"
Nov 02 15:45:02.151160 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:02.147826    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dhz7b\" (UniqueName: \"kubernetes.io/projected/a07ca06a-c083-4fe9-bca4-1614646e80fb-kube-api-access-dhz7b\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-cce791-29893905-qlp7q\" (UID: \"a07ca06a-c083-4fe9-bca4-1614646e80fb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-cce791-29893905-qlp7q"
Nov 02 15:45:02.151160 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:02.147924    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a07ca06a-c083-4fe9-bca4-1614646e80fb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-cce791-29893905-qlp7q\" (UID: \"a07ca06a-c083-4fe9-bca4-1614646e80fb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-cce791-29893905-qlp7q"
Nov 02 15:45:02.151160 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:02.148081    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ea83c17-d3cb-4d7e-a6ef-cbd731415216-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-6af50f-29893905-85m66\" (UID: \"8ea83c17-d3cb-4d7e-a6ef-cbd731415216\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-6af50f-29893905-85m66"
Nov 02 15:45:02.158863 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:02.158595    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-0a7163-29893905-rkqjw"
Nov 02 15:45:02.187760 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:02.187428    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7zqnq\" (UniqueName: \"kubernetes.io/projected/8ea83c17-d3cb-4d7e-a6ef-cbd731415216-kube-api-access-7zqnq\") pod \"maas-api-key-cleanup-e2e-worker-w0-6af50f-29893905-85m66\" (UID: \"8ea83c17-d3cb-4d7e-a6ef-cbd731415216\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-6af50f-29893905-85m66"
Nov 02 15:45:02.188670 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:02.188608    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a07ca06a-c083-4fe9-bca4-1614646e80fb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-cce791-29893905-qlp7q\" (UID: \"a07ca06a-c083-4fe9-bca4-1614646e80fb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-cce791-29893905-qlp7q"
Nov 02 15:45:02.264148 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:02.203486    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-1cc1b2-29893905-whfr9"
Nov 02 15:45:02.264148 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:02.258936    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dhz7b\" (UniqueName: \"kubernetes.io/projected/a07ca06a-c083-4fe9-bca4-1614646e80fb-kube-api-access-dhz7b\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-cce791-29893905-qlp7q\" (UID: \"a07ca06a-c083-4fe9-bca4-1614646e80fb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-cce791-29893905-qlp7q"
Nov 02 15:45:02.337434 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:02.335798    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-cce791-29893905-qlp7q"
Nov 02 15:45:02.337434 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:02.335881    5304 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-624254/simulator-w3-624254-kserve-56866b7699-z47sd" podUID="345c710f-b414-4e62-86c7-d51ff4e5be15" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/health\": context deadline exceeded"
Nov 02 15:45:02.337690 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:02.337475    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4tcl2\" (UniqueName: \"kubernetes.io/projected/1d05aa2a-2dfb-47be-9573-a8f32cf05ab7-kube-api-access-4tcl2\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-246137-29893905-gxb7t\" (UID: \"1d05aa2a-2dfb-47be-9573-a8f32cf05ab7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-246137-29893905-gxb7t"
Nov 02 15:45:02.337690 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:02.337557    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d05aa2a-2dfb-47be-9573-a8f32cf05ab7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-246137-29893905-gxb7t\" (UID: \"1d05aa2a-2dfb-47be-9573-a8f32cf05ab7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-246137-29893905-gxb7t"
Nov 02 15:45:02.337690 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:02.337678    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea2c4ec8-2ace-49dd-a172-99ced21dbc4f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-1cc1b2-29893905-whfr9\" (UID: \"ea2c4ec8-2ace-49dd-a172-99ced21dbc4f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-1cc1b2-29893905-whfr9"
Nov 02 15:45:02.337868 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:02.337714    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mq9sq\" (UniqueName: \"kubernetes.io/projected/ea2c4ec8-2ace-49dd-a172-99ced21dbc4f-kube-api-access-mq9sq\") pod \"maas-api-key-cleanup-e2e-worker-w5-1cc1b2-29893905-whfr9\" (UID: \"ea2c4ec8-2ace-49dd-a172-99ced21dbc4f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-1cc1b2-29893905-whfr9"
Nov 02 15:45:02.337868 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:02.337853    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ea83c17-d3cb-4d7e-a6ef-cbd731415216-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-6af50f-29893905-85m66\" (UID: \"8ea83c17-d3cb-4d7e-a6ef-cbd731415216\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-6af50f-29893905-85m66"
Nov 02 15:45:02.337988 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:02.337968    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7zqnq\" (UniqueName: \"kubernetes.io/projected/8ea83c17-d3cb-4d7e-a6ef-cbd731415216-kube-api-access-7zqnq\") pod \"maas-api-key-cleanup-e2e-worker-w0-6af50f-29893905-85m66\" (UID: \"8ea83c17-d3cb-4d7e-a6ef-cbd731415216\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-6af50f-29893905-85m66"
Nov 02 15:45:02.347172 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:02.339254    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d05aa2a-2dfb-47be-9573-a8f32cf05ab7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-246137-29893905-gxb7t\" (UID: \"1d05aa2a-2dfb-47be-9573-a8f32cf05ab7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-246137-29893905-gxb7t"
Nov 02 15:45:02.420305 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:02.420054    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-624254/distinct-w3-624254-kserve-5465dc4765-dmq67" podUID="1a5aa910-a6de-417b-8638-81e4ec5c1825" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded"
Nov 02 15:45:02.426159 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:02.421326    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ea83c17-d3cb-4d7e-a6ef-cbd731415216-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-6af50f-29893905-85m66\" (UID: \"8ea83c17-d3cb-4d7e-a6ef-cbd731415216\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-6af50f-29893905-85m66"
Nov 02 15:45:02.426159 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:02.421410    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5dd8e9/premium-w1-5dd8e9-kserve-67588456fb-j6lph" podUID="4ccc607a-14e4-43ca-8999-78100d4b799f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:45:02.426159 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:02.425346    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1cc1b2/distinct-w5-1cc1b2-kserve-5b7459bc6d-zl8xr" podUID="211ad15d-9234-4f57-b1b9-181dd2d74208" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:45:02.430192 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:02.426591    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w2-797488/sub-test-model-797488-kserve-79764959f-k4scd" podUID="2d6e474d-c5d3-4c69-b12f-993ff23910c1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.165:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:45:02.430192 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:02.426876    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6af50f/distinct-w0-6af50f-kserve-c6d994647-hbj9z" podUID="3dca5bf3-8e08-47a0-936e-205670d59ede" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded"
Nov 02 15:45:02.512665 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:02.490979    5304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-28da58-29893905-phwql"]
Nov 02 15:45:02.528160 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:02.519899    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea2c4ec8-2ace-49dd-a172-99ced21dbc4f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-1cc1b2-29893905-whfr9\" (UID: \"ea2c4ec8-2ace-49dd-a172-99ced21dbc4f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-1cc1b2-29893905-whfr9"
Nov 02 15:45:02.629422 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:02.520101    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mq9sq\" (UniqueName: \"kubernetes.io/projected/ea2c4ec8-2ace-49dd-a172-99ced21dbc4f-kube-api-access-mq9sq\") pod \"maas-api-key-cleanup-e2e-worker-w5-1cc1b2-29893905-whfr9\" (UID: \"ea2c4ec8-2ace-49dd-a172-99ced21dbc4f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-1cc1b2-29893905-whfr9"
Nov 02 15:45:02.629422 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:02.520264    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5dd8e9/simulator-w1-5dd8e9-kserve-6f9f755d6f-gnw5d" podUID="66b75e64-cb24-40be-9076-4984c132e208" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": context deadline exceeded"
Nov 02 15:45:02.629422 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:02.529239    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-739bf1/distinct-2-w2-739bf1-kserve-5c58479c48-qgfkh" podUID="9746bf8a-91bd-4715-a02e-ae2e651ca54d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:45:02.629422 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:02.550961    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea2c4ec8-2ace-49dd-a172-99ced21dbc4f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-1cc1b2-29893905-whfr9\" (UID: \"ea2c4ec8-2ace-49dd-a172-99ced21dbc4f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-1cc1b2-29893905-whfr9"
Nov 02 15:45:02.629422 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:02.556819    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7zqnq\" (UniqueName: \"kubernetes.io/projected/8ea83c17-d3cb-4d7e-a6ef-cbd731415216-kube-api-access-7zqnq\") pod \"maas-api-key-cleanup-e2e-worker-w0-6af50f-29893905-85m66\" (UID: \"8ea83c17-d3cb-4d7e-a6ef-cbd731415216\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-6af50f-29893905-85m66"
Nov 02 15:45:02.629422 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:02.622180    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mq9sq\" (UniqueName: \"kubernetes.io/projected/ea2c4ec8-2ace-49dd-a172-99ced21dbc4f-kube-api-access-mq9sq\") pod \"maas-api-key-cleanup-e2e-worker-w5-1cc1b2-29893905-whfr9\" (UID: \"ea2c4ec8-2ace-49dd-a172-99ced21dbc4f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-1cc1b2-29893905-whfr9"
Nov 02 15:45:02.668062 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:02.667922    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4tcl2\" (UniqueName: \"kubernetes.io/projected/1d05aa2a-2dfb-47be-9573-a8f32cf05ab7-kube-api-access-4tcl2\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-246137-29893905-gxb7t\" (UID: \"1d05aa2a-2dfb-47be-9573-a8f32cf05ab7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-246137-29893905-gxb7t"
Nov 02 15:45:02.815155 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:02.812789    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-624254/distinct-w3-624254-kserve-5465dc4765-dmq67"
Nov 02 15:45:02.815155 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:02.812940    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-1cc1b2/distinct-w5-1cc1b2-kserve-5b7459bc6d-zl8xr"
Nov 02 15:45:02.815155 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:02.812998    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w2-797488/sub-test-model-797488-kserve-79764959f-k4scd"
Nov 02 15:45:02.815155 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:02.813052    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-6af50f/distinct-w0-6af50f-kserve-c6d994647-hbj9z"
Nov 02 15:45:02.815155 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:02.813102    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5dd8e9/simulator-w1-5dd8e9-kserve-6f9f755d6f-gnw5d"
Nov 02 15:45:02.815155 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:02.813174    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-739bf1/distinct-2-w2-739bf1-kserve-5c58479c48-qgfkh"
Nov 02 15:45:02.815155 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:02.813596    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-246137-29893905-gxb7t"
Nov 02 15:45:02.823170 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:02.819711    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-28da58-29893905-phwql"
Nov 02 15:45:02.853983 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:02.826825    5304 patch_prober.go:29] interesting pod/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-lvrfc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused" start-of-body=
Nov 02 15:45:02.853983 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:02.826885    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-lvrfc" podUID="93e559b7-2a4c-4533-90d5-5f3fd3956bf0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused"
Nov 02 15:45:02.923521 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:02.915863    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-cce791-29893905-qlp7q"]
Nov 02 15:45:02.923521 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:02.917000    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nkbrk" podUID="1f68d837-fff2-418b-84d7-c2d7eb21a57e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.24:8000/ready\": context deadline exceeded"
Nov 02 15:45:02.923521 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:02.917863    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nkbrk"
Nov 02 15:45:02.923521 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:02.919448    5304 patch_prober.go:29] interesting pod/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-smtdr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused" start-of-body=
Nov 02 15:45:02.923521 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:02.919501    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-smtdr" podUID="5086f7a8-e10d-4aca-bc70-950a2f0cd585" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused"
Nov 02 15:45:02.923521 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:02.919609    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-739bf1/distinct-2-w2-739bf1-kserve-5c58479c48-qgfkh"
Nov 02 15:45:02.923521 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:02.919670    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5dd8e9/simulator-w1-5dd8e9-kserve-6f9f755d6f-gnw5d"
Nov 02 15:45:02.923521 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:02.922951    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-6af50f-29893905-85m66"
Nov 02 15:45:02.966514 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:02.958903    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-624254/simulator-w3-624254-kserve-56866b7699-z47sd" podUID="345c710f-b414-4e62-86c7-d51ff4e5be15" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:45:02.973153 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:02.972240    5304 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-7ecfe1/distinct-w4-7ecfe1-kserve-6c847c647d-d2jvv" podUID="f1f1ca5e-cca7-4c54-939f-06f095defcdd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:45:02.993152 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:02.958814    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-739bf1/simulator-w2-739bf1-kserve-7c74f85465-ltk4t" podUID="cf6a1ded-28a3-49eb-878b-b4ad77a0bb5c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded"
Nov 02 15:45:02.997477 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:02.997442    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wvx9t\" (UniqueName: \"kubernetes.io/projected/ada2b1b0-e598-4d38-b995-becab65c7e82-kube-api-access-wvx9t\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-28da58-29893905-phwql\" (UID: \"ada2b1b0-e598-4d38-b995-becab65c7e82\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-28da58-29893905-phwql"
Nov 02 15:45:02.997785 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:02.997762    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ada2b1b0-e598-4d38-b995-becab65c7e82-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-28da58-29893905-phwql\" (UID: \"ada2b1b0-e598-4d38-b995-becab65c7e82\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-28da58-29893905-phwql"
Nov 02 15:45:03.080151 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:03.068396    5304 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nkbrk" podUID="1f68d837-fff2-418b-84d7-c2d7eb21a57e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.24:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:45:03.123770 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:03.123694    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-1cc1b2-29893905-whfr9"
Nov 02 15:45:03.295615 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:03.295582    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-739bf1/simulator-w2-739bf1-kserve-7c74f85465-ltk4t"
Nov 02 15:45:03.305443 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:03.305399    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wvx9t\" (UniqueName: \"kubernetes.io/projected/ada2b1b0-e598-4d38-b995-becab65c7e82-kube-api-access-wvx9t\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-28da58-29893905-phwql\" (UID: \"ada2b1b0-e598-4d38-b995-becab65c7e82\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-28da58-29893905-phwql"
Nov 02 15:45:03.308852 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:03.308823    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ada2b1b0-e598-4d38-b995-becab65c7e82-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-28da58-29893905-phwql\" (UID: \"ada2b1b0-e598-4d38-b995-becab65c7e82\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-28da58-29893905-phwql"
Nov 02 15:45:03.310292 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:03.310267    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ada2b1b0-e598-4d38-b995-becab65c7e82-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-28da58-29893905-phwql\" (UID: \"ada2b1b0-e598-4d38-b995-becab65c7e82\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-28da58-29893905-phwql"
Nov 02 15:45:03.315043 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:03.314877    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-624254/simulator-w3-624254-kserve-56866b7699-z47sd"
Nov 02 15:45:03.334174 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:03.323811    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-6af50f/distinct-w0-6af50f-kserve-c6d994647-hbj9z"
Nov 02 15:45:03.334174 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:03.328842    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-624254/distinct-w3-624254-kserve-5465dc4765-dmq67"
Nov 02 15:45:03.363307 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:03.362200    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5113c2/premium-w6-5113c2-kserve-b84945cb5-pc6f8" podUID="9fb3e992-7bb0-497a-a82c-6e396ba241c1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:45:03.411178 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:03.410176    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-1cc1b2/distinct-w5-1cc1b2-kserve-5b7459bc6d-zl8xr"
Nov 02 15:45:03.411178 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:03.410238    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w2-797488/sub-test-model-797488-kserve-79764959f-k4scd"
Nov 02 15:45:03.421925 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:03.421876    5304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5113c2-29893905-bnvks"]
Nov 02 15:45:03.470280 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:03.448424    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5113c2-29893905-bnvks"
Nov 02 15:45:03.498506 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:03.498474    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-6af50f-29893905-85m66"]
Nov 02 15:45:03.544159 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:03.505897    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-28da58-29893905-phwql"]
Nov 02 15:45:03.544159 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:03.511779    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-1cc1b2-29893905-whfr9"]
Nov 02 15:45:03.544159 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:03.518007    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5113c2-29893905-bnvks"]
Nov 02 15:45:03.544159 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:03.518939    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nkbrk"
Nov 02 15:45:03.576702 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:03.564383    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wvx9t\" (UniqueName: \"kubernetes.io/projected/ada2b1b0-e598-4d38-b995-becab65c7e82-kube-api-access-wvx9t\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-28da58-29893905-phwql\" (UID: \"ada2b1b0-e598-4d38-b995-becab65c7e82\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-28da58-29893905-phwql"
Nov 02 15:45:03.591362 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:03.591327    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-246137-29893905-gxb7t"]
Nov 02 15:45:03.592043 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:03.592018    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-739bf1/simulator-w2-739bf1-kserve-7c74f85465-ltk4t"
Nov 02 15:45:03.621336 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:03.604746    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-624254/simulator-w3-624254-kserve-56866b7699-z47sd"
Nov 02 15:45:03.628913 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:03.628877    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5efd887c-66f3-445e-9e39-0d23e43cc9be-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-5113c2-29893905-bnvks\" (UID: \"5efd887c-66f3-445e-9e39-0d23e43cc9be\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5113c2-29893905-bnvks"
Nov 02 15:45:03.629468 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:03.629428    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hf7db\" (UniqueName: \"kubernetes.io/projected/5efd887c-66f3-445e-9e39-0d23e43cc9be-kube-api-access-hf7db\") pod \"maas-api-key-cleanup-e2e-worker-w6-5113c2-29893905-bnvks\" (UID: \"5efd887c-66f3-445e-9e39-0d23e43cc9be\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5113c2-29893905-bnvks"
Nov 02 15:45:03.703520 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:03.703482    5304 patch_prober.go:29] interesting pod/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-smtdr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused" start-of-body=
Nov 02 15:45:03.703651 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:03.703542    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-smtdr" podUID="5086f7a8-e10d-4aca-bc70-950a2f0cd585" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused"
Nov 02 15:45:03.717074 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:03.707827    5304 patch_prober.go:29] interesting pod/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-lvrfc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused" start-of-body=
Nov 02 15:45:03.717074 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:03.707878    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-lvrfc" podUID="93e559b7-2a4c-4533-90d5-5f3fd3956bf0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused"
Nov 02 15:45:03.717074 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:03.716407    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-b55da1-29893905-mv5qt"]
Nov 02 15:45:03.805447 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:03.805301    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-28da58-29893905-phwql"
Nov 02 15:45:03.806298 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:03.806269    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-78d075-29893905-sznt8"]
Nov 02 15:45:03.808517 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:03.808495    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893905-4d75m"]
Nov 02 15:45:03.870430 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:03.869098    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5efd887c-66f3-445e-9e39-0d23e43cc9be-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-5113c2-29893905-bnvks\" (UID: \"5efd887c-66f3-445e-9e39-0d23e43cc9be\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5113c2-29893905-bnvks"
Nov 02 15:45:03.899406 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:03.884829    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5efd887c-66f3-445e-9e39-0d23e43cc9be-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-5113c2-29893905-bnvks\" (UID: \"5efd887c-66f3-445e-9e39-0d23e43cc9be\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5113c2-29893905-bnvks"
Nov 02 15:45:03.899923 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:03.899885    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hf7db\" (UniqueName: \"kubernetes.io/projected/5efd887c-66f3-445e-9e39-0d23e43cc9be-kube-api-access-hf7db\") pod \"maas-api-key-cleanup-e2e-worker-w6-5113c2-29893905-bnvks\" (UID: \"5efd887c-66f3-445e-9e39-0d23e43cc9be\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5113c2-29893905-bnvks"
Nov 02 15:45:03.934267 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:03.930217    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-0a7163-29893905-rkqjw"]
Nov 02 15:45:03.939309 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:03.939248    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hf7db\" (UniqueName: \"kubernetes.io/projected/5efd887c-66f3-445e-9e39-0d23e43cc9be-kube-api-access-hf7db\") pod \"maas-api-key-cleanup-e2e-worker-w6-5113c2-29893905-bnvks\" (UID: \"5efd887c-66f3-445e-9e39-0d23e43cc9be\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5113c2-29893905-bnvks"
Nov 02 15:45:04.224451 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:04.224367    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5113c2-29893905-bnvks"
Nov 02 15:45:04.469003 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:04.433320    5304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-qcms9"]
Nov 02 15:45:04.591358 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:04.561390    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-cce791-29893905-qlp7q"]
Nov 02 15:45:04.591358 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:04.561453    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-qcms9"]
Nov 02 15:45:04.591358 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:04.561572    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-qcms9"
Nov 02 15:45:04.625197 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:04.607849    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-6af50f-29893905-85m66"]
Nov 02 15:45:04.626275 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:04.626251    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-246137-29893905-gxb7t"]
Nov 02 15:45:04.694872 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:04.694829    5304 patch_prober.go:29] interesting pod/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-lvrfc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused" start-of-body=
Nov 02 15:45:04.695079 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:04.695056    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-lvrfc" podUID="93e559b7-2a4c-4533-90d5-5f3fd3956bf0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused"
Nov 02 15:45:04.704351 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:04.698484    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-1cc1b2-29893905-whfr9"]
Nov 02 15:45:04.707234 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:04.705004    5304 patch_prober.go:29] interesting pod/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-smtdr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused" start-of-body=
Nov 02 15:45:04.707234 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:04.705051    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-smtdr" podUID="5086f7a8-e10d-4aca-bc70-950a2f0cd585" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused"
Nov 02 15:45:04.814359 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:04.733116    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jn78x\" (UniqueName: \"kubernetes.io/projected/f75de3cc-f69f-4dfc-9351-9e75a08723b3-kube-api-access-jn78x\") pod \"e2e-ipp-b-3805b8-openshift-default-6c9f89869-qcms9\" (UID: \"f75de3cc-f69f-4dfc-9351-9e75a08723b3\") " pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-qcms9"
Nov 02 15:45:04.814538 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:04.814477    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f75de3cc-f69f-4dfc-9351-9e75a08723b3-workload-certs\") pod \"e2e-ipp-b-3805b8-openshift-default-6c9f89869-qcms9\" (UID: \"f75de3cc-f69f-4dfc-9351-9e75a08723b3\") " pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-qcms9"
Nov 02 15:45:04.816322 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:04.816246    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f75de3cc-f69f-4dfc-9351-9e75a08723b3-istio-token\") pod \"e2e-ipp-b-3805b8-openshift-default-6c9f89869-qcms9\" (UID: \"f75de3cc-f69f-4dfc-9351-9e75a08723b3\") " pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-qcms9"
Nov 02 15:45:04.816322 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:04.816305    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f75de3cc-f69f-4dfc-9351-9e75a08723b3-credential-socket\") pod \"e2e-ipp-b-3805b8-openshift-default-6c9f89869-qcms9\" (UID: \"f75de3cc-f69f-4dfc-9351-9e75a08723b3\") " pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-qcms9"
Nov 02 15:45:04.816534 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:04.816355    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f75de3cc-f69f-4dfc-9351-9e75a08723b3-istio-data\") pod \"e2e-ipp-b-3805b8-openshift-default-6c9f89869-qcms9\" (UID: \"f75de3cc-f69f-4dfc-9351-9e75a08723b3\") " pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-qcms9"
Nov 02 15:45:04.816534 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:04.816401    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f75de3cc-f69f-4dfc-9351-9e75a08723b3-istio-podinfo\") pod \"e2e-ipp-b-3805b8-openshift-default-6c9f89869-qcms9\" (UID: \"f75de3cc-f69f-4dfc-9351-9e75a08723b3\") " pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-qcms9"
Nov 02 15:45:04.816534 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:04.816511    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f75de3cc-f69f-4dfc-9351-9e75a08723b3-workload-socket\") pod \"e2e-ipp-b-3805b8-openshift-default-6c9f89869-qcms9\" (UID: \"f75de3cc-f69f-4dfc-9351-9e75a08723b3\") " pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-qcms9"
Nov 02 15:45:04.816700 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:04.816557    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f75de3cc-f69f-4dfc-9351-9e75a08723b3-istio-envoy\") pod \"e2e-ipp-b-3805b8-openshift-default-6c9f89869-qcms9\" (UID: \"f75de3cc-f69f-4dfc-9351-9e75a08723b3\") " pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-qcms9"
Nov 02 15:45:04.816700 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:04.816595    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f75de3cc-f69f-4dfc-9351-9e75a08723b3-istiod-ca-cert\") pod \"e2e-ipp-b-3805b8-openshift-default-6c9f89869-qcms9\" (UID: \"f75de3cc-f69f-4dfc-9351-9e75a08723b3\") " pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-qcms9"
Nov 02 15:45:04.889875 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:04.889813    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893905-4d75m" event={"ID":"9846a62a-c4aa-4937-b1dd-2d9a342dd72a","Type":"ContainerStarted","Data":"50f5621dbfbce7f5a461ba67f09fd343c2886c53e42df3cac2ebddaca3a1b244"}
Nov 02 15:45:04.895998 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:04.895911    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-0a7163-29893905-rkqjw" event={"ID":"e0819318-3392-48ed-a0e4-4d7961144393","Type":"ContainerStarted","Data":"4e57cd5998bbcf2e7d4b654affff6de6e4afbdb852f62f5affa11b8f2477831f"}
Nov 02 15:45:04.902855 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:04.902619    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-78d075-29893905-sznt8" event={"ID":"4cce806c-5164-485a-a790-519176d79e26","Type":"ContainerStarted","Data":"84070c1e85cf71d95dc7ee9181430d56af573025097d6eaaa6757d8bd11f5606"}
Nov 02 15:45:04.910580 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:04.909994    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-b55da1-29893905-mv5qt" event={"ID":"647ad5be-64be-4aff-bff6-d0ac2943671c","Type":"ContainerStarted","Data":"bb48e24350c7e12d61335355d124cad3555e3fc3fb3ee44dfd363812948934d0"}
Nov 02 15:45:04.917457 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:04.917412    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jn78x\" (UniqueName: \"kubernetes.io/projected/f75de3cc-f69f-4dfc-9351-9e75a08723b3-kube-api-access-jn78x\") pod \"e2e-ipp-b-3805b8-openshift-default-6c9f89869-qcms9\" (UID: \"f75de3cc-f69f-4dfc-9351-9e75a08723b3\") " pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-qcms9"
Nov 02 15:45:04.917641 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:04.917492    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f75de3cc-f69f-4dfc-9351-9e75a08723b3-workload-certs\") pod \"e2e-ipp-b-3805b8-openshift-default-6c9f89869-qcms9\" (UID: \"f75de3cc-f69f-4dfc-9351-9e75a08723b3\") " pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-qcms9"
Nov 02 15:45:04.917641 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:04.917554    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f75de3cc-f69f-4dfc-9351-9e75a08723b3-istio-token\") pod \"e2e-ipp-b-3805b8-openshift-default-6c9f89869-qcms9\" (UID: \"f75de3cc-f69f-4dfc-9351-9e75a08723b3\") " pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-qcms9"
Nov 02 15:45:04.917641 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:04.917583    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f75de3cc-f69f-4dfc-9351-9e75a08723b3-credential-socket\") pod \"e2e-ipp-b-3805b8-openshift-default-6c9f89869-qcms9\" (UID: \"f75de3cc-f69f-4dfc-9351-9e75a08723b3\") " pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-qcms9"
Nov 02 15:45:04.917641 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:04.917629    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f75de3cc-f69f-4dfc-9351-9e75a08723b3-istio-data\") pod \"e2e-ipp-b-3805b8-openshift-default-6c9f89869-qcms9\" (UID: \"f75de3cc-f69f-4dfc-9351-9e75a08723b3\") " pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-qcms9"
Nov 02 15:45:04.917879 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:04.917664    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f75de3cc-f69f-4dfc-9351-9e75a08723b3-istio-podinfo\") pod \"e2e-ipp-b-3805b8-openshift-default-6c9f89869-qcms9\" (UID: \"f75de3cc-f69f-4dfc-9351-9e75a08723b3\") " pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-qcms9"
Nov 02 15:45:04.917879 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:04.917731    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f75de3cc-f69f-4dfc-9351-9e75a08723b3-workload-socket\") pod \"e2e-ipp-b-3805b8-openshift-default-6c9f89869-qcms9\" (UID: \"f75de3cc-f69f-4dfc-9351-9e75a08723b3\") " pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-qcms9"
Nov 02 15:45:04.917879 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:04.917774    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f75de3cc-f69f-4dfc-9351-9e75a08723b3-istio-envoy\") pod \"e2e-ipp-b-3805b8-openshift-default-6c9f89869-qcms9\" (UID: \"f75de3cc-f69f-4dfc-9351-9e75a08723b3\") " pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-qcms9"
Nov 02 15:45:04.917879 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:04.917804    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f75de3cc-f69f-4dfc-9351-9e75a08723b3-istiod-ca-cert\") pod \"e2e-ipp-b-3805b8-openshift-default-6c9f89869-qcms9\" (UID: \"f75de3cc-f69f-4dfc-9351-9e75a08723b3\") " pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-qcms9"
Nov 02 15:45:04.918828 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:04.918263    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f75de3cc-f69f-4dfc-9351-9e75a08723b3-workload-certs\") pod \"e2e-ipp-b-3805b8-openshift-default-6c9f89869-qcms9\" (UID: \"f75de3cc-f69f-4dfc-9351-9e75a08723b3\") " pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-qcms9"
Nov 02 15:45:04.918828 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:04.918478    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f75de3cc-f69f-4dfc-9351-9e75a08723b3-istio-data\") pod \"e2e-ipp-b-3805b8-openshift-default-6c9f89869-qcms9\" (UID: \"f75de3cc-f69f-4dfc-9351-9e75a08723b3\") " pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-qcms9"
Nov 02 15:45:04.918828 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:04.918493    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f75de3cc-f69f-4dfc-9351-9e75a08723b3-credential-socket\") pod \"e2e-ipp-b-3805b8-openshift-default-6c9f89869-qcms9\" (UID: \"f75de3cc-f69f-4dfc-9351-9e75a08723b3\") " pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-qcms9"
Nov 02 15:45:04.918828 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:04.918641    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f75de3cc-f69f-4dfc-9351-9e75a08723b3-workload-socket\") pod \"e2e-ipp-b-3805b8-openshift-default-6c9f89869-qcms9\" (UID: \"f75de3cc-f69f-4dfc-9351-9e75a08723b3\") " pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-qcms9"
Nov 02 15:45:04.919192 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:04.919173    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f75de3cc-f69f-4dfc-9351-9e75a08723b3-istiod-ca-cert\") pod \"e2e-ipp-b-3805b8-openshift-default-6c9f89869-qcms9\" (UID: \"f75de3cc-f69f-4dfc-9351-9e75a08723b3\") " pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-qcms9"
Nov 02 15:45:04.954439 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:04.946986    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f75de3cc-f69f-4dfc-9351-9e75a08723b3-istio-podinfo\") pod \"e2e-ipp-b-3805b8-openshift-default-6c9f89869-qcms9\" (UID: \"f75de3cc-f69f-4dfc-9351-9e75a08723b3\") " pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-qcms9"
Nov 02 15:45:04.954439 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:04.947251    5304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-78d075-29893905-sznt8" podStartSLOduration=4.947232888 podStartE2EDuration="4.947232888s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:45:04.94721976 +0000 UTC m=+2904.941044169" watchObservedRunningTime="2026-11-02 15:45:04.947232888 +0000 UTC m=+2904.941057512"
Nov 02 15:45:04.954439 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:04.953632    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f75de3cc-f69f-4dfc-9351-9e75a08723b3-istio-token\") pod \"e2e-ipp-b-3805b8-openshift-default-6c9f89869-qcms9\" (UID: \"f75de3cc-f69f-4dfc-9351-9e75a08723b3\") " pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-qcms9"
Nov 02 15:45:04.954439 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:04.954387    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f75de3cc-f69f-4dfc-9351-9e75a08723b3-istio-envoy\") pod \"e2e-ipp-b-3805b8-openshift-default-6c9f89869-qcms9\" (UID: \"f75de3cc-f69f-4dfc-9351-9e75a08723b3\") " pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-qcms9"
Nov 02 15:45:04.957655 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:04.957567    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jn78x\" (UniqueName: \"kubernetes.io/projected/f75de3cc-f69f-4dfc-9351-9e75a08723b3-kube-api-access-jn78x\") pod \"e2e-ipp-b-3805b8-openshift-default-6c9f89869-qcms9\" (UID: \"f75de3cc-f69f-4dfc-9351-9e75a08723b3\") " pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-qcms9"
Nov 02 15:45:04.995484 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:04.991241    5304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-b55da1-29893905-mv5qt" podStartSLOduration=4.991222633 podStartE2EDuration="4.991222633s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:45:04.991045384 +0000 UTC m=+2904.984869794" watchObservedRunningTime="2026-11-02 15:45:04.991222633 +0000 UTC m=+2904.985047043"
Nov 02 15:45:05.037634 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:05.034412    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-qcms9"
Nov 02 15:45:05.049178 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:05.047385    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-28da58-29893905-phwql"]
Nov 02 15:45:05.062408 ip-10-0-135-138 kubenswrapper[5304]: W1102 15:45:05.061327    5304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podada2b1b0_e598_4d38_b995_becab65c7e82.slice/crio-751a01d9262602427303d4c17f8a024ff86d87c92651f80d971d91d171da9599 WatchSource:0}: Error finding container 751a01d9262602427303d4c17f8a024ff86d87c92651f80d971d91d171da9599: Status 404 returned error can't find the container with id 751a01d9262602427303d4c17f8a024ff86d87c92651f80d971d91d171da9599
Nov 02 15:45:05.096942 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:05.096900    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5113c2-29893905-bnvks"]
Nov 02 15:45:05.439944 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:05.439760    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-qcms9"]
Nov 02 15:45:05.455346 ip-10-0-135-138 kubenswrapper[5304]: W1102 15:45:05.455320    5304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf75de3cc_f69f_4dfc_9351_9e75a08723b3.slice/crio-248579a75b74d73b7c680a64a92d6da161d423f1bd86ce327ce108dd2058bef1 WatchSource:0}: Error finding container 248579a75b74d73b7c680a64a92d6da161d423f1bd86ce327ce108dd2058bef1: Status 404 returned error can't find the container with id 248579a75b74d73b7c680a64a92d6da161d423f1bd86ce327ce108dd2058bef1
Nov 02 15:45:05.459800 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:05.459746    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:45:05.459907 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:05.459865    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:45:05.459972 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:05.459917    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:45:05.693084 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:05.691221    5304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-lvrfc"
Nov 02 15:45:05.693084 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:05.692916    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-lvrfc"
Nov 02 15:45:05.694426 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:05.694373    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-lvrfc"
Nov 02 15:45:05.711174 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:05.709526    5304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-smtdr"
Nov 02 15:45:05.711174 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:05.711006    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-smtdr"
Nov 02 15:45:05.712306 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:05.712256    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-smtdr"
Nov 02 15:45:05.978890 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:05.977774    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-246137-29893905-gxb7t" event={"ID":"1d05aa2a-2dfb-47be-9573-a8f32cf05ab7","Type":"ContainerStarted","Data":"6a539fbf85e392804b881d7357f265242d9cce3825ac8c56e367ba46a09ea28b"}
Nov 02 15:45:05.978890 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:05.977838    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-246137-29893905-gxb7t" event={"ID":"1d05aa2a-2dfb-47be-9573-a8f32cf05ab7","Type":"ContainerStarted","Data":"2683141ec66c16fe4f38385b80727eb37e9455f8c09e7d18725bbfffe511b1a0"}
Nov 02 15:45:05.978890 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:05.978214    5304 scope.go:122] "RemoveContainer" containerID="6a539fbf85e392804b881d7357f265242d9cce3825ac8c56e367ba46a09ea28b"
Nov 02 15:45:06.065116 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:06.065072    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-qcms9" event={"ID":"f75de3cc-f69f-4dfc-9351-9e75a08723b3","Type":"ContainerStarted","Data":"a9889cf15f06da4d089e1f321314b5f4b7903d51460a5dc49ebfa13b542b8c0d"}
Nov 02 15:45:06.153360 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:06.153327    5304 generic.go:356] "Generic (PLEG): container finished" podID="a07ca06a-c083-4fe9-bca4-1614646e80fb" containerID="e9cdab5689982a4df03de184d89b9fe3e0c7461da462354c40795fa9d21ccfbc" exitCode=60
Nov 02 15:45:06.159227 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:06.158987    5304 scope.go:122] "RemoveContainer" containerID="f3380c459dd0edff01989c0546d5756901c8d2cfd4433ed82314787623ecfd0a"
Nov 02 15:45:06.163052 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:06.162935    5304 scope.go:122] "RemoveContainer" containerID="e9cdab5689982a4df03de184d89b9fe3e0c7461da462354c40795fa9d21ccfbc"
Nov 02 15:45:06.189267 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:06.187753    5304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-qcms9" podStartSLOduration=2.187734548 podStartE2EDuration="2.187734548s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:45:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:45:06.185482442 +0000 UTC m=+2906.179306856" watchObservedRunningTime="2026-11-02 15:45:06.187734548 +0000 UTC m=+2906.181558958"
Nov 02 15:45:06.200403 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:06.200362    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-qcms9" event={"ID":"f75de3cc-f69f-4dfc-9351-9e75a08723b3","Type":"ContainerStarted","Data":"248579a75b74d73b7c680a64a92d6da161d423f1bd86ce327ce108dd2058bef1"}
Nov 02 15:45:06.200559 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:06.200457    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-6af50f-29893905-85m66" event={"ID":"8ea83c17-d3cb-4d7e-a6ef-cbd731415216","Type":"ContainerStarted","Data":"f3380c459dd0edff01989c0546d5756901c8d2cfd4433ed82314787623ecfd0a"}
Nov 02 15:45:06.200559 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:06.200510    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-6af50f-29893905-85m66" event={"ID":"8ea83c17-d3cb-4d7e-a6ef-cbd731415216","Type":"ContainerStarted","Data":"c935cd9ad7b009a67e300ef4bd45a2303efdb10ae4e679225e8ab2aa12b65ca0"}
Nov 02 15:45:06.200559 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:06.200537    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-cce791-29893905-qlp7q" event={"ID":"a07ca06a-c083-4fe9-bca4-1614646e80fb","Type":"ContainerDied","Data":"e9cdab5689982a4df03de184d89b9fe3e0c7461da462354c40795fa9d21ccfbc"}
Nov 02 15:45:06.200717 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:06.200575    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-cce791-29893905-qlp7q" event={"ID":"a07ca06a-c083-4fe9-bca4-1614646e80fb","Type":"ContainerStarted","Data":"72f3472a30af011638542fa53cbd87e666e149cccab824824c5ae1a114688158"}
Nov 02 15:45:06.200717 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:06.200607    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5113c2-29893905-bnvks" event={"ID":"5efd887c-66f3-445e-9e39-0d23e43cc9be","Type":"ContainerStarted","Data":"37c516c3fc5c86462904eeea7edbcd9f564b2d81b8d455e94d71c855c651cd02"}
Nov 02 15:45:06.200717 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:06.200642    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5113c2-29893905-bnvks" event={"ID":"5efd887c-66f3-445e-9e39-0d23e43cc9be","Type":"ContainerStarted","Data":"e7114e07c946ae8292b720c549f55b6bf25e5cfab7c6474741235bea40983442"}
Nov 02 15:45:06.211904 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:06.211706    5304 generic.go:356] "Generic (PLEG): container finished" podID="9846a62a-c4aa-4937-b1dd-2d9a342dd72a" containerID="794d1d3f659de4f705cf386c8a39cd3102677ae067fc312731bac9ea70c80df3" exitCode=60
Nov 02 15:45:06.211904 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:06.211848    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893905-4d75m" event={"ID":"9846a62a-c4aa-4937-b1dd-2d9a342dd72a","Type":"ContainerDied","Data":"794d1d3f659de4f705cf386c8a39cd3102677ae067fc312731bac9ea70c80df3"}
Nov 02 15:45:06.212395 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:06.212372    5304 scope.go:122] "RemoveContainer" containerID="794d1d3f659de4f705cf386c8a39cd3102677ae067fc312731bac9ea70c80df3"
Nov 02 15:45:06.223018 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:06.222990    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-1cc1b2-29893905-whfr9" event={"ID":"ea2c4ec8-2ace-49dd-a172-99ced21dbc4f","Type":"ContainerStarted","Data":"f83a2e6a695055b64f5df73566685d5abee9d011b9de1ea4a49b8099d5de735c"}
Nov 02 15:45:06.223197 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:06.223047    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-1cc1b2-29893905-whfr9" event={"ID":"ea2c4ec8-2ace-49dd-a172-99ced21dbc4f","Type":"ContainerStarted","Data":"33ba14be7000721034bb3b3740721f00c59b7c570f9754744178679b1b85d442"}
Nov 02 15:45:06.226378 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:06.226341    5304 generic.go:356] "Generic (PLEG): container finished" podID="e0819318-3392-48ed-a0e4-4d7961144393" containerID="9df713f660da3dfb49e5407ef1c9cbc0e3261c7ab6e0d43bb48c938106157c24" exitCode=60
Nov 02 15:45:06.226481 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:06.226424    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-0a7163-29893905-rkqjw" event={"ID":"e0819318-3392-48ed-a0e4-4d7961144393","Type":"ContainerDied","Data":"9df713f660da3dfb49e5407ef1c9cbc0e3261c7ab6e0d43bb48c938106157c24"}
Nov 02 15:45:06.226763 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:06.226740    5304 scope.go:122] "RemoveContainer" containerID="9df713f660da3dfb49e5407ef1c9cbc0e3261c7ab6e0d43bb48c938106157c24"
Nov 02 15:45:06.228473 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:06.228443    5304 generic.go:356] "Generic (PLEG): container finished" podID="4cce806c-5164-485a-a790-519176d79e26" containerID="1d467003381461610ef9024782d17d87b8b65c35e0b5b68ba14ce7e8ae6bf15a" exitCode=60
Nov 02 15:45:06.228797 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:06.228765    5304 scope.go:122] "RemoveContainer" containerID="1d467003381461610ef9024782d17d87b8b65c35e0b5b68ba14ce7e8ae6bf15a"
Nov 02 15:45:06.229102 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:06.228769    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-78d075-29893905-sznt8" event={"ID":"4cce806c-5164-485a-a790-519176d79e26","Type":"ContainerDied","Data":"1d467003381461610ef9024782d17d87b8b65c35e0b5b68ba14ce7e8ae6bf15a"}
Nov 02 15:45:06.248326 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:06.248262    5304 generic.go:356] "Generic (PLEG): container finished" podID="647ad5be-64be-4aff-bff6-d0ac2943671c" containerID="0f9261b24d66e306a3fd0cfbaa4aa2add9d139fc7cccbb467b230b7033c4b94b" exitCode=60
Nov 02 15:45:06.248528 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:06.248498    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-b55da1-29893905-mv5qt" event={"ID":"647ad5be-64be-4aff-bff6-d0ac2943671c","Type":"ContainerDied","Data":"0f9261b24d66e306a3fd0cfbaa4aa2add9d139fc7cccbb467b230b7033c4b94b"}
Nov 02 15:45:06.249100 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:06.249077    5304 scope.go:122] "RemoveContainer" containerID="0f9261b24d66e306a3fd0cfbaa4aa2add9d139fc7cccbb467b230b7033c4b94b"
Nov 02 15:45:06.253194 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:06.251903    5304 scope.go:122] "RemoveContainer" containerID="9b442641ace68788a36fd9024f95141849e2a1b3c7d6abbf16c8afb6c1e8e1d4"
Nov 02 15:45:06.253194 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:06.252365    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-28da58-29893905-phwql" event={"ID":"ada2b1b0-e598-4d38-b995-becab65c7e82","Type":"ContainerStarted","Data":"9b442641ace68788a36fd9024f95141849e2a1b3c7d6abbf16c8afb6c1e8e1d4"}
Nov 02 15:45:06.253194 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:06.252417    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-28da58-29893905-phwql" event={"ID":"ada2b1b0-e598-4d38-b995-becab65c7e82","Type":"ContainerStarted","Data":"751a01d9262602427303d4c17f8a024ff86d87c92651f80d971d91d171da9599"}
Nov 02 15:45:06.343296 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:06.343095    5304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5113c2-29893905-bnvks" podStartSLOduration=6.343077487 podStartE2EDuration="6.343077487s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:45:06.329749867 +0000 UTC m=+2906.323574275" watchObservedRunningTime="2026-11-02 15:45:06.343077487 +0000 UTC m=+2906.336901916"
Nov 02 15:45:06.508271 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:06.508220    5304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-1cc1b2-29893905-whfr9" podStartSLOduration=6.508205493 podStartE2EDuration="6.508205493s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:45:06.507819281 +0000 UTC m=+2906.501643691" watchObservedRunningTime="2026-11-02 15:45:06.508205493 +0000 UTC m=+2906.502029901"
Nov 02 15:45:06.784859 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:06.784833    5304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d30c0729-9597d"]
Nov 02 15:45:06.788758 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:06.788742    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d30c0729-9597d"
Nov 02 15:45:06.807500 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:06.807476    5304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 15:45:06.810879 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:06.810853    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d30c0729-9597d"]
Nov 02 15:45:06.889411 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:06.889388    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/094c52e5-8049-44dc-a80d-80b5444933ed-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d30c0729-9597d\" (UID: \"094c52e5-8049-44dc-a80d-80b5444933ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d30c0729-9597d"
Nov 02 15:45:06.889574 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:06.889558    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/094c52e5-8049-44dc-a80d-80b5444933ed-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d30c0729-9597d\" (UID: \"094c52e5-8049-44dc-a80d-80b5444933ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d30c0729-9597d"
Nov 02 15:45:06.889636 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:06.889623    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qgjp4\" (UniqueName: \"kubernetes.io/projected/094c52e5-8049-44dc-a80d-80b5444933ed-kube-api-access-qgjp4\") pod \"maas-api-revoke-sub-d30c0729-9597d\" (UID: \"094c52e5-8049-44dc-a80d-80b5444933ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d30c0729-9597d"
Nov 02 15:45:06.991253 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:06.991218    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/094c52e5-8049-44dc-a80d-80b5444933ed-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d30c0729-9597d\" (UID: \"094c52e5-8049-44dc-a80d-80b5444933ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d30c0729-9597d"
Nov 02 15:45:06.991675 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:06.991411    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/094c52e5-8049-44dc-a80d-80b5444933ed-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d30c0729-9597d\" (UID: \"094c52e5-8049-44dc-a80d-80b5444933ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d30c0729-9597d"
Nov 02 15:45:06.991675 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:06.991505    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qgjp4\" (UniqueName: \"kubernetes.io/projected/094c52e5-8049-44dc-a80d-80b5444933ed-kube-api-access-qgjp4\") pod \"maas-api-revoke-sub-d30c0729-9597d\" (UID: \"094c52e5-8049-44dc-a80d-80b5444933ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d30c0729-9597d"
Nov 02 15:45:06.992071 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:06.992053    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/094c52e5-8049-44dc-a80d-80b5444933ed-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d30c0729-9597d\" (UID: \"094c52e5-8049-44dc-a80d-80b5444933ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d30c0729-9597d"
Nov 02 15:45:06.993443 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:06.993422    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/094c52e5-8049-44dc-a80d-80b5444933ed-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d30c0729-9597d\" (UID: \"094c52e5-8049-44dc-a80d-80b5444933ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d30c0729-9597d"
Nov 02 15:45:07.019495 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:07.019435    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qgjp4\" (UniqueName: \"kubernetes.io/projected/094c52e5-8049-44dc-a80d-80b5444933ed-kube-api-access-qgjp4\") pod \"maas-api-revoke-sub-d30c0729-9597d\" (UID: \"094c52e5-8049-44dc-a80d-80b5444933ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d30c0729-9597d"
Nov 02 15:45:07.060470 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:07.060381    5304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-qcms9"
Nov 02 15:45:07.061055 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:07.061030    5304 patch_prober.go:29] interesting pod/e2e-ipp-b-3805b8-openshift-default-6c9f89869-qcms9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.188:15021/healthz/ready\": dial tcp 10.133.0.188:15021: connect: connection refused" start-of-body=
Nov 02 15:45:07.061213 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:07.061090    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-qcms9" podUID="f75de3cc-f69f-4dfc-9351-9e75a08723b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.188:15021/healthz/ready\": dial tcp 10.133.0.188:15021: connect: connection refused"
Nov 02 15:45:07.101654 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:07.101622    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d30c0729-9597d"
Nov 02 15:45:07.270249 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:07.268336    5304 generic.go:356] "Generic (PLEG): container finished" podID="ea2c4ec8-2ace-49dd-a172-99ced21dbc4f" containerID="f83a2e6a695055b64f5df73566685d5abee9d011b9de1ea4a49b8099d5de735c" exitCode=60
Nov 02 15:45:07.270249 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:07.268427    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-1cc1b2-29893905-whfr9" event={"ID":"ea2c4ec8-2ace-49dd-a172-99ced21dbc4f","Type":"ContainerDied","Data":"f83a2e6a695055b64f5df73566685d5abee9d011b9de1ea4a49b8099d5de735c"}
Nov 02 15:45:07.270249 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:07.268927    5304 scope.go:122] "RemoveContainer" containerID="f83a2e6a695055b64f5df73566685d5abee9d011b9de1ea4a49b8099d5de735c"
Nov 02 15:45:07.277443 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:07.277414    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-0a7163-29893905-rkqjw" event={"ID":"e0819318-3392-48ed-a0e4-4d7961144393","Type":"ContainerStarted","Data":"fca4b94c7f454e54863a50ae64ee3ade7aa4f9daaccfd37be27481df18c72746"}
Nov 02 15:45:07.280306 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:07.280278    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-78d075-29893905-sznt8" event={"ID":"4cce806c-5164-485a-a790-519176d79e26","Type":"ContainerStarted","Data":"f868467d6714e57f7e9ac86b2a4e1313ee0846e90ef5f79f944bec4a72981bed"}
Nov 02 15:45:07.282865 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:07.282839    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-b55da1-29893905-mv5qt" event={"ID":"647ad5be-64be-4aff-bff6-d0ac2943671c","Type":"ContainerStarted","Data":"b59c1235242ea6c32f4df009d662d7a874369a0584275ba8454bf5d8713704b1"}
Nov 02 15:45:07.284973 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:07.284951    5304 generic.go:356] "Generic (PLEG): container finished" podID="ada2b1b0-e598-4d38-b995-becab65c7e82" containerID="9b442641ace68788a36fd9024f95141849e2a1b3c7d6abbf16c8afb6c1e8e1d4" exitCode=60
Nov 02 15:45:07.285168 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:07.284990    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-28da58-29893905-phwql" event={"ID":"ada2b1b0-e598-4d38-b995-becab65c7e82","Type":"ContainerDied","Data":"9b442641ace68788a36fd9024f95141849e2a1b3c7d6abbf16c8afb6c1e8e1d4"}
Nov 02 15:45:07.285850 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:07.285830    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-28da58-29893905-phwql" event={"ID":"ada2b1b0-e598-4d38-b995-becab65c7e82","Type":"ContainerStarted","Data":"1f8869f2c5a853c053b524b14968d0f0478b4ed819327f67881baee4bb902cd3"}
Nov 02 15:45:07.287839 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:07.287350    5304 generic.go:356] "Generic (PLEG): container finished" podID="1d05aa2a-2dfb-47be-9573-a8f32cf05ab7" containerID="6a539fbf85e392804b881d7357f265242d9cce3825ac8c56e367ba46a09ea28b" exitCode=60
Nov 02 15:45:07.287839 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:07.287413    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-246137-29893905-gxb7t" event={"ID":"1d05aa2a-2dfb-47be-9573-a8f32cf05ab7","Type":"ContainerDied","Data":"6a539fbf85e392804b881d7357f265242d9cce3825ac8c56e367ba46a09ea28b"}
Nov 02 15:45:07.287839 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:07.287440    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-246137-29893905-gxb7t" event={"ID":"1d05aa2a-2dfb-47be-9573-a8f32cf05ab7","Type":"ContainerStarted","Data":"4e8879989f2a249b835f019d34f10353e22d01579f40ba2ca60807a15c74ae45"}
Nov 02 15:45:07.290221 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:07.289725    5304 generic.go:356] "Generic (PLEG): container finished" podID="8ea83c17-d3cb-4d7e-a6ef-cbd731415216" containerID="f3380c459dd0edff01989c0546d5756901c8d2cfd4433ed82314787623ecfd0a" exitCode=60
Nov 02 15:45:07.290221 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:07.289787    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-6af50f-29893905-85m66" event={"ID":"8ea83c17-d3cb-4d7e-a6ef-cbd731415216","Type":"ContainerDied","Data":"f3380c459dd0edff01989c0546d5756901c8d2cfd4433ed82314787623ecfd0a"}
Nov 02 15:45:07.290221 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:07.289814    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-6af50f-29893905-85m66" event={"ID":"8ea83c17-d3cb-4d7e-a6ef-cbd731415216","Type":"ContainerStarted","Data":"9eb5a5fb998ead1c4afd489e63c3aa159c305084578d0db9d6b1e940bae767e3"}
Nov 02 15:45:07.292651 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:07.292244    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-cce791-29893905-qlp7q" event={"ID":"a07ca06a-c083-4fe9-bca4-1614646e80fb","Type":"ContainerStarted","Data":"a36cce492b2e27474d39fede54de27a69c83675430f04fab29552a0f2f02f7eb"}
Nov 02 15:45:07.294628 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:07.294046    5304 generic.go:356] "Generic (PLEG): container finished" podID="5efd887c-66f3-445e-9e39-0d23e43cc9be" containerID="37c516c3fc5c86462904eeea7edbcd9f564b2d81b8d455e94d71c855c651cd02" exitCode=60
Nov 02 15:45:07.294628 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:07.294104    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5113c2-29893905-bnvks" event={"ID":"5efd887c-66f3-445e-9e39-0d23e43cc9be","Type":"ContainerDied","Data":"37c516c3fc5c86462904eeea7edbcd9f564b2d81b8d455e94d71c855c651cd02"}
Nov 02 15:45:07.294628 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:07.294348    5304 scope.go:122] "RemoveContainer" containerID="37c516c3fc5c86462904eeea7edbcd9f564b2d81b8d455e94d71c855c651cd02"
Nov 02 15:45:07.303610 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:07.302545    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893905-4d75m" event={"ID":"9846a62a-c4aa-4937-b1dd-2d9a342dd72a","Type":"ContainerStarted","Data":"b4c582b9e99a38343bbbcc682867a2038c2f593d79f8fa8c52ed540c573c122f"}
Nov 02 15:45:07.472474 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:07.472414    5304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893905-4d75m" podStartSLOduration=7.472396398 podStartE2EDuration="7.472396398s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:45:07.465006025 +0000 UTC m=+2907.458830436" watchObservedRunningTime="2026-11-02 15:45:07.472396398 +0000 UTC m=+2907.466220811"
Nov 02 15:45:07.473531 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:07.473509    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d30c0729-9597d"]
Nov 02 15:45:07.547714 ip-10-0-135-138 kubenswrapper[5304]: W1102 15:45:07.547682    5304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod094c52e5_8049_44dc_a80d_80b5444933ed.slice/crio-be81d39a41061309eea9d86b4f454448bf41ea537a3b86f93363e4cb9d9b580e WatchSource:0}: Error finding container be81d39a41061309eea9d86b4f454448bf41ea537a3b86f93363e4cb9d9b580e: Status 404 returned error can't find the container with id be81d39a41061309eea9d86b4f454448bf41ea537a3b86f93363e4cb9d9b580e
Nov 02 15:45:07.661860 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:07.661812    5304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-0a7163-29893905-rkqjw" podStartSLOduration=7.661798781 podStartE2EDuration="7.661798781s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:45:07.659557564 +0000 UTC m=+2907.653381974" watchObservedRunningTime="2026-11-02 15:45:07.661798781 +0000 UTC m=+2907.655623189"
Nov 02 15:45:07.702760 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:07.702711    5304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-cce791-29893905-qlp7q" podStartSLOduration=7.7026971379999996 podStartE2EDuration="7.702697138s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:45:07.701484173 +0000 UTC m=+2907.695308573" watchObservedRunningTime="2026-11-02 15:45:07.702697138 +0000 UTC m=+2907.696521546"
Nov 02 15:45:07.804860 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:07.804479    5304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-246137-29893905-gxb7t" podStartSLOduration=7.804457541 podStartE2EDuration="7.804457541s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:45:07.801048885 +0000 UTC m=+2907.794873293" watchObservedRunningTime="2026-11-02 15:45:07.804457541 +0000 UTC m=+2907.798281951"
Nov 02 15:45:07.889398 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:07.889342    5304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-6af50f-29893905-85m66" podStartSLOduration=7.889328069 podStartE2EDuration="7.889328069s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:45:07.888434556 +0000 UTC m=+2907.882258966" watchObservedRunningTime="2026-11-02 15:45:07.889328069 +0000 UTC m=+2907.883152479"
Nov 02 15:45:07.950593 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:07.950515    5304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-28da58-29893905-phwql" podStartSLOduration=7.950494608 podStartE2EDuration="7.950494608s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:45:07.946512362 +0000 UTC m=+2907.940336772" watchObservedRunningTime="2026-11-02 15:45:07.950494608 +0000 UTC m=+2907.944319018"
Nov 02 15:45:08.046158 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:08.045684    5304 patch_prober.go:29] interesting pod/e2e-ipp-b-3805b8-openshift-default-6c9f89869-qcms9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.188:15021/healthz/ready\": dial tcp 10.133.0.188:15021: connect: connection refused" start-of-body=
Nov 02 15:45:08.046158 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:08.045743    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-qcms9" podUID="f75de3cc-f69f-4dfc-9351-9e75a08723b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.188:15021/healthz/ready\": dial tcp 10.133.0.188:15021: connect: connection refused"
Nov 02 15:45:08.275558 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:08.275525    5304 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-wxjl7_2f905a4e-b982-4d66-a7bc-171cda1c78dd/payload-processing/0.log"
Nov 02 15:45:08.311753 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:08.311718    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d30c0729-9597d" event={"ID":"094c52e5-8049-44dc-a80d-80b5444933ed","Type":"ContainerStarted","Data":"be81d39a41061309eea9d86b4f454448bf41ea537a3b86f93363e4cb9d9b580e"}
Nov 02 15:45:08.313482 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:08.313455    5304 generic.go:356] "Generic (PLEG): container finished" podID="1d05aa2a-2dfb-47be-9573-a8f32cf05ab7" containerID="4e8879989f2a249b835f019d34f10353e22d01579f40ba2ca60807a15c74ae45" exitCode=60
Nov 02 15:45:08.313704 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:08.313682    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-246137-29893905-gxb7t" event={"ID":"1d05aa2a-2dfb-47be-9573-a8f32cf05ab7","Type":"ContainerDied","Data":"4e8879989f2a249b835f019d34f10353e22d01579f40ba2ca60807a15c74ae45"}
Nov 02 15:45:08.313749 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:08.313739    5304 scope.go:122] "RemoveContainer" containerID="6a539fbf85e392804b881d7357f265242d9cce3825ac8c56e367ba46a09ea28b"
Nov 02 15:45:08.313824 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:08.313810    5304 scope.go:122] "RemoveContainer" containerID="4e8879989f2a249b835f019d34f10353e22d01579f40ba2ca60807a15c74ae45"
Nov 02 15:45:08.314107 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:45:08.314091    5304 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w5-246137-29893905-gxb7t_odh-ai-gateway-infra(1d05aa2a-2dfb-47be-9573-a8f32cf05ab7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-246137-29893905-gxb7t" podUID="1d05aa2a-2dfb-47be-9573-a8f32cf05ab7"
Nov 02 15:45:08.316701 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:08.316677    5304 generic.go:356] "Generic (PLEG): container finished" podID="8ea83c17-d3cb-4d7e-a6ef-cbd731415216" containerID="9eb5a5fb998ead1c4afd489e63c3aa159c305084578d0db9d6b1e940bae767e3" exitCode=60
Nov 02 15:45:08.316798 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:08.316732    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-6af50f-29893905-85m66" event={"ID":"8ea83c17-d3cb-4d7e-a6ef-cbd731415216","Type":"ContainerDied","Data":"9eb5a5fb998ead1c4afd489e63c3aa159c305084578d0db9d6b1e940bae767e3"}
Nov 02 15:45:08.317036 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:08.317023    5304 scope.go:122] "RemoveContainer" containerID="9eb5a5fb998ead1c4afd489e63c3aa159c305084578d0db9d6b1e940bae767e3"
Nov 02 15:45:08.317378 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:45:08.317352    5304 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-6af50f-29893905-85m66_odh-ai-gateway-infra(8ea83c17-d3cb-4d7e-a6ef-cbd731415216)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-6af50f-29893905-85m66" podUID="8ea83c17-d3cb-4d7e-a6ef-cbd731415216"
Nov 02 15:45:08.318618 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:08.318601    5304 generic.go:356] "Generic (PLEG): container finished" podID="a07ca06a-c083-4fe9-bca4-1614646e80fb" containerID="a36cce492b2e27474d39fede54de27a69c83675430f04fab29552a0f2f02f7eb" exitCode=60
Nov 02 15:45:08.319227 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:08.318672    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-cce791-29893905-qlp7q" event={"ID":"a07ca06a-c083-4fe9-bca4-1614646e80fb","Type":"ContainerDied","Data":"a36cce492b2e27474d39fede54de27a69c83675430f04fab29552a0f2f02f7eb"}
Nov 02 15:45:08.319227 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:08.318938    5304 scope.go:122] "RemoveContainer" containerID="a36cce492b2e27474d39fede54de27a69c83675430f04fab29552a0f2f02f7eb"
Nov 02 15:45:08.319227 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:45:08.319183    5304 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-cce791-29893905-qlp7q_odh-ai-gateway-infra(a07ca06a-c083-4fe9-bca4-1614646e80fb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-cce791-29893905-qlp7q" podUID="a07ca06a-c083-4fe9-bca4-1614646e80fb"
Nov 02 15:45:08.321463 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:08.321442    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5113c2-29893905-bnvks" event={"ID":"5efd887c-66f3-445e-9e39-0d23e43cc9be","Type":"ContainerStarted","Data":"c9a1497d61cf44d6c2fc56cad2b62203dacb573aec8a6b2389b1a14a5bb5a4cf"}
Nov 02 15:45:08.323477 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:08.323459    5304 generic.go:356] "Generic (PLEG): container finished" podID="9846a62a-c4aa-4937-b1dd-2d9a342dd72a" containerID="b4c582b9e99a38343bbbcc682867a2038c2f593d79f8fa8c52ed540c573c122f" exitCode=60
Nov 02 15:45:08.323558 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:08.323545    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893905-4d75m" event={"ID":"9846a62a-c4aa-4937-b1dd-2d9a342dd72a","Type":"ContainerDied","Data":"b4c582b9e99a38343bbbcc682867a2038c2f593d79f8fa8c52ed540c573c122f"}
Nov 02 15:45:08.323821 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:08.323807    5304 scope.go:122] "RemoveContainer" containerID="b4c582b9e99a38343bbbcc682867a2038c2f593d79f8fa8c52ed540c573c122f"
Nov 02 15:45:08.324022 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:45:08.324005    5304 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29893905-4d75m_odh-ai-gateway-infra(9846a62a-c4aa-4937-b1dd-2d9a342dd72a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893905-4d75m" podUID="9846a62a-c4aa-4937-b1dd-2d9a342dd72a"
Nov 02 15:45:08.325879 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:08.325860    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-1cc1b2-29893905-whfr9" event={"ID":"ea2c4ec8-2ace-49dd-a172-99ced21dbc4f","Type":"ContainerStarted","Data":"1bd49a527b9b72cc455bd79351cd7c9e8119b3a457e0f6f430d422c85a007b95"}
Nov 02 15:45:08.326841 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:08.326824    5304 scope.go:122] "RemoveContainer" containerID="f3380c459dd0edff01989c0546d5756901c8d2cfd4433ed82314787623ecfd0a"
Nov 02 15:45:08.327884 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:08.327864    5304 generic.go:356] "Generic (PLEG): container finished" podID="e0819318-3392-48ed-a0e4-4d7961144393" containerID="fca4b94c7f454e54863a50ae64ee3ade7aa4f9daaccfd37be27481df18c72746" exitCode=60
Nov 02 15:45:08.328090 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:08.328066    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-0a7163-29893905-rkqjw" event={"ID":"e0819318-3392-48ed-a0e4-4d7961144393","Type":"ContainerDied","Data":"fca4b94c7f454e54863a50ae64ee3ade7aa4f9daaccfd37be27481df18c72746"}
Nov 02 15:45:08.328273 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:08.328257    5304 scope.go:122] "RemoveContainer" containerID="fca4b94c7f454e54863a50ae64ee3ade7aa4f9daaccfd37be27481df18c72746"
Nov 02 15:45:08.328507 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:45:08.328485    5304 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w5-0a7163-29893905-rkqjw_odh-ai-gateway-infra(e0819318-3392-48ed-a0e4-4d7961144393)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-0a7163-29893905-rkqjw" podUID="e0819318-3392-48ed-a0e4-4d7961144393"
Nov 02 15:45:08.329808 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:08.329789    5304 generic.go:356] "Generic (PLEG): container finished" podID="4cce806c-5164-485a-a790-519176d79e26" containerID="f868467d6714e57f7e9ac86b2a4e1313ee0846e90ef5f79f944bec4a72981bed" exitCode=60
Nov 02 15:45:08.330078 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:08.330061    5304 scope.go:122] "RemoveContainer" containerID="f868467d6714e57f7e9ac86b2a4e1313ee0846e90ef5f79f944bec4a72981bed"
Nov 02 15:45:08.330177 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:08.329897    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-78d075-29893905-sznt8" event={"ID":"4cce806c-5164-485a-a790-519176d79e26","Type":"ContainerDied","Data":"f868467d6714e57f7e9ac86b2a4e1313ee0846e90ef5f79f944bec4a72981bed"}
Nov 02 15:45:08.349224 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:45:08.348646    5304 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-78d075-29893905-sznt8_odh-ai-gateway-infra(4cce806c-5164-485a-a790-519176d79e26)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-78d075-29893905-sznt8" podUID="4cce806c-5164-485a-a790-519176d79e26"
Nov 02 15:45:08.357158 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:08.354025    5304 generic.go:356] "Generic (PLEG): container finished" podID="647ad5be-64be-4aff-bff6-d0ac2943671c" containerID="b59c1235242ea6c32f4df009d662d7a874369a0584275ba8454bf5d8713704b1" exitCode=60
Nov 02 15:45:08.357342 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:08.357168    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-b55da1-29893905-mv5qt" event={"ID":"647ad5be-64be-4aff-bff6-d0ac2943671c","Type":"ContainerDied","Data":"b59c1235242ea6c32f4df009d662d7a874369a0584275ba8454bf5d8713704b1"}
Nov 02 15:45:08.357643 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:08.357625    5304 scope.go:122] "RemoveContainer" containerID="b59c1235242ea6c32f4df009d662d7a874369a0584275ba8454bf5d8713704b1"
Nov 02 15:45:08.358642 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:45:08.358621    5304 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w6-b55da1-29893905-mv5qt_odh-ai-gateway-infra(647ad5be-64be-4aff-bff6-d0ac2943671c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-b55da1-29893905-mv5qt" podUID="647ad5be-64be-4aff-bff6-d0ac2943671c"
Nov 02 15:45:08.361065 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:08.361046    5304 generic.go:356] "Generic (PLEG): container finished" podID="ada2b1b0-e598-4d38-b995-becab65c7e82" containerID="1f8869f2c5a853c053b524b14968d0f0478b4ed819327f67881baee4bb902cd3" exitCode=60
Nov 02 15:45:08.361269 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:08.361249    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-28da58-29893905-phwql" event={"ID":"ada2b1b0-e598-4d38-b995-becab65c7e82","Type":"ContainerDied","Data":"1f8869f2c5a853c053b524b14968d0f0478b4ed819327f67881baee4bb902cd3"}
Nov 02 15:45:08.361407 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:08.361389    5304 scope.go:122] "RemoveContainer" containerID="1f8869f2c5a853c053b524b14968d0f0478b4ed819327f67881baee4bb902cd3"
Nov 02 15:45:08.361679 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:45:08.361658    5304 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-28da58-29893905-phwql_odh-ai-gateway-infra(ada2b1b0-e598-4d38-b995-becab65c7e82)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-28da58-29893905-phwql" podUID="ada2b1b0-e598-4d38-b995-becab65c7e82"
Nov 02 15:45:08.369777 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:08.369761    5304 scope.go:122] "RemoveContainer" containerID="e9cdab5689982a4df03de184d89b9fe3e0c7461da462354c40795fa9d21ccfbc"
Nov 02 15:45:08.384027 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:08.384005    5304 scope.go:122] "RemoveContainer" containerID="794d1d3f659de4f705cf386c8a39cd3102677ae067fc312731bac9ea70c80df3"
Nov 02 15:45:08.397995 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:08.397937    5304 scope.go:122] "RemoveContainer" containerID="9df713f660da3dfb49e5407ef1c9cbc0e3261c7ab6e0d43bb48c938106157c24"
Nov 02 15:45:08.411324 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:08.411300    5304 scope.go:122] "RemoveContainer" containerID="1d467003381461610ef9024782d17d87b8b65c35e0b5b68ba14ce7e8ae6bf15a"
Nov 02 15:45:08.424945 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:08.424912    5304 scope.go:122] "RemoveContainer" containerID="0f9261b24d66e306a3fd0cfbaa4aa2add9d139fc7cccbb467b230b7033c4b94b"
Nov 02 15:45:08.467526 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:08.467507    5304 scope.go:122] "RemoveContainer" containerID="9b442641ace68788a36fd9024f95141849e2a1b3c7d6abbf16c8afb6c1e8e1d4"
Nov 02 15:45:08.483777 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:08.483758    5304 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-wxjl7_2f905a4e-b982-4d66-a7bc-171cda1c78dd/payload-processing/0.log"
Nov 02 15:45:09.053186 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:09.052656    5304 patch_prober.go:29] interesting pod/e2e-ipp-b-3805b8-openshift-default-6c9f89869-qcms9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.188:15021/healthz/ready\": dial tcp 10.133.0.188:15021: connect: connection refused" start-of-body=
Nov 02 15:45:09.053186 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:09.052723    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-qcms9" podUID="f75de3cc-f69f-4dfc-9351-9e75a08723b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.188:15021/healthz/ready\": dial tcp 10.133.0.188:15021: connect: connection refused"
Nov 02 15:45:09.385594 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:09.385565    5304 scope.go:122] "RemoveContainer" containerID="b59c1235242ea6c32f4df009d662d7a874369a0584275ba8454bf5d8713704b1"
Nov 02 15:45:09.386235 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:45:09.386201    5304 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w6-b55da1-29893905-mv5qt_odh-ai-gateway-infra(647ad5be-64be-4aff-bff6-d0ac2943671c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-b55da1-29893905-mv5qt" podUID="647ad5be-64be-4aff-bff6-d0ac2943671c"
Nov 02 15:45:09.387785 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:09.387763    5304 scope.go:122] "RemoveContainer" containerID="1f8869f2c5a853c053b524b14968d0f0478b4ed819327f67881baee4bb902cd3"
Nov 02 15:45:09.388072 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:45:09.388052    5304 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-28da58-29893905-phwql_odh-ai-gateway-infra(ada2b1b0-e598-4d38-b995-becab65c7e82)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-28da58-29893905-phwql" podUID="ada2b1b0-e598-4d38-b995-becab65c7e82"
Nov 02 15:45:09.389044 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:09.389021    5304 generic.go:356] "Generic (PLEG): container finished" podID="094c52e5-8049-44dc-a80d-80b5444933ed" containerID="96ea1c70384ba8e82a5d8e2d54cea95e3177263337f5f57689a238f43e01a60a" exitCode=0
Nov 02 15:45:09.389155 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:09.389089    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d30c0729-9597d" event={"ID":"094c52e5-8049-44dc-a80d-80b5444933ed","Type":"ContainerDied","Data":"96ea1c70384ba8e82a5d8e2d54cea95e3177263337f5f57689a238f43e01a60a"}
Nov 02 15:45:09.391060 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:09.391039    5304 scope.go:122] "RemoveContainer" containerID="4e8879989f2a249b835f019d34f10353e22d01579f40ba2ca60807a15c74ae45"
Nov 02 15:45:09.391354 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:45:09.391331    5304 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w5-246137-29893905-gxb7t_odh-ai-gateway-infra(1d05aa2a-2dfb-47be-9573-a8f32cf05ab7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-246137-29893905-gxb7t" podUID="1d05aa2a-2dfb-47be-9573-a8f32cf05ab7"
Nov 02 15:45:09.392769 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:09.392751    5304 scope.go:122] "RemoveContainer" containerID="9eb5a5fb998ead1c4afd489e63c3aa159c305084578d0db9d6b1e940bae767e3"
Nov 02 15:45:09.393028 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:45:09.393007    5304 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-6af50f-29893905-85m66_odh-ai-gateway-infra(8ea83c17-d3cb-4d7e-a6ef-cbd731415216)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-6af50f-29893905-85m66" podUID="8ea83c17-d3cb-4d7e-a6ef-cbd731415216"
Nov 02 15:45:09.394195 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:09.394181    5304 scope.go:122] "RemoveContainer" containerID="a36cce492b2e27474d39fede54de27a69c83675430f04fab29552a0f2f02f7eb"
Nov 02 15:45:09.394399 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:45:09.394382    5304 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-cce791-29893905-qlp7q_odh-ai-gateway-infra(a07ca06a-c083-4fe9-bca4-1614646e80fb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-cce791-29893905-qlp7q" podUID="a07ca06a-c083-4fe9-bca4-1614646e80fb"
Nov 02 15:45:09.396098 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:09.396076    5304 generic.go:356] "Generic (PLEG): container finished" podID="5efd887c-66f3-445e-9e39-0d23e43cc9be" containerID="c9a1497d61cf44d6c2fc56cad2b62203dacb573aec8a6b2389b1a14a5bb5a4cf" exitCode=60
Nov 02 15:45:09.396203 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:09.396172    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5113c2-29893905-bnvks" event={"ID":"5efd887c-66f3-445e-9e39-0d23e43cc9be","Type":"ContainerDied","Data":"c9a1497d61cf44d6c2fc56cad2b62203dacb573aec8a6b2389b1a14a5bb5a4cf"}
Nov 02 15:45:09.396272 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:09.396238    5304 scope.go:122] "RemoveContainer" containerID="37c516c3fc5c86462904eeea7edbcd9f564b2d81b8d455e94d71c855c651cd02"
Nov 02 15:45:09.396390 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:09.396370    5304 scope.go:122] "RemoveContainer" containerID="c9a1497d61cf44d6c2fc56cad2b62203dacb573aec8a6b2389b1a14a5bb5a4cf"
Nov 02 15:45:09.396622 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:45:09.396602    5304 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-5113c2-29893905-bnvks_odh-ai-gateway-infra(5efd887c-66f3-445e-9e39-0d23e43cc9be)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5113c2-29893905-bnvks" podUID="5efd887c-66f3-445e-9e39-0d23e43cc9be"
Nov 02 15:45:09.398277 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:09.398261    5304 scope.go:122] "RemoveContainer" containerID="b4c582b9e99a38343bbbcc682867a2038c2f593d79f8fa8c52ed540c573c122f"
Nov 02 15:45:09.398443 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:45:09.398427    5304 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29893905-4d75m_odh-ai-gateway-infra(9846a62a-c4aa-4937-b1dd-2d9a342dd72a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893905-4d75m" podUID="9846a62a-c4aa-4937-b1dd-2d9a342dd72a"
Nov 02 15:45:09.400412 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:09.400318    5304 generic.go:356] "Generic (PLEG): container finished" podID="ea2c4ec8-2ace-49dd-a172-99ced21dbc4f" containerID="1bd49a527b9b72cc455bd79351cd7c9e8119b3a457e0f6f430d422c85a007b95" exitCode=60
Nov 02 15:45:09.400412 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:09.400342    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-1cc1b2-29893905-whfr9" event={"ID":"ea2c4ec8-2ace-49dd-a172-99ced21dbc4f","Type":"ContainerDied","Data":"1bd49a527b9b72cc455bd79351cd7c9e8119b3a457e0f6f430d422c85a007b95"}
Nov 02 15:45:09.400638 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:09.400621    5304 scope.go:122] "RemoveContainer" containerID="1bd49a527b9b72cc455bd79351cd7c9e8119b3a457e0f6f430d422c85a007b95"
Nov 02 15:45:09.400901 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:45:09.400880    5304 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-1cc1b2-29893905-whfr9_odh-ai-gateway-infra(ea2c4ec8-2ace-49dd-a172-99ced21dbc4f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-1cc1b2-29893905-whfr9" podUID="ea2c4ec8-2ace-49dd-a172-99ced21dbc4f"
Nov 02 15:45:09.403744 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:09.403730    5304 scope.go:122] "RemoveContainer" containerID="fca4b94c7f454e54863a50ae64ee3ade7aa4f9daaccfd37be27481df18c72746"
Nov 02 15:45:09.404319 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:45:09.404291    5304 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w5-0a7163-29893905-rkqjw_odh-ai-gateway-infra(e0819318-3392-48ed-a0e4-4d7961144393)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-0a7163-29893905-rkqjw" podUID="e0819318-3392-48ed-a0e4-4d7961144393"
Nov 02 15:45:09.405442 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:09.405427    5304 scope.go:122] "RemoveContainer" containerID="f868467d6714e57f7e9ac86b2a4e1313ee0846e90ef5f79f944bec4a72981bed"
Nov 02 15:45:09.405619 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:45:09.405603    5304 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-78d075-29893905-sznt8_odh-ai-gateway-infra(4cce806c-5164-485a-a790-519176d79e26)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-78d075-29893905-sznt8" podUID="4cce806c-5164-485a-a790-519176d79e26"
Nov 02 15:45:09.409773 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:09.409756    5304 scope.go:122] "RemoveContainer" containerID="f83a2e6a695055b64f5df73566685d5abee9d011b9de1ea4a49b8099d5de735c"
Nov 02 15:45:09.760235 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:09.759438    5304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f84ad8-nn4jm"]
Nov 02 15:45:09.768172 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:09.764675    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f84ad8-nn4jm"
Nov 02 15:45:09.780593 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:09.780566    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f84ad8-nn4jm"]
Nov 02 15:45:09.865703 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:09.865669    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97f24cb0-604f-4969-ba8b-90dd2373c61a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-f84ad8-nn4jm\" (UID: \"97f24cb0-604f-4969-ba8b-90dd2373c61a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f84ad8-nn4jm"
Nov 02 15:45:09.865973 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:09.865950    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cc78n\" (UniqueName: \"kubernetes.io/projected/97f24cb0-604f-4969-ba8b-90dd2373c61a-kube-api-access-cc78n\") pod \"maas-api-revoke-keys-e2e-ipp-a-f84ad8-nn4jm\" (UID: \"97f24cb0-604f-4969-ba8b-90dd2373c61a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f84ad8-nn4jm"
Nov 02 15:45:09.866174 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:09.866158    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97f24cb0-604f-4969-ba8b-90dd2373c61a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-f84ad8-nn4jm\" (UID: \"97f24cb0-604f-4969-ba8b-90dd2373c61a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f84ad8-nn4jm"
Nov 02 15:45:09.971160 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:09.970655    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97f24cb0-604f-4969-ba8b-90dd2373c61a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-f84ad8-nn4jm\" (UID: \"97f24cb0-604f-4969-ba8b-90dd2373c61a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f84ad8-nn4jm"
Nov 02 15:45:09.971160 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:09.970742    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cc78n\" (UniqueName: \"kubernetes.io/projected/97f24cb0-604f-4969-ba8b-90dd2373c61a-kube-api-access-cc78n\") pod \"maas-api-revoke-keys-e2e-ipp-a-f84ad8-nn4jm\" (UID: \"97f24cb0-604f-4969-ba8b-90dd2373c61a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f84ad8-nn4jm"
Nov 02 15:45:09.971160 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:09.970823    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97f24cb0-604f-4969-ba8b-90dd2373c61a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-f84ad8-nn4jm\" (UID: \"97f24cb0-604f-4969-ba8b-90dd2373c61a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f84ad8-nn4jm"
Nov 02 15:45:09.972477 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:09.972416    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97f24cb0-604f-4969-ba8b-90dd2373c61a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-f84ad8-nn4jm\" (UID: \"97f24cb0-604f-4969-ba8b-90dd2373c61a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f84ad8-nn4jm"
Nov 02 15:45:09.994867 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:09.994841    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97f24cb0-604f-4969-ba8b-90dd2373c61a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-f84ad8-nn4jm\" (UID: \"97f24cb0-604f-4969-ba8b-90dd2373c61a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f84ad8-nn4jm"
Nov 02 15:45:10.002456 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:10.002425    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cc78n\" (UniqueName: \"kubernetes.io/projected/97f24cb0-604f-4969-ba8b-90dd2373c61a-kube-api-access-cc78n\") pod \"maas-api-revoke-keys-e2e-ipp-a-f84ad8-nn4jm\" (UID: \"97f24cb0-604f-4969-ba8b-90dd2373c61a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f84ad8-nn4jm"
Nov 02 15:45:10.043687 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:10.043596    5304 patch_prober.go:29] interesting pod/e2e-ipp-b-3805b8-openshift-default-6c9f89869-qcms9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.188:15021/healthz/ready\": dial tcp 10.133.0.188:15021: connect: connection refused" start-of-body=
Nov 02 15:45:10.043968 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:10.043937    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-qcms9" podUID="f75de3cc-f69f-4dfc-9351-9e75a08723b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.188:15021/healthz/ready\": dial tcp 10.133.0.188:15021: connect: connection refused"
Nov 02 15:45:10.085484 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:10.085451    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f84ad8-nn4jm"
Nov 02 15:45:10.414448 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:10.414400    5304 scope.go:122] "RemoveContainer" containerID="1bd49a527b9b72cc455bd79351cd7c9e8119b3a457e0f6f430d422c85a007b95"
Nov 02 15:45:10.414830 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:45:10.414800    5304 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-1cc1b2-29893905-whfr9_odh-ai-gateway-infra(ea2c4ec8-2ace-49dd-a172-99ced21dbc4f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-1cc1b2-29893905-whfr9" podUID="ea2c4ec8-2ace-49dd-a172-99ced21dbc4f"
Nov 02 15:45:10.421873 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:10.421440    5304 scope.go:122] "RemoveContainer" containerID="c9a1497d61cf44d6c2fc56cad2b62203dacb573aec8a6b2389b1a14a5bb5a4cf"
Nov 02 15:45:10.421873 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:10.421599    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f84ad8-nn4jm"]
Nov 02 15:45:10.421873 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:45:10.421826    5304 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-5113c2-29893905-bnvks_odh-ai-gateway-infra(5efd887c-66f3-445e-9e39-0d23e43cc9be)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5113c2-29893905-bnvks" podUID="5efd887c-66f3-445e-9e39-0d23e43cc9be"
Nov 02 15:45:10.467147 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:10.467067    5304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-797488-openshift-default-5f79c8745f-7mjt4"]
Nov 02 15:45:10.474623 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:10.473289    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-797488-openshift-default-5f79c8745f-7mjt4"
Nov 02 15:45:10.491290 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:10.491065    5304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-797488-openshift-default-dockercfg-crhd7\""
Nov 02 15:45:10.493963 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:10.493935    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-797488-openshift-default-5f79c8745f-7mjt4"]
Nov 02 15:45:10.684392 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:10.684365    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d30c0729-9597d"
Nov 02 15:45:10.687016 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:10.686994    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ecdcaba8-df41-4c1b-a442-d03f9994a5e4-istio-data\") pod \"e2e-shared-b-w2-797488-openshift-default-5f79c8745f-7mjt4\" (UID: \"ecdcaba8-df41-4c1b-a442-d03f9994a5e4\") " pod="openshift-ingress/e2e-shared-b-w2-797488-openshift-default-5f79c8745f-7mjt4"
Nov 02 15:45:10.687120 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:10.687061    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ecdcaba8-df41-4c1b-a442-d03f9994a5e4-istio-podinfo\") pod \"e2e-shared-b-w2-797488-openshift-default-5f79c8745f-7mjt4\" (UID: \"ecdcaba8-df41-4c1b-a442-d03f9994a5e4\") " pod="openshift-ingress/e2e-shared-b-w2-797488-openshift-default-5f79c8745f-7mjt4"
Nov 02 15:45:10.687272 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:10.687198    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ecdcaba8-df41-4c1b-a442-d03f9994a5e4-istiod-ca-cert\") pod \"e2e-shared-b-w2-797488-openshift-default-5f79c8745f-7mjt4\" (UID: \"ecdcaba8-df41-4c1b-a442-d03f9994a5e4\") " pod="openshift-ingress/e2e-shared-b-w2-797488-openshift-default-5f79c8745f-7mjt4"
Nov 02 15:45:10.687272 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:10.687234    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ecdcaba8-df41-4c1b-a442-d03f9994a5e4-workload-certs\") pod \"e2e-shared-b-w2-797488-openshift-default-5f79c8745f-7mjt4\" (UID: \"ecdcaba8-df41-4c1b-a442-d03f9994a5e4\") " pod="openshift-ingress/e2e-shared-b-w2-797488-openshift-default-5f79c8745f-7mjt4"
Nov 02 15:45:10.687350 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:10.687274    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecdcaba8-df41-4c1b-a442-d03f9994a5e4-credential-socket\") pod \"e2e-shared-b-w2-797488-openshift-default-5f79c8745f-7mjt4\" (UID: \"ecdcaba8-df41-4c1b-a442-d03f9994a5e4\") " pod="openshift-ingress/e2e-shared-b-w2-797488-openshift-default-5f79c8745f-7mjt4"
Nov 02 15:45:10.687412 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:10.687396    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ecdcaba8-df41-4c1b-a442-d03f9994a5e4-istio-token\") pod \"e2e-shared-b-w2-797488-openshift-default-5f79c8745f-7mjt4\" (UID: \"ecdcaba8-df41-4c1b-a442-d03f9994a5e4\") " pod="openshift-ingress/e2e-shared-b-w2-797488-openshift-default-5f79c8745f-7mjt4"
Nov 02 15:45:10.687457 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:10.687445    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d6fqb\" (UniqueName: \"kubernetes.io/projected/ecdcaba8-df41-4c1b-a442-d03f9994a5e4-kube-api-access-d6fqb\") pod \"e2e-shared-b-w2-797488-openshift-default-5f79c8745f-7mjt4\" (UID: \"ecdcaba8-df41-4c1b-a442-d03f9994a5e4\") " pod="openshift-ingress/e2e-shared-b-w2-797488-openshift-default-5f79c8745f-7mjt4"
Nov 02 15:45:10.687585 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:10.687564    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ecdcaba8-df41-4c1b-a442-d03f9994a5e4-istio-envoy\") pod \"e2e-shared-b-w2-797488-openshift-default-5f79c8745f-7mjt4\" (UID: \"ecdcaba8-df41-4c1b-a442-d03f9994a5e4\") " pod="openshift-ingress/e2e-shared-b-w2-797488-openshift-default-5f79c8745f-7mjt4"
Nov 02 15:45:10.687627 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:10.687606    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecdcaba8-df41-4c1b-a442-d03f9994a5e4-workload-socket\") pod \"e2e-shared-b-w2-797488-openshift-default-5f79c8745f-7mjt4\" (UID: \"ecdcaba8-df41-4c1b-a442-d03f9994a5e4\") " pod="openshift-ingress/e2e-shared-b-w2-797488-openshift-default-5f79c8745f-7mjt4"
Nov 02 15:45:10.788456 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:10.788421    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/094c52e5-8049-44dc-a80d-80b5444933ed-kube-api-access-qgjp4\" (UniqueName: \"kubernetes.io/projected/094c52e5-8049-44dc-a80d-80b5444933ed-kube-api-access-qgjp4\") pod \"094c52e5-8049-44dc-a80d-80b5444933ed\" (UID: \"094c52e5-8049-44dc-a80d-80b5444933ed\") "
Nov 02 15:45:10.788760 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:10.788736    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/094c52e5-8049-44dc-a80d-80b5444933ed-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/094c52e5-8049-44dc-a80d-80b5444933ed-maas-api-xks-ca\") pod \"094c52e5-8049-44dc-a80d-80b5444933ed\" (UID: \"094c52e5-8049-44dc-a80d-80b5444933ed\") "
Nov 02 15:45:10.788968 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:10.788948    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/094c52e5-8049-44dc-a80d-80b5444933ed-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/094c52e5-8049-44dc-a80d-80b5444933ed-maas-api-openshift-ca\") pod \"094c52e5-8049-44dc-a80d-80b5444933ed\" (UID: \"094c52e5-8049-44dc-a80d-80b5444933ed\") "
Nov 02 15:45:10.789571 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:10.789543    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ecdcaba8-df41-4c1b-a442-d03f9994a5e4-istio-data\") pod \"e2e-shared-b-w2-797488-openshift-default-5f79c8745f-7mjt4\" (UID: \"ecdcaba8-df41-4c1b-a442-d03f9994a5e4\") " pod="openshift-ingress/e2e-shared-b-w2-797488-openshift-default-5f79c8745f-7mjt4"
Nov 02 15:45:10.789772 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:10.789758    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ecdcaba8-df41-4c1b-a442-d03f9994a5e4-istio-podinfo\") pod \"e2e-shared-b-w2-797488-openshift-default-5f79c8745f-7mjt4\" (UID: \"ecdcaba8-df41-4c1b-a442-d03f9994a5e4\") " pod="openshift-ingress/e2e-shared-b-w2-797488-openshift-default-5f79c8745f-7mjt4"
Nov 02 15:45:10.789892 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:10.789883    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ecdcaba8-df41-4c1b-a442-d03f9994a5e4-istiod-ca-cert\") pod \"e2e-shared-b-w2-797488-openshift-default-5f79c8745f-7mjt4\" (UID: \"ecdcaba8-df41-4c1b-a442-d03f9994a5e4\") " pod="openshift-ingress/e2e-shared-b-w2-797488-openshift-default-5f79c8745f-7mjt4"
Nov 02 15:45:10.789960 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:10.789952    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ecdcaba8-df41-4c1b-a442-d03f9994a5e4-workload-certs\") pod \"e2e-shared-b-w2-797488-openshift-default-5f79c8745f-7mjt4\" (UID: \"ecdcaba8-df41-4c1b-a442-d03f9994a5e4\") " pod="openshift-ingress/e2e-shared-b-w2-797488-openshift-default-5f79c8745f-7mjt4"
Nov 02 15:45:10.790038 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:10.790030    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecdcaba8-df41-4c1b-a442-d03f9994a5e4-credential-socket\") pod \"e2e-shared-b-w2-797488-openshift-default-5f79c8745f-7mjt4\" (UID: \"ecdcaba8-df41-4c1b-a442-d03f9994a5e4\") " pod="openshift-ingress/e2e-shared-b-w2-797488-openshift-default-5f79c8745f-7mjt4"
Nov 02 15:45:10.790161 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:10.790147    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ecdcaba8-df41-4c1b-a442-d03f9994a5e4-istio-token\") pod \"e2e-shared-b-w2-797488-openshift-default-5f79c8745f-7mjt4\" (UID: \"ecdcaba8-df41-4c1b-a442-d03f9994a5e4\") " pod="openshift-ingress/e2e-shared-b-w2-797488-openshift-default-5f79c8745f-7mjt4"
Nov 02 15:45:10.790248 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:10.790238    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d6fqb\" (UniqueName: \"kubernetes.io/projected/ecdcaba8-df41-4c1b-a442-d03f9994a5e4-kube-api-access-d6fqb\") pod \"e2e-shared-b-w2-797488-openshift-default-5f79c8745f-7mjt4\" (UID: \"ecdcaba8-df41-4c1b-a442-d03f9994a5e4\") " pod="openshift-ingress/e2e-shared-b-w2-797488-openshift-default-5f79c8745f-7mjt4"
Nov 02 15:45:10.790347 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:10.790336    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ecdcaba8-df41-4c1b-a442-d03f9994a5e4-istio-envoy\") pod \"e2e-shared-b-w2-797488-openshift-default-5f79c8745f-7mjt4\" (UID: \"ecdcaba8-df41-4c1b-a442-d03f9994a5e4\") " pod="openshift-ingress/e2e-shared-b-w2-797488-openshift-default-5f79c8745f-7mjt4"
Nov 02 15:45:10.790408 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:10.790400    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecdcaba8-df41-4c1b-a442-d03f9994a5e4-workload-socket\") pod \"e2e-shared-b-w2-797488-openshift-default-5f79c8745f-7mjt4\" (UID: \"ecdcaba8-df41-4c1b-a442-d03f9994a5e4\") " pod="openshift-ingress/e2e-shared-b-w2-797488-openshift-default-5f79c8745f-7mjt4"
Nov 02 15:45:10.790850 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:10.790826    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecdcaba8-df41-4c1b-a442-d03f9994a5e4-workload-socket\") pod \"e2e-shared-b-w2-797488-openshift-default-5f79c8745f-7mjt4\" (UID: \"ecdcaba8-df41-4c1b-a442-d03f9994a5e4\") " pod="openshift-ingress/e2e-shared-b-w2-797488-openshift-default-5f79c8745f-7mjt4"
Nov 02 15:45:10.791246 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:10.791223    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/094c52e5-8049-44dc-a80d-80b5444933ed-maas-api-openshift-ca" pod "094c52e5-8049-44dc-a80d-80b5444933ed" (UID: "094c52e5-8049-44dc-a80d-80b5444933ed"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:45:10.791599 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:10.791583    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ecdcaba8-df41-4c1b-a442-d03f9994a5e4-istio-data\") pod \"e2e-shared-b-w2-797488-openshift-default-5f79c8745f-7mjt4\" (UID: \"ecdcaba8-df41-4c1b-a442-d03f9994a5e4\") " pod="openshift-ingress/e2e-shared-b-w2-797488-openshift-default-5f79c8745f-7mjt4"
Nov 02 15:45:10.791718 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:10.791690    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecdcaba8-df41-4c1b-a442-d03f9994a5e4-credential-socket\") pod \"e2e-shared-b-w2-797488-openshift-default-5f79c8745f-7mjt4\" (UID: \"ecdcaba8-df41-4c1b-a442-d03f9994a5e4\") " pod="openshift-ingress/e2e-shared-b-w2-797488-openshift-default-5f79c8745f-7mjt4"
Nov 02 15:45:10.792836 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:10.792785    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ecdcaba8-df41-4c1b-a442-d03f9994a5e4-workload-certs\") pod \"e2e-shared-b-w2-797488-openshift-default-5f79c8745f-7mjt4\" (UID: \"ecdcaba8-df41-4c1b-a442-d03f9994a5e4\") " pod="openshift-ingress/e2e-shared-b-w2-797488-openshift-default-5f79c8745f-7mjt4"
Nov 02 15:45:10.793193 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:10.793163    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ecdcaba8-df41-4c1b-a442-d03f9994a5e4-istiod-ca-cert\") pod \"e2e-shared-b-w2-797488-openshift-default-5f79c8745f-7mjt4\" (UID: \"ecdcaba8-df41-4c1b-a442-d03f9994a5e4\") " pod="openshift-ingress/e2e-shared-b-w2-797488-openshift-default-5f79c8745f-7mjt4"
Nov 02 15:45:10.795118 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:10.795072    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ecdcaba8-df41-4c1b-a442-d03f9994a5e4-istio-podinfo\") pod \"e2e-shared-b-w2-797488-openshift-default-5f79c8745f-7mjt4\" (UID: \"ecdcaba8-df41-4c1b-a442-d03f9994a5e4\") " pod="openshift-ingress/e2e-shared-b-w2-797488-openshift-default-5f79c8745f-7mjt4"
Nov 02 15:45:10.795741 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:10.795713    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/094c52e5-8049-44dc-a80d-80b5444933ed-kube-api-access-qgjp4" pod "094c52e5-8049-44dc-a80d-80b5444933ed" (UID: "094c52e5-8049-44dc-a80d-80b5444933ed"). InnerVolumeSpecName "kube-api-access-qgjp4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:45:10.796556 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:10.796512    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/094c52e5-8049-44dc-a80d-80b5444933ed-maas-api-xks-ca" pod "094c52e5-8049-44dc-a80d-80b5444933ed" (UID: "094c52e5-8049-44dc-a80d-80b5444933ed"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:45:10.797181 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:10.797118    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ecdcaba8-df41-4c1b-a442-d03f9994a5e4-istio-envoy\") pod \"e2e-shared-b-w2-797488-openshift-default-5f79c8745f-7mjt4\" (UID: \"ecdcaba8-df41-4c1b-a442-d03f9994a5e4\") " pod="openshift-ingress/e2e-shared-b-w2-797488-openshift-default-5f79c8745f-7mjt4"
Nov 02 15:45:10.816159 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:10.815293    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ecdcaba8-df41-4c1b-a442-d03f9994a5e4-istio-token\") pod \"e2e-shared-b-w2-797488-openshift-default-5f79c8745f-7mjt4\" (UID: \"ecdcaba8-df41-4c1b-a442-d03f9994a5e4\") " pod="openshift-ingress/e2e-shared-b-w2-797488-openshift-default-5f79c8745f-7mjt4"
Nov 02 15:45:10.816159 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:10.815490    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d6fqb\" (UniqueName: \"kubernetes.io/projected/ecdcaba8-df41-4c1b-a442-d03f9994a5e4-kube-api-access-d6fqb\") pod \"e2e-shared-b-w2-797488-openshift-default-5f79c8745f-7mjt4\" (UID: \"ecdcaba8-df41-4c1b-a442-d03f9994a5e4\") " pod="openshift-ingress/e2e-shared-b-w2-797488-openshift-default-5f79c8745f-7mjt4"
Nov 02 15:45:10.892372 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:10.892336    5304 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/094c52e5-8049-44dc-a80d-80b5444933ed-maas-api-xks-ca\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:45:10.892372 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:10.892366    5304 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/094c52e5-8049-44dc-a80d-80b5444933ed-maas-api-openshift-ca\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:45:10.892372 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:10.892377    5304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qgjp4\" (UniqueName: \"kubernetes.io/projected/094c52e5-8049-44dc-a80d-80b5444933ed-kube-api-access-qgjp4\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:45:11.055156 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:11.054321    5304 patch_prober.go:29] interesting pod/e2e-ipp-b-3805b8-openshift-default-6c9f89869-qcms9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.188:15021/healthz/ready\": dial tcp 10.133.0.188:15021: connect: connection refused" start-of-body=
Nov 02 15:45:11.055156 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:11.054382    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-qcms9" podUID="f75de3cc-f69f-4dfc-9351-9e75a08723b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.188:15021/healthz/ready\": dial tcp 10.133.0.188:15021: connect: connection refused"
Nov 02 15:45:11.100093 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:11.100063    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-797488-openshift-default-5f79c8745f-7mjt4"
Nov 02 15:45:11.462685 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:11.462650    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d30c0729-9597d"
Nov 02 15:45:11.462878 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:11.462733    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d30c0729-9597d" event={"ID":"094c52e5-8049-44dc-a80d-80b5444933ed","Type":"ContainerDied","Data":"be81d39a41061309eea9d86b4f454448bf41ea537a3b86f93363e4cb9d9b580e"}
Nov 02 15:45:11.462878 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:11.462787    5304 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="be81d39a41061309eea9d86b4f454448bf41ea537a3b86f93363e4cb9d9b580e"
Nov 02 15:45:11.464949 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:11.464915    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f84ad8-nn4jm" event={"ID":"97f24cb0-604f-4969-ba8b-90dd2373c61a","Type":"ContainerStarted","Data":"ab2550b18afc069e02c937d4a37eac19199136fef39bc5e683795aceac45c368"}
Nov 02 15:45:11.465086 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:11.464968    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f84ad8-nn4jm" event={"ID":"97f24cb0-604f-4969-ba8b-90dd2373c61a","Type":"ContainerStarted","Data":"50c33f7689a64d80305cb8a1a0bcaf56e827f1d6683962f28b8158aa8f64ff9a"}
Nov 02 15:45:11.475711 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:11.474230    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-797488-openshift-default-5f79c8745f-7mjt4"]
Nov 02 15:45:11.480401 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:11.480365    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:45:11.480510 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:11.480456    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:45:11.480587 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:11.480513    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:45:11.515671 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:11.515622    5304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f84ad8-nn4jm" podStartSLOduration=2.515607773 podStartE2EDuration="2.515607773s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:45:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:45:11.511115971 +0000 UTC m=+2911.504940380" watchObservedRunningTime="2026-11-02 15:45:11.515607773 +0000 UTC m=+2911.509432183"
Nov 02 15:45:12.075120 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:12.075086    5304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-qcms9"
Nov 02 15:45:12.075561 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:12.075530    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-qcms9"
Nov 02 15:45:12.076769 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:12.076748    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-qcms9"
Nov 02 15:45:12.637054 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:12.637023    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-797488-openshift-default-5f79c8745f-7mjt4" event={"ID":"ecdcaba8-df41-4c1b-a442-d03f9994a5e4","Type":"ContainerStarted","Data":"34672f21390ce22237ff0ab63fe954251fd27d630e81bd9d33ffad0b12ba168f"}
Nov 02 15:45:12.637568 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:12.637218    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-797488-openshift-default-5f79c8745f-7mjt4" event={"ID":"ecdcaba8-df41-4c1b-a442-d03f9994a5e4","Type":"ContainerStarted","Data":"2ed90fd35e803742244f5cc2b268269be2116a35afedc228fe7ba66f8e6fda34"}
Nov 02 15:45:12.646067 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:12.646027    5304 generic.go:356] "Generic (PLEG): container finished" podID="97f24cb0-604f-4969-ba8b-90dd2373c61a" containerID="ab2550b18afc069e02c937d4a37eac19199136fef39bc5e683795aceac45c368" exitCode=0
Nov 02 15:45:12.647173 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:12.647143    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f84ad8-nn4jm" event={"ID":"97f24cb0-604f-4969-ba8b-90dd2373c61a","Type":"ContainerDied","Data":"ab2550b18afc069e02c937d4a37eac19199136fef39bc5e683795aceac45c368"}
Nov 02 15:45:12.688758 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:12.688687    5304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-797488-openshift-default-5f79c8745f-7mjt4" podStartSLOduration=2.6886718309999997 podStartE2EDuration="2.688671831s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:45:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:45:12.687693667 +0000 UTC m=+2912.681518075" watchObservedRunningTime="2026-11-02 15:45:12.688671831 +0000 UTC m=+2912.682496240"
Nov 02 15:45:13.101072 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:13.101036    5304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-797488-openshift-default-5f79c8745f-7mjt4"
Nov 02 15:45:13.102546 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:13.102519    5304 patch_prober.go:29] interesting pod/e2e-shared-b-w2-797488-openshift-default-5f79c8745f-7mjt4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused" start-of-body=
Nov 02 15:45:13.102684 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:13.102567    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-797488-openshift-default-5f79c8745f-7mjt4" podUID="ecdcaba8-df41-4c1b-a442-d03f9994a5e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused"
Nov 02 15:45:13.875328 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:13.875300    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f84ad8-nn4jm"
Nov 02 15:45:13.970205 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:13.966964    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/97f24cb0-604f-4969-ba8b-90dd2373c61a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97f24cb0-604f-4969-ba8b-90dd2373c61a-maas-api-xks-ca\") pod \"97f24cb0-604f-4969-ba8b-90dd2373c61a\" (UID: \"97f24cb0-604f-4969-ba8b-90dd2373c61a\") "
Nov 02 15:45:14.012385 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:13.967117    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97f24cb0-604f-4969-ba8b-90dd2373c61a-kube-api-access-cc78n\" (UniqueName: \"kubernetes.io/projected/97f24cb0-604f-4969-ba8b-90dd2373c61a-kube-api-access-cc78n\") pod \"97f24cb0-604f-4969-ba8b-90dd2373c61a\" (UID: \"97f24cb0-604f-4969-ba8b-90dd2373c61a\") "
Nov 02 15:45:14.012385 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:14.009820    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/97f24cb0-604f-4969-ba8b-90dd2373c61a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97f24cb0-604f-4969-ba8b-90dd2373c61a-maas-api-openshift-ca\") pod \"97f24cb0-604f-4969-ba8b-90dd2373c61a\" (UID: \"97f24cb0-604f-4969-ba8b-90dd2373c61a\") "
Nov 02 15:45:14.012849 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:14.012815    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97f24cb0-604f-4969-ba8b-90dd2373c61a-maas-api-openshift-ca" pod "97f24cb0-604f-4969-ba8b-90dd2373c61a" (UID: "97f24cb0-604f-4969-ba8b-90dd2373c61a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:45:14.069856 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:14.069811    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97f24cb0-604f-4969-ba8b-90dd2373c61a-kube-api-access-cc78n" pod "97f24cb0-604f-4969-ba8b-90dd2373c61a" (UID: "97f24cb0-604f-4969-ba8b-90dd2373c61a"). InnerVolumeSpecName "kube-api-access-cc78n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:45:14.070790 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:14.070759    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/97f24cb0-604f-4969-ba8b-90dd2373c61a-maas-api-xks-ca" pod "97f24cb0-604f-4969-ba8b-90dd2373c61a" (UID: "97f24cb0-604f-4969-ba8b-90dd2373c61a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:45:14.100597 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:14.100564    5304 patch_prober.go:29] interesting pod/e2e-shared-b-w2-797488-openshift-default-5f79c8745f-7mjt4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused" start-of-body=
Nov 02 15:45:14.100749 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:14.100629    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-797488-openshift-default-5f79c8745f-7mjt4" podUID="ecdcaba8-df41-4c1b-a442-d03f9994a5e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused"
Nov 02 15:45:14.111813 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:14.111790    5304 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97f24cb0-604f-4969-ba8b-90dd2373c61a-maas-api-xks-ca\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:45:14.111813 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:14.111815    5304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cc78n\" (UniqueName: \"kubernetes.io/projected/97f24cb0-604f-4969-ba8b-90dd2373c61a-kube-api-access-cc78n\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:45:14.111954 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:14.111827    5304 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97f24cb0-604f-4969-ba8b-90dd2373c61a-maas-api-openshift-ca\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:45:14.668891 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:14.668856    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f84ad8-nn4jm"
Nov 02 15:45:14.669095 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:14.668884    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f84ad8-nn4jm" event={"ID":"97f24cb0-604f-4969-ba8b-90dd2373c61a","Type":"ContainerDied","Data":"50c33f7689a64d80305cb8a1a0bcaf56e827f1d6683962f28b8158aa8f64ff9a"}
Nov 02 15:45:14.669095 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:14.668944    5304 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="50c33f7689a64d80305cb8a1a0bcaf56e827f1d6683962f28b8158aa8f64ff9a"
Nov 02 15:45:15.101098 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:15.101060    5304 patch_prober.go:29] interesting pod/e2e-shared-b-w2-797488-openshift-default-5f79c8745f-7mjt4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused" start-of-body=
Nov 02 15:45:15.101558 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:15.101115    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-797488-openshift-default-5f79c8745f-7mjt4" podUID="ecdcaba8-df41-4c1b-a442-d03f9994a5e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused"
Nov 02 15:45:16.100501 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:16.100462    5304 patch_prober.go:29] interesting pod/e2e-shared-b-w2-797488-openshift-default-5f79c8745f-7mjt4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused" start-of-body=
Nov 02 15:45:16.100702 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:16.100520    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-797488-openshift-default-5f79c8745f-7mjt4" podUID="ecdcaba8-df41-4c1b-a442-d03f9994a5e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused"
Nov 02 15:45:17.100813 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:17.100775    5304 patch_prober.go:29] interesting pod/e2e-shared-b-w2-797488-openshift-default-5f79c8745f-7mjt4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused" start-of-body=
Nov 02 15:45:17.101233 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:17.100841    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-797488-openshift-default-5f79c8745f-7mjt4" podUID="ecdcaba8-df41-4c1b-a442-d03f9994a5e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused"
Nov 02 15:45:17.357944 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:17.357852    5304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-16c3c8ee-l2z4h"]
Nov 02 15:45:17.359016 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:17.358992    5304 cpu_manager.go:414] "containerMap: removing container" podUID="094c52e5-8049-44dc-a80d-80b5444933ed" containerName="revoke-keys"
Nov 02 15:45:17.359016 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:17.359016    5304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="094c52e5-8049-44dc-a80d-80b5444933ed" containerName="revoke-keys"
Nov 02 15:45:17.359212 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:17.359099    5304 cpu_manager.go:414] "containerMap: removing container" podUID="97f24cb0-604f-4969-ba8b-90dd2373c61a" containerName="revoke-keys"
Nov 02 15:45:17.359212 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:17.359108    5304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97f24cb0-604f-4969-ba8b-90dd2373c61a" containerName="revoke-keys"
Nov 02 15:45:17.359320 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:17.359306    5304 memory_manager.go:365] "RemoveStaleState removing state" podUID="97f24cb0-604f-4969-ba8b-90dd2373c61a" containerName="revoke-keys"
Nov 02 15:45:17.359362 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:17.359329    5304 memory_manager.go:365] "RemoveStaleState removing state" podUID="094c52e5-8049-44dc-a80d-80b5444933ed" containerName="revoke-keys"
Nov 02 15:45:17.363737 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:17.363716    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16c3c8ee-l2z4h"
Nov 02 15:45:17.373651 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:17.373626    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-16c3c8ee-l2z4h"]
Nov 02 15:45:17.381883 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:17.381859    5304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 15:45:17.396513 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:17.396479    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7cc94d9d-b85d-4bce-910e-471f19291c5f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-16c3c8ee-l2z4h\" (UID: \"7cc94d9d-b85d-4bce-910e-471f19291c5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16c3c8ee-l2z4h"
Nov 02 15:45:17.396661 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:17.396638    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-brgm7\" (UniqueName: \"kubernetes.io/projected/7cc94d9d-b85d-4bce-910e-471f19291c5f-kube-api-access-brgm7\") pod \"maas-api-revoke-sub-16c3c8ee-l2z4h\" (UID: \"7cc94d9d-b85d-4bce-910e-471f19291c5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16c3c8ee-l2z4h"
Nov 02 15:45:17.396733 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:17.396675    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7cc94d9d-b85d-4bce-910e-471f19291c5f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-16c3c8ee-l2z4h\" (UID: \"7cc94d9d-b85d-4bce-910e-471f19291c5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16c3c8ee-l2z4h"
Nov 02 15:45:17.497698 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:17.497666    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7cc94d9d-b85d-4bce-910e-471f19291c5f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-16c3c8ee-l2z4h\" (UID: \"7cc94d9d-b85d-4bce-910e-471f19291c5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16c3c8ee-l2z4h"
Nov 02 15:45:17.497911 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:17.497781    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-brgm7\" (UniqueName: \"kubernetes.io/projected/7cc94d9d-b85d-4bce-910e-471f19291c5f-kube-api-access-brgm7\") pod \"maas-api-revoke-sub-16c3c8ee-l2z4h\" (UID: \"7cc94d9d-b85d-4bce-910e-471f19291c5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16c3c8ee-l2z4h"
Nov 02 15:45:17.497911 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:17.497811    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7cc94d9d-b85d-4bce-910e-471f19291c5f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-16c3c8ee-l2z4h\" (UID: \"7cc94d9d-b85d-4bce-910e-471f19291c5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16c3c8ee-l2z4h"
Nov 02 15:45:17.498455 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:17.498429    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7cc94d9d-b85d-4bce-910e-471f19291c5f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-16c3c8ee-l2z4h\" (UID: \"7cc94d9d-b85d-4bce-910e-471f19291c5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16c3c8ee-l2z4h"
Nov 02 15:45:17.500242 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:17.500216    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7cc94d9d-b85d-4bce-910e-471f19291c5f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-16c3c8ee-l2z4h\" (UID: \"7cc94d9d-b85d-4bce-910e-471f19291c5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16c3c8ee-l2z4h"
Nov 02 15:45:17.521232 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:17.521209    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-brgm7\" (UniqueName: \"kubernetes.io/projected/7cc94d9d-b85d-4bce-910e-471f19291c5f-kube-api-access-brgm7\") pod \"maas-api-revoke-sub-16c3c8ee-l2z4h\" (UID: \"7cc94d9d-b85d-4bce-910e-471f19291c5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16c3c8ee-l2z4h"
Nov 02 15:45:17.681281 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:17.681200    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16c3c8ee-l2z4h"
Nov 02 15:45:18.024847 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:18.024816    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-16c3c8ee-l2z4h"]
Nov 02 15:45:18.026962 ip-10-0-135-138 kubenswrapper[5304]: W1102 15:45:18.026932    5304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7cc94d9d_b85d_4bce_910e_471f19291c5f.slice/crio-34e1cf2908b565a989ea19364a047dbe82bee2ffed97ad0ceaa14271990c9ee2 WatchSource:0}: Error finding container 34e1cf2908b565a989ea19364a047dbe82bee2ffed97ad0ceaa14271990c9ee2: Status 404 returned error can't find the container with id 34e1cf2908b565a989ea19364a047dbe82bee2ffed97ad0ceaa14271990c9ee2
Nov 02 15:45:18.100873 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:18.100836    5304 patch_prober.go:29] interesting pod/e2e-shared-b-w2-797488-openshift-default-5f79c8745f-7mjt4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused" start-of-body=
Nov 02 15:45:18.101290 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:18.100889    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-797488-openshift-default-5f79c8745f-7mjt4" podUID="ecdcaba8-df41-4c1b-a442-d03f9994a5e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused"
Nov 02 15:45:18.111593 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:18.111569    5304 scope.go:122] "RemoveContainer" containerID="fca4b94c7f454e54863a50ae64ee3ade7aa4f9daaccfd37be27481df18c72746"
Nov 02 15:45:18.111780 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:18.111668    5304 scope.go:122] "RemoveContainer" containerID="4e8879989f2a249b835f019d34f10353e22d01579f40ba2ca60807a15c74ae45"
Nov 02 15:45:18.874947 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:18.874878    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16c3c8ee-l2z4h" event={"ID":"7cc94d9d-b85d-4bce-910e-471f19291c5f","Type":"ContainerStarted","Data":"34e1cf2908b565a989ea19364a047dbe82bee2ffed97ad0ceaa14271990c9ee2"}
Nov 02 15:45:19.105958 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:19.105930    5304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-797488-openshift-default-5f79c8745f-7mjt4"
Nov 02 15:45:19.106420 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:19.106406    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-797488-openshift-default-5f79c8745f-7mjt4"
Nov 02 15:45:19.107220 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:19.107200    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-797488-openshift-default-5f79c8745f-7mjt4"
Nov 02 15:45:19.111793 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:19.111770    5304 scope.go:122] "RemoveContainer" containerID="1f8869f2c5a853c053b524b14968d0f0478b4ed819327f67881baee4bb902cd3"
Nov 02 15:45:19.247334 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:19.247295    5304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-wr624"]
Nov 02 15:45:19.254030 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:19.253999    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-wr624"
Nov 02 15:45:19.277781 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:19.275693    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-wr624"]
Nov 02 15:45:19.393339 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:19.393302    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20517696-bd32-4370-a351-67c838f8ccf6-istiod-ca-cert\") pod \"e2e-ipp-b-3805b8-openshift-default-6c9f89869-wr624\" (UID: \"20517696-bd32-4370-a351-67c838f8ccf6\") " pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-wr624"
Nov 02 15:45:19.393339 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:19.393342    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20517696-bd32-4370-a351-67c838f8ccf6-workload-certs\") pod \"e2e-ipp-b-3805b8-openshift-default-6c9f89869-wr624\" (UID: \"20517696-bd32-4370-a351-67c838f8ccf6\") " pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-wr624"
Nov 02 15:45:19.393670 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:19.393637    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20517696-bd32-4370-a351-67c838f8ccf6-credential-socket\") pod \"e2e-ipp-b-3805b8-openshift-default-6c9f89869-wr624\" (UID: \"20517696-bd32-4370-a351-67c838f8ccf6\") " pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-wr624"
Nov 02 15:45:19.394285 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:19.393689    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20517696-bd32-4370-a351-67c838f8ccf6-istio-token\") pod \"e2e-ipp-b-3805b8-openshift-default-6c9f89869-wr624\" (UID: \"20517696-bd32-4370-a351-67c838f8ccf6\") " pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-wr624"
Nov 02 15:45:19.394285 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:19.393762    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20517696-bd32-4370-a351-67c838f8ccf6-istio-podinfo\") pod \"e2e-ipp-b-3805b8-openshift-default-6c9f89869-wr624\" (UID: \"20517696-bd32-4370-a351-67c838f8ccf6\") " pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-wr624"
Nov 02 15:45:19.394285 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:19.393824    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20517696-bd32-4370-a351-67c838f8ccf6-istio-envoy\") pod \"e2e-ipp-b-3805b8-openshift-default-6c9f89869-wr624\" (UID: \"20517696-bd32-4370-a351-67c838f8ccf6\") " pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-wr624"
Nov 02 15:45:19.394285 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:19.393882    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q9p87\" (UniqueName: \"kubernetes.io/projected/20517696-bd32-4370-a351-67c838f8ccf6-kube-api-access-q9p87\") pod \"e2e-ipp-b-3805b8-openshift-default-6c9f89869-wr624\" (UID: \"20517696-bd32-4370-a351-67c838f8ccf6\") " pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-wr624"
Nov 02 15:45:19.394285 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:19.393939    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20517696-bd32-4370-a351-67c838f8ccf6-workload-socket\") pod \"e2e-ipp-b-3805b8-openshift-default-6c9f89869-wr624\" (UID: \"20517696-bd32-4370-a351-67c838f8ccf6\") " pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-wr624"
Nov 02 15:45:19.394285 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:19.393987    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20517696-bd32-4370-a351-67c838f8ccf6-istio-data\") pod \"e2e-ipp-b-3805b8-openshift-default-6c9f89869-wr624\" (UID: \"20517696-bd32-4370-a351-67c838f8ccf6\") " pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-wr624"
Nov 02 15:45:19.495552 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:19.495517    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20517696-bd32-4370-a351-67c838f8ccf6-istio-envoy\") pod \"e2e-ipp-b-3805b8-openshift-default-6c9f89869-wr624\" (UID: \"20517696-bd32-4370-a351-67c838f8ccf6\") " pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-wr624"
Nov 02 15:45:19.495728 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:19.495561    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q9p87\" (UniqueName: \"kubernetes.io/projected/20517696-bd32-4370-a351-67c838f8ccf6-kube-api-access-q9p87\") pod \"e2e-ipp-b-3805b8-openshift-default-6c9f89869-wr624\" (UID: \"20517696-bd32-4370-a351-67c838f8ccf6\") " pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-wr624"
Nov 02 15:45:19.495728 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:19.495591    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20517696-bd32-4370-a351-67c838f8ccf6-workload-socket\") pod \"e2e-ipp-b-3805b8-openshift-default-6c9f89869-wr624\" (UID: \"20517696-bd32-4370-a351-67c838f8ccf6\") " pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-wr624"
Nov 02 15:45:19.495728 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:19.495626    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20517696-bd32-4370-a351-67c838f8ccf6-istio-data\") pod \"e2e-ipp-b-3805b8-openshift-default-6c9f89869-wr624\" (UID: \"20517696-bd32-4370-a351-67c838f8ccf6\") " pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-wr624"
Nov 02 15:45:19.495728 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:19.495681    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20517696-bd32-4370-a351-67c838f8ccf6-istiod-ca-cert\") pod \"e2e-ipp-b-3805b8-openshift-default-6c9f89869-wr624\" (UID: \"20517696-bd32-4370-a351-67c838f8ccf6\") " pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-wr624"
Nov 02 15:45:19.495728 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:19.495701    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20517696-bd32-4370-a351-67c838f8ccf6-workload-certs\") pod \"e2e-ipp-b-3805b8-openshift-default-6c9f89869-wr624\" (UID: \"20517696-bd32-4370-a351-67c838f8ccf6\") " pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-wr624"
Nov 02 15:45:19.496139 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:19.495890    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20517696-bd32-4370-a351-67c838f8ccf6-credential-socket\") pod \"e2e-ipp-b-3805b8-openshift-default-6c9f89869-wr624\" (UID: \"20517696-bd32-4370-a351-67c838f8ccf6\") " pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-wr624"
Nov 02 15:45:19.496139 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:19.495934    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20517696-bd32-4370-a351-67c838f8ccf6-istio-token\") pod \"e2e-ipp-b-3805b8-openshift-default-6c9f89869-wr624\" (UID: \"20517696-bd32-4370-a351-67c838f8ccf6\") " pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-wr624"
Nov 02 15:45:19.496139 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:19.496017    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20517696-bd32-4370-a351-67c838f8ccf6-istio-podinfo\") pod \"e2e-ipp-b-3805b8-openshift-default-6c9f89869-wr624\" (UID: \"20517696-bd32-4370-a351-67c838f8ccf6\") " pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-wr624"
Nov 02 15:45:19.496139 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:19.496084    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20517696-bd32-4370-a351-67c838f8ccf6-istio-data\") pod \"e2e-ipp-b-3805b8-openshift-default-6c9f89869-wr624\" (UID: \"20517696-bd32-4370-a351-67c838f8ccf6\") " pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-wr624"
Nov 02 15:45:19.496365 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:19.496121    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20517696-bd32-4370-a351-67c838f8ccf6-workload-socket\") pod \"e2e-ipp-b-3805b8-openshift-default-6c9f89869-wr624\" (UID: \"20517696-bd32-4370-a351-67c838f8ccf6\") " pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-wr624"
Nov 02 15:45:19.496365 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:19.496200    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20517696-bd32-4370-a351-67c838f8ccf6-workload-certs\") pod \"e2e-ipp-b-3805b8-openshift-default-6c9f89869-wr624\" (UID: \"20517696-bd32-4370-a351-67c838f8ccf6\") " pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-wr624"
Nov 02 15:45:19.496365 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:19.496340    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20517696-bd32-4370-a351-67c838f8ccf6-credential-socket\") pod \"e2e-ipp-b-3805b8-openshift-default-6c9f89869-wr624\" (UID: \"20517696-bd32-4370-a351-67c838f8ccf6\") " pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-wr624"
Nov 02 15:45:19.496539 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:19.496524    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20517696-bd32-4370-a351-67c838f8ccf6-istiod-ca-cert\") pod \"e2e-ipp-b-3805b8-openshift-default-6c9f89869-wr624\" (UID: \"20517696-bd32-4370-a351-67c838f8ccf6\") " pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-wr624"
Nov 02 15:45:19.498051 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:19.498025    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20517696-bd32-4370-a351-67c838f8ccf6-istio-envoy\") pod \"e2e-ipp-b-3805b8-openshift-default-6c9f89869-wr624\" (UID: \"20517696-bd32-4370-a351-67c838f8ccf6\") " pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-wr624"
Nov 02 15:45:19.498504 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:19.498486    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20517696-bd32-4370-a351-67c838f8ccf6-istio-podinfo\") pod \"e2e-ipp-b-3805b8-openshift-default-6c9f89869-wr624\" (UID: \"20517696-bd32-4370-a351-67c838f8ccf6\") " pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-wr624"
Nov 02 15:45:19.520606 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:19.520576    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20517696-bd32-4370-a351-67c838f8ccf6-istio-token\") pod \"e2e-ipp-b-3805b8-openshift-default-6c9f89869-wr624\" (UID: \"20517696-bd32-4370-a351-67c838f8ccf6\") " pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-wr624"
Nov 02 15:45:19.521467 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:19.521440    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q9p87\" (UniqueName: \"kubernetes.io/projected/20517696-bd32-4370-a351-67c838f8ccf6-kube-api-access-q9p87\") pod \"e2e-ipp-b-3805b8-openshift-default-6c9f89869-wr624\" (UID: \"20517696-bd32-4370-a351-67c838f8ccf6\") " pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-wr624"
Nov 02 15:45:19.580723 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:19.580680    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-wr624"
Nov 02 15:45:19.885211 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:19.885180    5304 generic.go:356] "Generic (PLEG): container finished" podID="7cc94d9d-b85d-4bce-910e-471f19291c5f" containerID="82cb496a400aa30d70a7faa9a1af71be6a94237fcc1b90726ad7cc9fe7386048" exitCode=0
Nov 02 15:45:19.885414 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:19.885276    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16c3c8ee-l2z4h" event={"ID":"7cc94d9d-b85d-4bce-910e-471f19291c5f","Type":"ContainerDied","Data":"82cb496a400aa30d70a7faa9a1af71be6a94237fcc1b90726ad7cc9fe7386048"}
Nov 02 15:45:19.887576 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:19.887547    5304 generic.go:356] "Generic (PLEG): container finished" podID="1d05aa2a-2dfb-47be-9573-a8f32cf05ab7" containerID="a923e6703ef01192ea5ed1f395fede73e110b4c46cee132a7426e72d4f310b33" exitCode=60
Nov 02 15:45:19.887711 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:19.887587    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-246137-29893905-gxb7t" event={"ID":"1d05aa2a-2dfb-47be-9573-a8f32cf05ab7","Type":"ContainerDied","Data":"a923e6703ef01192ea5ed1f395fede73e110b4c46cee132a7426e72d4f310b33"}
Nov 02 15:45:19.887711 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:19.887649    5304 scope.go:122] "RemoveContainer" containerID="4e8879989f2a249b835f019d34f10353e22d01579f40ba2ca60807a15c74ae45"
Nov 02 15:45:19.888099 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:19.888081    5304 scope.go:122] "RemoveContainer" containerID="a923e6703ef01192ea5ed1f395fede73e110b4c46cee132a7426e72d4f310b33"
Nov 02 15:45:19.888507 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:45:19.888479    5304 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w5-246137-29893905-gxb7t_odh-ai-gateway-infra(1d05aa2a-2dfb-47be-9573-a8f32cf05ab7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-246137-29893905-gxb7t" podUID="1d05aa2a-2dfb-47be-9573-a8f32cf05ab7"
Nov 02 15:45:19.889856 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:19.889826    5304 generic.go:356] "Generic (PLEG): container finished" podID="e0819318-3392-48ed-a0e4-4d7961144393" containerID="1a7f6087f5c80dc007a045352d67c4838b557ba37d472ca002331282a9b6680c" exitCode=60
Nov 02 15:45:19.889856 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:19.889846    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-0a7163-29893905-rkqjw" event={"ID":"e0819318-3392-48ed-a0e4-4d7961144393","Type":"ContainerDied","Data":"1a7f6087f5c80dc007a045352d67c4838b557ba37d472ca002331282a9b6680c"}
Nov 02 15:45:19.890254 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:19.890238    5304 scope.go:122] "RemoveContainer" containerID="1a7f6087f5c80dc007a045352d67c4838b557ba37d472ca002331282a9b6680c"
Nov 02 15:45:19.890543 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:45:19.890524    5304 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w5-0a7163-29893905-rkqjw_odh-ai-gateway-infra(e0819318-3392-48ed-a0e4-4d7961144393)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-0a7163-29893905-rkqjw" podUID="e0819318-3392-48ed-a0e4-4d7961144393"
Nov 02 15:45:19.891963 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:19.891943    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-28da58-29893905-phwql" event={"ID":"ada2b1b0-e598-4d38-b995-becab65c7e82","Type":"ContainerStarted","Data":"9edb06e03bccf2aa82decb430fe4c26ebd5f3c738e641487888187cf711ba624"}
Nov 02 15:45:19.902108 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:19.902090    5304 scope.go:122] "RemoveContainer" containerID="fca4b94c7f454e54863a50ae64ee3ade7aa4f9daaccfd37be27481df18c72746"
Nov 02 15:45:19.919583 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:19.919554    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-wr624"]
Nov 02 15:45:19.921190 ip-10-0-135-138 kubenswrapper[5304]: W1102 15:45:19.921163    5304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20517696_bd32_4370_a351_67c838f8ccf6.slice/crio-83cce077490e40001fcea35a4c1f1295b5aa01c871b3f964a32ae4e0a3c92ffe WatchSource:0}: Error finding container 83cce077490e40001fcea35a4c1f1295b5aa01c871b3f964a32ae4e0a3c92ffe: Status 404 returned error can't find the container with id 83cce077490e40001fcea35a4c1f1295b5aa01c871b3f964a32ae4e0a3c92ffe
Nov 02 15:45:19.924541 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:19.924507    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:45:19.924648 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:19.924581    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:45:19.928471 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:19.925143    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:45:20.124523 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:20.111718    5304 scope.go:122] "RemoveContainer" containerID="f868467d6714e57f7e9ac86b2a4e1313ee0846e90ef5f79f944bec4a72981bed"
Nov 02 15:45:20.124523 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:20.112748    5304 scope.go:122] "RemoveContainer" containerID="a36cce492b2e27474d39fede54de27a69c83675430f04fab29552a0f2f02f7eb"
Nov 02 15:45:20.901653 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:20.901619    5304 generic.go:356] "Generic (PLEG): container finished" podID="ada2b1b0-e598-4d38-b995-becab65c7e82" containerID="9edb06e03bccf2aa82decb430fe4c26ebd5f3c738e641487888187cf711ba624" exitCode=60
Nov 02 15:45:20.901845 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:20.901688    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-28da58-29893905-phwql" event={"ID":"ada2b1b0-e598-4d38-b995-becab65c7e82","Type":"ContainerDied","Data":"9edb06e03bccf2aa82decb430fe4c26ebd5f3c738e641487888187cf711ba624"}
Nov 02 15:45:20.901845 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:20.901746    5304 scope.go:122] "RemoveContainer" containerID="1f8869f2c5a853c053b524b14968d0f0478b4ed819327f67881baee4bb902cd3"
Nov 02 15:45:20.902148 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:20.902103    5304 scope.go:122] "RemoveContainer" containerID="9edb06e03bccf2aa82decb430fe4c26ebd5f3c738e641487888187cf711ba624"
Nov 02 15:45:20.902510 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:45:20.902473    5304 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-28da58-29893905-phwql_odh-ai-gateway-infra(ada2b1b0-e598-4d38-b995-becab65c7e82)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-28da58-29893905-phwql" podUID="ada2b1b0-e598-4d38-b995-becab65c7e82"
Nov 02 15:45:20.905975 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:20.905948    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-cce791-29893905-qlp7q" event={"ID":"a07ca06a-c083-4fe9-bca4-1614646e80fb","Type":"ContainerStarted","Data":"fb25186b8cd35022b560867dcb07ac9529d935e5137dd4989fa3ab0ee96899b9"}
Nov 02 15:45:20.909784 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:20.909760    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-78d075-29893905-sznt8" event={"ID":"4cce806c-5164-485a-a790-519176d79e26","Type":"ContainerStarted","Data":"c71e085b691f83741aeed1175cc07185d03f17b7d431d4b996a0d58b134d8116"}
Nov 02 15:45:20.911782 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:20.911754    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-wr624" event={"ID":"20517696-bd32-4370-a351-67c838f8ccf6","Type":"ContainerStarted","Data":"58b6c2093d5f573e612edbf1a557dce96641aac727276c87303fb92c48097a92"}
Nov 02 15:45:20.911896 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:20.911809    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-wr624" event={"ID":"20517696-bd32-4370-a351-67c838f8ccf6","Type":"ContainerStarted","Data":"83cce077490e40001fcea35a4c1f1295b5aa01c871b3f964a32ae4e0a3c92ffe"}
Nov 02 15:45:21.001748 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:21.001714    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-0a7163-29893905-rkqjw"]
Nov 02 15:45:21.127164 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:21.126589    5304 scope.go:122] "RemoveContainer" containerID="9eb5a5fb998ead1c4afd489e63c3aa159c305084578d0db9d6b1e940bae767e3"
Nov 02 15:45:21.127164 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:21.126767    5304 scope.go:122] "RemoveContainer" containerID="b59c1235242ea6c32f4df009d662d7a874369a0584275ba8454bf5d8713704b1"
Nov 02 15:45:21.127611 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:21.127409    5304 scope.go:122] "RemoveContainer" containerID="b4c582b9e99a38343bbbcc682867a2038c2f593d79f8fa8c52ed540c573c122f"
Nov 02 15:45:21.186627 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:21.186587    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-246137-29893905-gxb7t"]
Nov 02 15:45:21.218225 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:21.217061    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-28da58-29893905-phwql"]
Nov 02 15:45:21.221905 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:21.221858    5304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-wr624" podStartSLOduration=2.221845304 podStartE2EDuration="2.221845304s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:45:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:45:21.179988496 +0000 UTC m=+2921.173812907" watchObservedRunningTime="2026-11-02 15:45:21.221845304 +0000 UTC m=+2921.215669758"
Nov 02 15:45:21.321599 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:21.321257    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16c3c8ee-l2z4h"
Nov 02 15:45:21.472707 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:21.472678    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7cc94d9d-b85d-4bce-910e-471f19291c5f-kube-api-access-brgm7\" (UniqueName: \"kubernetes.io/projected/7cc94d9d-b85d-4bce-910e-471f19291c5f-kube-api-access-brgm7\") pod \"7cc94d9d-b85d-4bce-910e-471f19291c5f\" (UID: \"7cc94d9d-b85d-4bce-910e-471f19291c5f\") "
Nov 02 15:45:21.472877 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:21.472781    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7cc94d9d-b85d-4bce-910e-471f19291c5f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7cc94d9d-b85d-4bce-910e-471f19291c5f-maas-api-xks-ca\") pod \"7cc94d9d-b85d-4bce-910e-471f19291c5f\" (UID: \"7cc94d9d-b85d-4bce-910e-471f19291c5f\") "
Nov 02 15:45:21.473013 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:21.472991    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7cc94d9d-b85d-4bce-910e-471f19291c5f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7cc94d9d-b85d-4bce-910e-471f19291c5f-maas-api-openshift-ca\") pod \"7cc94d9d-b85d-4bce-910e-471f19291c5f\" (UID: \"7cc94d9d-b85d-4bce-910e-471f19291c5f\") "
Nov 02 15:45:21.473441 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:21.473415    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7cc94d9d-b85d-4bce-910e-471f19291c5f-maas-api-openshift-ca" pod "7cc94d9d-b85d-4bce-910e-471f19291c5f" (UID: "7cc94d9d-b85d-4bce-910e-471f19291c5f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:45:21.473696 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:21.473671    5304 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7cc94d9d-b85d-4bce-910e-471f19291c5f-maas-api-openshift-ca\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:45:21.475335 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:21.475301    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7cc94d9d-b85d-4bce-910e-471f19291c5f-kube-api-access-brgm7" pod "7cc94d9d-b85d-4bce-910e-471f19291c5f" (UID: "7cc94d9d-b85d-4bce-910e-471f19291c5f"). InnerVolumeSpecName "kube-api-access-brgm7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:45:21.475948 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:21.475918    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7cc94d9d-b85d-4bce-910e-471f19291c5f-maas-api-xks-ca" pod "7cc94d9d-b85d-4bce-910e-471f19291c5f" (UID: "7cc94d9d-b85d-4bce-910e-471f19291c5f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:45:21.567643 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:21.567612    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-0a7163-29893905-rkqjw"
Nov 02 15:45:21.570224 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:21.570198    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-246137-29893905-gxb7t"
Nov 02 15:45:21.574815 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:21.574794    5304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-brgm7\" (UniqueName: \"kubernetes.io/projected/7cc94d9d-b85d-4bce-910e-471f19291c5f-kube-api-access-brgm7\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:45:21.574875 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:21.574819    5304 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7cc94d9d-b85d-4bce-910e-471f19291c5f-maas-api-xks-ca\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:45:21.581285 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:21.581261    5304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-wr624"
Nov 02 15:45:21.583367 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:21.583346    5304 patch_prober.go:29] interesting pod/e2e-ipp-b-3805b8-openshift-default-6c9f89869-wr624 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused" start-of-body=
Nov 02 15:45:21.583459 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:21.583395    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-wr624" podUID="20517696-bd32-4370-a351-67c838f8ccf6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused"
Nov 02 15:45:21.676258 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:21.676212    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1d05aa2a-2dfb-47be-9573-a8f32cf05ab7-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d05aa2a-2dfb-47be-9573-a8f32cf05ab7-maas-api-service-ca\") pod \"1d05aa2a-2dfb-47be-9573-a8f32cf05ab7\" (UID: \"1d05aa2a-2dfb-47be-9573-a8f32cf05ab7\") "
Nov 02 15:45:21.676390 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:21.676301    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e0819318-3392-48ed-a0e4-4d7961144393-kube-api-access-44zq4\" (UniqueName: \"kubernetes.io/projected/e0819318-3392-48ed-a0e4-4d7961144393-kube-api-access-44zq4\") pod \"e0819318-3392-48ed-a0e4-4d7961144393\" (UID: \"e0819318-3392-48ed-a0e4-4d7961144393\") "
Nov 02 15:45:21.677056 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:21.677025    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d05aa2a-2dfb-47be-9573-a8f32cf05ab7-kube-api-access-4tcl2\" (UniqueName: \"kubernetes.io/projected/1d05aa2a-2dfb-47be-9573-a8f32cf05ab7-kube-api-access-4tcl2\") pod \"1d05aa2a-2dfb-47be-9573-a8f32cf05ab7\" (UID: \"1d05aa2a-2dfb-47be-9573-a8f32cf05ab7\") "
Nov 02 15:45:21.677190 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:21.677147    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e0819318-3392-48ed-a0e4-4d7961144393-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0819318-3392-48ed-a0e4-4d7961144393-maas-api-service-ca\") pod \"e0819318-3392-48ed-a0e4-4d7961144393\" (UID: \"e0819318-3392-48ed-a0e4-4d7961144393\") "
Nov 02 15:45:21.677634 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:21.677602    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0819318-3392-48ed-a0e4-4d7961144393-maas-api-service-ca" pod "e0819318-3392-48ed-a0e4-4d7961144393" (UID: "e0819318-3392-48ed-a0e4-4d7961144393"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:45:21.677898 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:21.677872    5304 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0819318-3392-48ed-a0e4-4d7961144393-maas-api-service-ca\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:45:21.678098 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:21.678062    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d05aa2a-2dfb-47be-9573-a8f32cf05ab7-maas-api-service-ca" pod "1d05aa2a-2dfb-47be-9573-a8f32cf05ab7" (UID: "1d05aa2a-2dfb-47be-9573-a8f32cf05ab7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:45:21.679737 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:21.679709    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0819318-3392-48ed-a0e4-4d7961144393-kube-api-access-44zq4" pod "e0819318-3392-48ed-a0e4-4d7961144393" (UID: "e0819318-3392-48ed-a0e4-4d7961144393"). InnerVolumeSpecName "kube-api-access-44zq4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:45:21.681601 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:21.681571    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d05aa2a-2dfb-47be-9573-a8f32cf05ab7-kube-api-access-4tcl2" pod "1d05aa2a-2dfb-47be-9573-a8f32cf05ab7" (UID: "1d05aa2a-2dfb-47be-9573-a8f32cf05ab7"). InnerVolumeSpecName "kube-api-access-4tcl2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:45:21.779216 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:21.779190    5304 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d05aa2a-2dfb-47be-9573-a8f32cf05ab7-maas-api-service-ca\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:45:21.779216 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:21.779219    5304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-44zq4\" (UniqueName: \"kubernetes.io/projected/e0819318-3392-48ed-a0e4-4d7961144393-kube-api-access-44zq4\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:45:21.779454 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:21.779231    5304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4tcl2\" (UniqueName: \"kubernetes.io/projected/1d05aa2a-2dfb-47be-9573-a8f32cf05ab7-kube-api-access-4tcl2\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:45:21.921202 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:21.921161    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-6af50f-29893905-85m66" event={"ID":"8ea83c17-d3cb-4d7e-a6ef-cbd731415216","Type":"ContainerStarted","Data":"d614965c5d2ae810fd079f858e8d96d5cd42711b9e7fe0c80591edcb46562109"}
Nov 02 15:45:21.923454 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:21.923425    5304 generic.go:356] "Generic (PLEG): container finished" podID="a07ca06a-c083-4fe9-bca4-1614646e80fb" containerID="fb25186b8cd35022b560867dcb07ac9529d935e5137dd4989fa3ab0ee96899b9" exitCode=60
Nov 02 15:45:21.923577 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:21.923538    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-cce791-29893905-qlp7q" event={"ID":"a07ca06a-c083-4fe9-bca4-1614646e80fb","Type":"ContainerDied","Data":"fb25186b8cd35022b560867dcb07ac9529d935e5137dd4989fa3ab0ee96899b9"}
Nov 02 15:45:21.923637 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:21.923587    5304 scope.go:122] "RemoveContainer" containerID="a36cce492b2e27474d39fede54de27a69c83675430f04fab29552a0f2f02f7eb"
Nov 02 15:45:21.924006 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:21.923989    5304 scope.go:122] "RemoveContainer" containerID="fb25186b8cd35022b560867dcb07ac9529d935e5137dd4989fa3ab0ee96899b9"
Nov 02 15:45:21.924465 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:45:21.924420    5304 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-cce791-29893905-qlp7q_odh-ai-gateway-infra(a07ca06a-c083-4fe9-bca4-1614646e80fb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-cce791-29893905-qlp7q" podUID="a07ca06a-c083-4fe9-bca4-1614646e80fb"
Nov 02 15:45:21.926294 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:21.926269    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893905-4d75m" event={"ID":"9846a62a-c4aa-4937-b1dd-2d9a342dd72a","Type":"ContainerStarted","Data":"b1e0a6f1ff2ca4c43df2b8590c33d028079b47b11f8a8b8503ca1e6c12aa0089"}
Nov 02 15:45:21.928244 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:21.927924    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-0a7163-29893905-rkqjw"
Nov 02 15:45:21.928244 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:21.928167    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-0a7163-29893905-rkqjw" event={"ID":"e0819318-3392-48ed-a0e4-4d7961144393","Type":"ContainerDied","Data":"4e57cd5998bbcf2e7d4b654affff6de6e4afbdb852f62f5affa11b8f2477831f"}
Nov 02 15:45:21.958209 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:21.945878    5304 generic.go:356] "Generic (PLEG): container finished" podID="4cce806c-5164-485a-a790-519176d79e26" containerID="c71e085b691f83741aeed1175cc07185d03f17b7d431d4b996a0d58b134d8116" exitCode=60
Nov 02 15:45:21.958209 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:21.946081    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-78d075-29893905-sznt8" event={"ID":"4cce806c-5164-485a-a790-519176d79e26","Type":"ContainerDied","Data":"c71e085b691f83741aeed1175cc07185d03f17b7d431d4b996a0d58b134d8116"}
Nov 02 15:45:21.961519 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:21.961487    5304 scope.go:122] "RemoveContainer" containerID="c71e085b691f83741aeed1175cc07185d03f17b7d431d4b996a0d58b134d8116"
Nov 02 15:45:21.961921 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:45:21.961894    5304 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-78d075-29893905-sznt8_odh-ai-gateway-infra(4cce806c-5164-485a-a790-519176d79e26)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-78d075-29893905-sznt8" podUID="4cce806c-5164-485a-a790-519176d79e26"
Nov 02 15:45:21.963047 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:21.963015    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-b55da1-29893905-mv5qt" event={"ID":"647ad5be-64be-4aff-bff6-d0ac2943671c","Type":"ContainerStarted","Data":"08bfdebaae5169fd787988c5d21192d32984c434ff7a02274371a34179334fdc"}
Nov 02 15:45:21.971752 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:21.970350    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16c3c8ee-l2z4h"
Nov 02 15:45:21.971752 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:21.970473    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16c3c8ee-l2z4h" event={"ID":"7cc94d9d-b85d-4bce-910e-471f19291c5f","Type":"ContainerDied","Data":"34e1cf2908b565a989ea19364a047dbe82bee2ffed97ad0ceaa14271990c9ee2"}
Nov 02 15:45:21.971752 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:21.970532    5304 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="34e1cf2908b565a989ea19364a047dbe82bee2ffed97ad0ceaa14271990c9ee2"
Nov 02 15:45:21.972984 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:21.972964    5304 scope.go:122] "RemoveContainer" containerID="1a7f6087f5c80dc007a045352d67c4838b557ba37d472ca002331282a9b6680c"
Nov 02 15:45:21.974730 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:21.973946    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-246137-29893905-gxb7t" event={"ID":"1d05aa2a-2dfb-47be-9573-a8f32cf05ab7","Type":"ContainerDied","Data":"2683141ec66c16fe4f38385b80727eb37e9455f8c09e7d18725bbfffe511b1a0"}
Nov 02 15:45:21.974730 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:21.974090    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-246137-29893905-gxb7t"
Nov 02 15:45:21.988428 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:21.988245    5304 scope.go:122] "RemoveContainer" containerID="f868467d6714e57f7e9ac86b2a4e1313ee0846e90ef5f79f944bec4a72981bed"
Nov 02 15:45:22.010643 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:22.010617    5304 scope.go:122] "RemoveContainer" containerID="a923e6703ef01192ea5ed1f395fede73e110b4c46cee132a7426e72d4f310b33"
Nov 02 15:45:22.016120 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:22.016057    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-cce791-29893905-qlp7q"]
Nov 02 15:45:22.073248 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:22.073213    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-0a7163-29893905-rkqjw"]
Nov 02 15:45:22.085169 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:22.085098    5304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-0a7163-29893905-rkqjw"]
Nov 02 15:45:22.089217 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:22.089192    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-78d075-29893905-sznt8"]
Nov 02 15:45:22.123158 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:22.123104    5304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e0819318-3392-48ed-a0e4-4d7961144393" path="/var/lib/kubelet/pods/e0819318-3392-48ed-a0e4-4d7961144393/volumes"
Nov 02 15:45:22.124178 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:22.124154    5304 scope.go:122] "RemoveContainer" containerID="c9a1497d61cf44d6c2fc56cad2b62203dacb573aec8a6b2389b1a14a5bb5a4cf"
Nov 02 15:45:22.229560 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:22.229527    5304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3bbd4709-nbnqr"]
Nov 02 15:45:22.234666 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:22.234629    5304 cpu_manager.go:414] "containerMap: removing container" podUID="e0819318-3392-48ed-a0e4-4d7961144393" containerName="cleanup"
Nov 02 15:45:22.234666 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:22.234663    5304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e0819318-3392-48ed-a0e4-4d7961144393" containerName="cleanup"
Nov 02 15:45:22.234958 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:22.234702    5304 cpu_manager.go:414] "containerMap: removing container" podUID="7cc94d9d-b85d-4bce-910e-471f19291c5f" containerName="revoke-keys"
Nov 02 15:45:22.234958 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:22.234710    5304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7cc94d9d-b85d-4bce-910e-471f19291c5f" containerName="revoke-keys"
Nov 02 15:45:22.234958 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:22.234749    5304 cpu_manager.go:414] "containerMap: removing container" podUID="1d05aa2a-2dfb-47be-9573-a8f32cf05ab7" containerName="cleanup"
Nov 02 15:45:22.234958 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:22.234757    5304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d05aa2a-2dfb-47be-9573-a8f32cf05ab7" containerName="cleanup"
Nov 02 15:45:22.234958 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:22.234771    5304 cpu_manager.go:414] "containerMap: removing container" podUID="e0819318-3392-48ed-a0e4-4d7961144393" containerName="cleanup"
Nov 02 15:45:22.234958 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:22.234778    5304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e0819318-3392-48ed-a0e4-4d7961144393" containerName="cleanup"
Nov 02 15:45:22.234958 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:22.234792    5304 cpu_manager.go:414] "containerMap: removing container" podUID="1d05aa2a-2dfb-47be-9573-a8f32cf05ab7" containerName="cleanup"
Nov 02 15:45:22.234958 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:22.234799    5304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d05aa2a-2dfb-47be-9573-a8f32cf05ab7" containerName="cleanup"
Nov 02 15:45:22.234958 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:22.234826    5304 cpu_manager.go:414] "containerMap: removing container" podUID="e0819318-3392-48ed-a0e4-4d7961144393" containerName="cleanup"
Nov 02 15:45:22.234958 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:22.234833    5304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e0819318-3392-48ed-a0e4-4d7961144393" containerName="cleanup"
Nov 02 15:45:22.235564 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:22.235024    5304 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d05aa2a-2dfb-47be-9573-a8f32cf05ab7" containerName="cleanup"
Nov 02 15:45:22.235564 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:22.235039    5304 memory_manager.go:365] "RemoveStaleState removing state" podUID="e0819318-3392-48ed-a0e4-4d7961144393" containerName="cleanup"
Nov 02 15:45:22.235564 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:22.235051    5304 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d05aa2a-2dfb-47be-9573-a8f32cf05ab7" containerName="cleanup"
Nov 02 15:45:22.235564 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:22.235066    5304 memory_manager.go:365] "RemoveStaleState removing state" podUID="e0819318-3392-48ed-a0e4-4d7961144393" containerName="cleanup"
Nov 02 15:45:22.235564 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:22.235084    5304 memory_manager.go:365] "RemoveStaleState removing state" podUID="7cc94d9d-b85d-4bce-910e-471f19291c5f" containerName="revoke-keys"
Nov 02 15:45:22.265878 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:22.265844    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bbd4709-nbnqr"
Nov 02 15:45:22.271591 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:22.271503    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3bbd4709-nbnqr"]
Nov 02 15:45:22.276585 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:22.276452    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-28da58-29893905-phwql"
Nov 02 15:45:22.285672 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:22.285644    5304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 15:45:22.290163 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:22.290104    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96b6b761-e07b-41be-8460-f0b68cd55d4a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3bbd4709-nbnqr\" (UID: \"96b6b761-e07b-41be-8460-f0b68cd55d4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bbd4709-nbnqr"
Nov 02 15:45:22.290298 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:22.290262    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96b6b761-e07b-41be-8460-f0b68cd55d4a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3bbd4709-nbnqr\" (UID: \"96b6b761-e07b-41be-8460-f0b68cd55d4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bbd4709-nbnqr"
Nov 02 15:45:22.290378 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:22.290354    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-99tmq\" (UniqueName: \"kubernetes.io/projected/96b6b761-e07b-41be-8460-f0b68cd55d4a-kube-api-access-99tmq\") pod \"maas-api-revoke-sub-3bbd4709-nbnqr\" (UID: \"96b6b761-e07b-41be-8460-f0b68cd55d4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bbd4709-nbnqr"
Nov 02 15:45:22.391135 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:22.391093    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ada2b1b0-e598-4d38-b995-becab65c7e82-kube-api-access-wvx9t\" (UniqueName: \"kubernetes.io/projected/ada2b1b0-e598-4d38-b995-becab65c7e82-kube-api-access-wvx9t\") pod \"ada2b1b0-e598-4d38-b995-becab65c7e82\" (UID: \"ada2b1b0-e598-4d38-b995-becab65c7e82\") "
Nov 02 15:45:22.391330 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:22.391274    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ada2b1b0-e598-4d38-b995-becab65c7e82-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ada2b1b0-e598-4d38-b995-becab65c7e82-maas-api-service-ca\") pod \"ada2b1b0-e598-4d38-b995-becab65c7e82\" (UID: \"ada2b1b0-e598-4d38-b995-becab65c7e82\") "
Nov 02 15:45:22.391576 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:22.391559    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96b6b761-e07b-41be-8460-f0b68cd55d4a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3bbd4709-nbnqr\" (UID: \"96b6b761-e07b-41be-8460-f0b68cd55d4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bbd4709-nbnqr"
Nov 02 15:45:22.391648 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:22.391600    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ada2b1b0-e598-4d38-b995-becab65c7e82-maas-api-service-ca" pod "ada2b1b0-e598-4d38-b995-becab65c7e82" (UID: "ada2b1b0-e598-4d38-b995-becab65c7e82"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:45:22.391707 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:22.391658    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-99tmq\" (UniqueName: \"kubernetes.io/projected/96b6b761-e07b-41be-8460-f0b68cd55d4a-kube-api-access-99tmq\") pod \"maas-api-revoke-sub-3bbd4709-nbnqr\" (UID: \"96b6b761-e07b-41be-8460-f0b68cd55d4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bbd4709-nbnqr"
Nov 02 15:45:22.391829 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:22.391816    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96b6b761-e07b-41be-8460-f0b68cd55d4a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3bbd4709-nbnqr\" (UID: \"96b6b761-e07b-41be-8460-f0b68cd55d4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bbd4709-nbnqr"
Nov 02 15:45:22.391900 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:22.391890    5304 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ada2b1b0-e598-4d38-b995-becab65c7e82-maas-api-service-ca\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:45:22.392276 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:22.392257    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96b6b761-e07b-41be-8460-f0b68cd55d4a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3bbd4709-nbnqr\" (UID: \"96b6b761-e07b-41be-8460-f0b68cd55d4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bbd4709-nbnqr"
Nov 02 15:45:22.393257 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:22.393239    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ada2b1b0-e598-4d38-b995-becab65c7e82-kube-api-access-wvx9t" pod "ada2b1b0-e598-4d38-b995-becab65c7e82" (UID: "ada2b1b0-e598-4d38-b995-becab65c7e82"). InnerVolumeSpecName "kube-api-access-wvx9t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:45:22.393970 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:22.393953    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96b6b761-e07b-41be-8460-f0b68cd55d4a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3bbd4709-nbnqr\" (UID: \"96b6b761-e07b-41be-8460-f0b68cd55d4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bbd4709-nbnqr"
Nov 02 15:45:22.417197 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:22.417167    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-246137-29893905-gxb7t"]
Nov 02 15:45:22.419773 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:22.419743    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-99tmq\" (UniqueName: \"kubernetes.io/projected/96b6b761-e07b-41be-8460-f0b68cd55d4a-kube-api-access-99tmq\") pod \"maas-api-revoke-sub-3bbd4709-nbnqr\" (UID: \"96b6b761-e07b-41be-8460-f0b68cd55d4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bbd4709-nbnqr"
Nov 02 15:45:22.426690 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:22.426665    5304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-246137-29893905-gxb7t"]
Nov 02 15:45:22.493738 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:22.493704    5304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wvx9t\" (UniqueName: \"kubernetes.io/projected/ada2b1b0-e598-4d38-b995-becab65c7e82-kube-api-access-wvx9t\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:45:22.581514 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:22.581477    5304 patch_prober.go:29] interesting pod/e2e-ipp-b-3805b8-openshift-default-6c9f89869-wr624 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused" start-of-body=
Nov 02 15:45:22.581679 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:22.581536    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-wr624" podUID="20517696-bd32-4370-a351-67c838f8ccf6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused"
Nov 02 15:45:22.590764 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:22.590733    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bbd4709-nbnqr"
Nov 02 15:45:23.027356 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:23.027319    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3bbd4709-nbnqr"]
Nov 02 15:45:23.050955 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:23.050916    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-6af50f-29893905-85m66"]
Nov 02 15:45:23.065851 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:23.065818    5304 generic.go:356] "Generic (PLEG): container finished" podID="8ea83c17-d3cb-4d7e-a6ef-cbd731415216" containerID="d614965c5d2ae810fd079f858e8d96d5cd42711b9e7fe0c80591edcb46562109" exitCode=60
Nov 02 15:45:23.065957 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:23.065929    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-6af50f-29893905-85m66" event={"ID":"8ea83c17-d3cb-4d7e-a6ef-cbd731415216","Type":"ContainerDied","Data":"d614965c5d2ae810fd079f858e8d96d5cd42711b9e7fe0c80591edcb46562109"}
Nov 02 15:45:23.066013 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:23.066000    5304 scope.go:122] "RemoveContainer" containerID="9eb5a5fb998ead1c4afd489e63c3aa159c305084578d0db9d6b1e940bae767e3"
Nov 02 15:45:23.076266 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:23.076241    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5113c2-29893905-bnvks" event={"ID":"5efd887c-66f3-445e-9e39-0d23e43cc9be","Type":"ContainerStarted","Data":"01a29ee5a8a29880062f14bf9d5237f19e808147c0448414092c8d2bce9799af"}
Nov 02 15:45:23.079476 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:23.079445    5304 generic.go:356] "Generic (PLEG): container finished" podID="9846a62a-c4aa-4937-b1dd-2d9a342dd72a" containerID="b1e0a6f1ff2ca4c43df2b8590c33d028079b47b11f8a8b8503ca1e6c12aa0089" exitCode=60
Nov 02 15:45:23.079603 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:23.079538    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893905-4d75m" event={"ID":"9846a62a-c4aa-4937-b1dd-2d9a342dd72a","Type":"ContainerDied","Data":"b1e0a6f1ff2ca4c43df2b8590c33d028079b47b11f8a8b8503ca1e6c12aa0089"}
Nov 02 15:45:23.079975 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:23.079951    5304 scope.go:122] "RemoveContainer" containerID="b1e0a6f1ff2ca4c43df2b8590c33d028079b47b11f8a8b8503ca1e6c12aa0089"
Nov 02 15:45:23.080379 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:45:23.080353    5304 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29893905-4d75m_odh-ai-gateway-infra(9846a62a-c4aa-4937-b1dd-2d9a342dd72a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893905-4d75m" podUID="9846a62a-c4aa-4937-b1dd-2d9a342dd72a"
Nov 02 15:45:23.102595 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:23.090567    5304 generic.go:356] "Generic (PLEG): container finished" podID="647ad5be-64be-4aff-bff6-d0ac2943671c" containerID="08bfdebaae5169fd787988c5d21192d32984c434ff7a02274371a34179334fdc" exitCode=60
Nov 02 15:45:23.102595 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:23.090637    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-b55da1-29893905-mv5qt" event={"ID":"647ad5be-64be-4aff-bff6-d0ac2943671c","Type":"ContainerDied","Data":"08bfdebaae5169fd787988c5d21192d32984c434ff7a02274371a34179334fdc"}
Nov 02 15:45:23.102595 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:23.090993    5304 scope.go:122] "RemoveContainer" containerID="08bfdebaae5169fd787988c5d21192d32984c434ff7a02274371a34179334fdc"
Nov 02 15:45:23.102595 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:45:23.091323    5304 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w6-b55da1-29893905-mv5qt_odh-ai-gateway-infra(647ad5be-64be-4aff-bff6-d0ac2943671c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-b55da1-29893905-mv5qt" podUID="647ad5be-64be-4aff-bff6-d0ac2943671c"
Nov 02 15:45:23.102595 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:23.093051    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-28da58-29893905-phwql"
Nov 02 15:45:23.102595 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:23.093181    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-28da58-29893905-phwql" event={"ID":"ada2b1b0-e598-4d38-b995-becab65c7e82","Type":"ContainerDied","Data":"751a01d9262602427303d4c17f8a024ff86d87c92651f80d971d91d171da9599"}
Nov 02 15:45:23.112714 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:23.112677    5304 scope.go:122] "RemoveContainer" containerID="1bd49a527b9b72cc455bd79351cd7c9e8119b3a457e0f6f430d422c85a007b95"
Nov 02 15:45:23.179264 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:23.178751    5304 scope.go:122] "RemoveContainer" containerID="b4c582b9e99a38343bbbcc682867a2038c2f593d79f8fa8c52ed540c573c122f"
Nov 02 15:45:23.181413 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:45:23.179924    5304 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podada2b1b0_e598_4d38_b995_becab65c7e82.slice\": RecentStats: unable to find data in memory cache]"
Nov 02 15:45:23.214854 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:23.214806    5304 scope.go:122] "RemoveContainer" containerID="b59c1235242ea6c32f4df009d662d7a874369a0584275ba8454bf5d8713704b1"
Nov 02 15:45:23.265201 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:23.265110    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-b55da1-29893905-mv5qt"]
Nov 02 15:45:23.273200 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:23.273095    5304 scope.go:122] "RemoveContainer" containerID="9edb06e03bccf2aa82decb430fe4c26ebd5f3c738e641487888187cf711ba624"
Nov 02 15:45:23.326271 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:23.326236    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893905-4d75m"]
Nov 02 15:45:23.465892 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:23.465861    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-28da58-29893905-phwql"]
Nov 02 15:45:23.472993 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:23.472960    5304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-28da58-29893905-phwql"]
Nov 02 15:45:23.529485 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:23.529416    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-cce791-29893905-qlp7q"
Nov 02 15:45:23.583195 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:23.581111    5304 patch_prober.go:29] interesting pod/e2e-ipp-b-3805b8-openshift-default-6c9f89869-wr624 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused" start-of-body=
Nov 02 15:45:23.583343 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:23.583237    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-wr624" podUID="20517696-bd32-4370-a351-67c838f8ccf6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused"
Nov 02 15:45:23.583412 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:23.583396    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-78d075-29893905-sznt8"
Nov 02 15:45:23.588787 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:23.588751    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-6af50f-29893905-85m66"
Nov 02 15:45:23.672893 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:23.672833    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4cce806c-5164-485a-a790-519176d79e26-kube-api-access-kk24l\" (UniqueName: \"kubernetes.io/projected/4cce806c-5164-485a-a790-519176d79e26-kube-api-access-kk24l\") pod \"4cce806c-5164-485a-a790-519176d79e26\" (UID: \"4cce806c-5164-485a-a790-519176d79e26\") "
Nov 02 15:45:23.672893 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:23.672884    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8ea83c17-d3cb-4d7e-a6ef-cbd731415216-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ea83c17-d3cb-4d7e-a6ef-cbd731415216-maas-api-service-ca\") pod \"8ea83c17-d3cb-4d7e-a6ef-cbd731415216\" (UID: \"8ea83c17-d3cb-4d7e-a6ef-cbd731415216\") "
Nov 02 15:45:23.673211 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:23.672993    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a07ca06a-c083-4fe9-bca4-1614646e80fb-kube-api-access-dhz7b\" (UniqueName: \"kubernetes.io/projected/a07ca06a-c083-4fe9-bca4-1614646e80fb-kube-api-access-dhz7b\") pod \"a07ca06a-c083-4fe9-bca4-1614646e80fb\" (UID: \"a07ca06a-c083-4fe9-bca4-1614646e80fb\") "
Nov 02 15:45:23.673211 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:23.673021    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4cce806c-5164-485a-a790-519176d79e26-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4cce806c-5164-485a-a790-519176d79e26-maas-api-service-ca\") pod \"4cce806c-5164-485a-a790-519176d79e26\" (UID: \"4cce806c-5164-485a-a790-519176d79e26\") "
Nov 02 15:45:23.673335 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:23.673261    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a07ca06a-c083-4fe9-bca4-1614646e80fb-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a07ca06a-c083-4fe9-bca4-1614646e80fb-maas-api-service-ca\") pod \"a07ca06a-c083-4fe9-bca4-1614646e80fb\" (UID: \"a07ca06a-c083-4fe9-bca4-1614646e80fb\") "
Nov 02 15:45:23.673335 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:23.673295    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ea83c17-d3cb-4d7e-a6ef-cbd731415216-maas-api-service-ca" pod "8ea83c17-d3cb-4d7e-a6ef-cbd731415216" (UID: "8ea83c17-d3cb-4d7e-a6ef-cbd731415216"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:45:23.673451 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:23.673368    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ea83c17-d3cb-4d7e-a6ef-cbd731415216-kube-api-access-7zqnq\" (UniqueName: \"kubernetes.io/projected/8ea83c17-d3cb-4d7e-a6ef-cbd731415216-kube-api-access-7zqnq\") pod \"8ea83c17-d3cb-4d7e-a6ef-cbd731415216\" (UID: \"8ea83c17-d3cb-4d7e-a6ef-cbd731415216\") "
Nov 02 15:45:23.674591 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:23.673928    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a07ca06a-c083-4fe9-bca4-1614646e80fb-maas-api-service-ca" pod "a07ca06a-c083-4fe9-bca4-1614646e80fb" (UID: "a07ca06a-c083-4fe9-bca4-1614646e80fb"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:45:23.674591 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:23.674052    5304 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ea83c17-d3cb-4d7e-a6ef-cbd731415216-maas-api-service-ca\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:45:23.674591 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:23.674068    5304 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a07ca06a-c083-4fe9-bca4-1614646e80fb-maas-api-service-ca\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:45:23.674888 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:23.674863    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4cce806c-5164-485a-a790-519176d79e26-maas-api-service-ca" pod "4cce806c-5164-485a-a790-519176d79e26" (UID: "4cce806c-5164-485a-a790-519176d79e26"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:45:23.675891 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:23.675864    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a07ca06a-c083-4fe9-bca4-1614646e80fb-kube-api-access-dhz7b" pod "a07ca06a-c083-4fe9-bca4-1614646e80fb" (UID: "a07ca06a-c083-4fe9-bca4-1614646e80fb"). InnerVolumeSpecName "kube-api-access-dhz7b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:45:23.676065 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:23.676046    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4cce806c-5164-485a-a790-519176d79e26-kube-api-access-kk24l" pod "4cce806c-5164-485a-a790-519176d79e26" (UID: "4cce806c-5164-485a-a790-519176d79e26"). InnerVolumeSpecName "kube-api-access-kk24l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:45:23.676918 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:23.676898    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ea83c17-d3cb-4d7e-a6ef-cbd731415216-kube-api-access-7zqnq" pod "8ea83c17-d3cb-4d7e-a6ef-cbd731415216" (UID: "8ea83c17-d3cb-4d7e-a6ef-cbd731415216"). InnerVolumeSpecName "kube-api-access-7zqnq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:45:23.777658 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:23.777627    5304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7zqnq\" (UniqueName: \"kubernetes.io/projected/8ea83c17-d3cb-4d7e-a6ef-cbd731415216-kube-api-access-7zqnq\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:45:23.778258 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:23.778221    5304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kk24l\" (UniqueName: \"kubernetes.io/projected/4cce806c-5164-485a-a790-519176d79e26-kube-api-access-kk24l\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:45:23.778582 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:23.778563    5304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dhz7b\" (UniqueName: \"kubernetes.io/projected/a07ca06a-c083-4fe9-bca4-1614646e80fb-kube-api-access-dhz7b\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:45:23.778824 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:23.778807    5304 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4cce806c-5164-485a-a790-519176d79e26-maas-api-service-ca\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:45:24.106221 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:24.106178    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-6af50f-29893905-85m66" event={"ID":"8ea83c17-d3cb-4d7e-a6ef-cbd731415216","Type":"ContainerDied","Data":"c935cd9ad7b009a67e300ef4bd45a2303efdb10ae4e679225e8ab2aa12b65ca0"}
Nov 02 15:45:24.106410 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:24.106246    5304 scope.go:122] "RemoveContainer" containerID="d614965c5d2ae810fd079f858e8d96d5cd42711b9e7fe0c80591edcb46562109"
Nov 02 15:45:24.106410 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:24.106352    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-6af50f-29893905-85m66"
Nov 02 15:45:24.111524 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:24.110251    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-cce791-29893905-qlp7q"
Nov 02 15:45:24.111524 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:24.110270    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-cce791-29893905-qlp7q" event={"ID":"a07ca06a-c083-4fe9-bca4-1614646e80fb","Type":"ContainerDied","Data":"72f3472a30af011638542fa53cbd87e666e149cccab824824c5ae1a114688158"}
Nov 02 15:45:24.122549 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:24.121685    5304 generic.go:356] "Generic (PLEG): container finished" podID="5efd887c-66f3-445e-9e39-0d23e43cc9be" containerID="01a29ee5a8a29880062f14bf9d5237f19e808147c0448414092c8d2bce9799af" exitCode=60
Nov 02 15:45:24.129613 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:24.129367    5304 scope.go:122] "RemoveContainer" containerID="fb25186b8cd35022b560867dcb07ac9529d935e5137dd4989fa3ab0ee96899b9"
Nov 02 15:45:24.135212 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:24.130597    5304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1d05aa2a-2dfb-47be-9573-a8f32cf05ab7" path="/var/lib/kubelet/pods/1d05aa2a-2dfb-47be-9573-a8f32cf05ab7/volumes"
Nov 02 15:45:24.146637 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:24.146609    5304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ada2b1b0-e598-4d38-b995-becab65c7e82" path="/var/lib/kubelet/pods/ada2b1b0-e598-4d38-b995-becab65c7e82/volumes"
Nov 02 15:45:24.154860 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:24.154828    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5113c2-29893905-bnvks" event={"ID":"5efd887c-66f3-445e-9e39-0d23e43cc9be","Type":"ContainerDied","Data":"01a29ee5a8a29880062f14bf9d5237f19e808147c0448414092c8d2bce9799af"}
Nov 02 15:45:24.158554 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:24.156660    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-1cc1b2-29893905-whfr9" event={"ID":"ea2c4ec8-2ace-49dd-a172-99ced21dbc4f","Type":"ContainerStarted","Data":"21987302651eeabb9267207030c7a2aded621a4f3a2c46e57ed7c45e2c00dc73"}
Nov 02 15:45:24.158554 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:24.156717    5304 scope.go:122] "RemoveContainer" containerID="01a29ee5a8a29880062f14bf9d5237f19e808147c0448414092c8d2bce9799af"
Nov 02 15:45:24.158554 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:45:24.157435    5304 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-5113c2-29893905-bnvks_odh-ai-gateway-infra(5efd887c-66f3-445e-9e39-0d23e43cc9be)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5113c2-29893905-bnvks" podUID="5efd887c-66f3-445e-9e39-0d23e43cc9be"
Nov 02 15:45:24.160169 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:24.159536    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-78d075-29893905-sznt8" event={"ID":"4cce806c-5164-485a-a790-519176d79e26","Type":"ContainerDied","Data":"84070c1e85cf71d95dc7ee9181430d56af573025097d6eaaa6757d8bd11f5606"}
Nov 02 15:45:24.160169 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:24.159626    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-78d075-29893905-sznt8"
Nov 02 15:45:24.164880 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:24.164853    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bbd4709-nbnqr" event={"ID":"96b6b761-e07b-41be-8460-f0b68cd55d4a","Type":"ContainerStarted","Data":"fccc01c9ac53fde16b503bec6219393089d9a23485208ba252d89e3a508b275d"}
Nov 02 15:45:24.165048 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:24.164900    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bbd4709-nbnqr" event={"ID":"96b6b761-e07b-41be-8460-f0b68cd55d4a","Type":"ContainerStarted","Data":"c55d534d729a465a8673b3653376eb526e7240c225071fd91670ba976cecc8b5"}
Nov 02 15:45:24.177626 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:24.177322    5304 scope.go:122] "RemoveContainer" containerID="c9a1497d61cf44d6c2fc56cad2b62203dacb573aec8a6b2389b1a14a5bb5a4cf"
Nov 02 15:45:24.207281 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:24.206713    5304 scope.go:122] "RemoveContainer" containerID="c71e085b691f83741aeed1175cc07185d03f17b7d431d4b996a0d58b134d8116"
Nov 02 15:45:24.207281 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:24.206770    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-6af50f-29893905-85m66"]
Nov 02 15:45:24.210138 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:24.210095    5304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-6af50f-29893905-85m66"]
Nov 02 15:45:24.217469 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:24.217425    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5113c2-29893905-bnvks"]
Nov 02 15:45:24.306177 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:24.306139    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-cce791-29893905-qlp7q"]
Nov 02 15:45:24.315269 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:24.315243    5304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-cce791-29893905-qlp7q"]
Nov 02 15:45:24.457349 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:24.446317    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893905-4d75m"
Nov 02 15:45:24.471250 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:24.471196    5304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bbd4709-nbnqr" podStartSLOduration=2.471181856 podStartE2EDuration="2.471181856s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:45:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:45:24.470343714 +0000 UTC m=+2924.464168122" watchObservedRunningTime="2026-11-02 15:45:24.471181856 +0000 UTC m=+2924.465006265"
Nov 02 15:45:24.496701 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:24.496676    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-b55da1-29893905-mv5qt"
Nov 02 15:45:24.513804 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:24.513770    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-78d075-29893905-sznt8"]
Nov 02 15:45:24.518852 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:24.518824    5304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-78d075-29893905-sznt8"]
Nov 02 15:45:24.581286 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:24.581251    5304 patch_prober.go:29] interesting pod/e2e-ipp-b-3805b8-openshift-default-6c9f89869-wr624 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused" start-of-body=
Nov 02 15:45:24.581461 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:24.581308    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-wr624" podUID="20517696-bd32-4370-a351-67c838f8ccf6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused"
Nov 02 15:45:24.604732 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:24.604702    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/647ad5be-64be-4aff-bff6-d0ac2943671c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/647ad5be-64be-4aff-bff6-d0ac2943671c-maas-api-service-ca\") pod \"647ad5be-64be-4aff-bff6-d0ac2943671c\" (UID: \"647ad5be-64be-4aff-bff6-d0ac2943671c\") "
Nov 02 15:45:24.604919 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:24.604820    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9846a62a-c4aa-4937-b1dd-2d9a342dd72a-kube-api-access-9dsnf\" (UniqueName: \"kubernetes.io/projected/9846a62a-c4aa-4937-b1dd-2d9a342dd72a-kube-api-access-9dsnf\") pod \"9846a62a-c4aa-4937-b1dd-2d9a342dd72a\" (UID: \"9846a62a-c4aa-4937-b1dd-2d9a342dd72a\") "
Nov 02 15:45:24.604980 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:24.604962    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9846a62a-c4aa-4937-b1dd-2d9a342dd72a-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9846a62a-c4aa-4937-b1dd-2d9a342dd72a-maas-api-service-ca\") pod \"9846a62a-c4aa-4937-b1dd-2d9a342dd72a\" (UID: \"9846a62a-c4aa-4937-b1dd-2d9a342dd72a\") "
Nov 02 15:45:24.605518 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:24.605053    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/647ad5be-64be-4aff-bff6-d0ac2943671c-kube-api-access-qh7nb\" (UniqueName: \"kubernetes.io/projected/647ad5be-64be-4aff-bff6-d0ac2943671c-kube-api-access-qh7nb\") pod \"647ad5be-64be-4aff-bff6-d0ac2943671c\" (UID: \"647ad5be-64be-4aff-bff6-d0ac2943671c\") "
Nov 02 15:45:24.605518 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:24.605343    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9846a62a-c4aa-4937-b1dd-2d9a342dd72a-maas-api-service-ca" pod "9846a62a-c4aa-4937-b1dd-2d9a342dd72a" (UID: "9846a62a-c4aa-4937-b1dd-2d9a342dd72a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:45:24.605816 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:24.605793    5304 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9846a62a-c4aa-4937-b1dd-2d9a342dd72a-maas-api-service-ca\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:45:24.606972 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:24.606950    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/647ad5be-64be-4aff-bff6-d0ac2943671c-maas-api-service-ca" pod "647ad5be-64be-4aff-bff6-d0ac2943671c" (UID: "647ad5be-64be-4aff-bff6-d0ac2943671c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:45:24.608058 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:24.608030    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/647ad5be-64be-4aff-bff6-d0ac2943671c-kube-api-access-qh7nb" pod "647ad5be-64be-4aff-bff6-d0ac2943671c" (UID: "647ad5be-64be-4aff-bff6-d0ac2943671c"). InnerVolumeSpecName "kube-api-access-qh7nb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:45:24.608562 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:24.608538    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9846a62a-c4aa-4937-b1dd-2d9a342dd72a-kube-api-access-9dsnf" pod "9846a62a-c4aa-4937-b1dd-2d9a342dd72a" (UID: "9846a62a-c4aa-4937-b1dd-2d9a342dd72a"). InnerVolumeSpecName "kube-api-access-9dsnf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:45:24.716931 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:24.709209    5304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qh7nb\" (UniqueName: \"kubernetes.io/projected/647ad5be-64be-4aff-bff6-d0ac2943671c-kube-api-access-qh7nb\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:45:24.716931 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:24.709257    5304 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/647ad5be-64be-4aff-bff6-d0ac2943671c-maas-api-service-ca\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:45:24.716931 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:24.709274    5304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9dsnf\" (UniqueName: \"kubernetes.io/projected/9846a62a-c4aa-4937-b1dd-2d9a342dd72a-kube-api-access-9dsnf\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:45:24.800605 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:45:24.800559    5304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k6fjk" containerName="extractor"
Nov 02 15:45:25.188382 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:25.188223    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893905-4d75m"
Nov 02 15:45:25.188382 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:25.188222    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893905-4d75m" event={"ID":"9846a62a-c4aa-4937-b1dd-2d9a342dd72a","Type":"ContainerDied","Data":"50f5621dbfbce7f5a461ba67f09fd343c2886c53e42df3cac2ebddaca3a1b244"}
Nov 02 15:45:25.188382 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:25.188374    5304 scope.go:122] "RemoveContainer" containerID="b1e0a6f1ff2ca4c43df2b8590c33d028079b47b11f8a8b8503ca1e6c12aa0089"
Nov 02 15:45:25.191863 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:25.190330    5304 generic.go:356] "Generic (PLEG): container finished" podID="ea2c4ec8-2ace-49dd-a172-99ced21dbc4f" containerID="21987302651eeabb9267207030c7a2aded621a4f3a2c46e57ed7c45e2c00dc73" exitCode=60
Nov 02 15:45:25.191863 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:25.190435    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-1cc1b2-29893905-whfr9" event={"ID":"ea2c4ec8-2ace-49dd-a172-99ced21dbc4f","Type":"ContainerDied","Data":"21987302651eeabb9267207030c7a2aded621a4f3a2c46e57ed7c45e2c00dc73"}
Nov 02 15:45:25.191863 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:25.190894    5304 scope.go:122] "RemoveContainer" containerID="21987302651eeabb9267207030c7a2aded621a4f3a2c46e57ed7c45e2c00dc73"
Nov 02 15:45:25.191863 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:45:25.191283    5304 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-1cc1b2-29893905-whfr9_odh-ai-gateway-infra(ea2c4ec8-2ace-49dd-a172-99ced21dbc4f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-1cc1b2-29893905-whfr9" podUID="ea2c4ec8-2ace-49dd-a172-99ced21dbc4f"
Nov 02 15:45:25.195953 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:25.195930    5304 generic.go:356] "Generic (PLEG): container finished" podID="96b6b761-e07b-41be-8460-f0b68cd55d4a" containerID="fccc01c9ac53fde16b503bec6219393089d9a23485208ba252d89e3a508b275d" exitCode=0
Nov 02 15:45:25.196061 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:25.196044    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bbd4709-nbnqr" event={"ID":"96b6b761-e07b-41be-8460-f0b68cd55d4a","Type":"ContainerDied","Data":"fccc01c9ac53fde16b503bec6219393089d9a23485208ba252d89e3a508b275d"}
Nov 02 15:45:25.198426 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:25.198407    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-b55da1-29893905-mv5qt"
Nov 02 15:45:25.198872 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:25.198435    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-b55da1-29893905-mv5qt" event={"ID":"647ad5be-64be-4aff-bff6-d0ac2943671c","Type":"ContainerDied","Data":"bb48e24350c7e12d61335355d124cad3555e3fc3fb3ee44dfd363812948934d0"}
Nov 02 15:45:25.211556 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:25.211533    5304 scope.go:122] "RemoveContainer" containerID="1bd49a527b9b72cc455bd79351cd7c9e8119b3a457e0f6f430d422c85a007b95"
Nov 02 15:45:25.259889 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:25.256686    5304 scope.go:122] "RemoveContainer" containerID="08bfdebaae5169fd787988c5d21192d32984c434ff7a02274371a34179334fdc"
Nov 02 15:45:25.284563 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:25.284535    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-b55da1-29893905-mv5qt"]
Nov 02 15:45:25.293055 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:25.293008    5304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-b55da1-29893905-mv5qt"]
Nov 02 15:45:25.326439 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:25.326180    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893905-4d75m"]
Nov 02 15:45:25.366282 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:25.366230    5304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893905-4d75m"]
Nov 02 15:45:25.472631 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:25.472510    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5113c2-29893905-bnvks"
Nov 02 15:45:25.567328 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:25.567289    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-1cc1b2-29893905-whfr9"]
Nov 02 15:45:25.581829 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:25.581798    5304 patch_prober.go:29] interesting pod/e2e-ipp-b-3805b8-openshift-default-6c9f89869-wr624 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused" start-of-body=
Nov 02 15:45:25.582082 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:25.581853    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-wr624" podUID="20517696-bd32-4370-a351-67c838f8ccf6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused"
Nov 02 15:45:25.622357 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:25.622329    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5efd887c-66f3-445e-9e39-0d23e43cc9be-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5efd887c-66f3-445e-9e39-0d23e43cc9be-maas-api-service-ca\") pod \"5efd887c-66f3-445e-9e39-0d23e43cc9be\" (UID: \"5efd887c-66f3-445e-9e39-0d23e43cc9be\") "
Nov 02 15:45:25.622612 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:25.622598    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5efd887c-66f3-445e-9e39-0d23e43cc9be-kube-api-access-hf7db\" (UniqueName: \"kubernetes.io/projected/5efd887c-66f3-445e-9e39-0d23e43cc9be-kube-api-access-hf7db\") pod \"5efd887c-66f3-445e-9e39-0d23e43cc9be\" (UID: \"5efd887c-66f3-445e-9e39-0d23e43cc9be\") "
Nov 02 15:45:25.622745 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:25.622718    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5efd887c-66f3-445e-9e39-0d23e43cc9be-maas-api-service-ca" pod "5efd887c-66f3-445e-9e39-0d23e43cc9be" (UID: "5efd887c-66f3-445e-9e39-0d23e43cc9be"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:45:25.623068 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:25.623051    5304 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5efd887c-66f3-445e-9e39-0d23e43cc9be-maas-api-service-ca\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:45:25.624743 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:25.624714    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5efd887c-66f3-445e-9e39-0d23e43cc9be-kube-api-access-hf7db" pod "5efd887c-66f3-445e-9e39-0d23e43cc9be" (UID: "5efd887c-66f3-445e-9e39-0d23e43cc9be"). InnerVolumeSpecName "kube-api-access-hf7db". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:45:25.724500 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:25.724410    5304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hf7db\" (UniqueName: \"kubernetes.io/projected/5efd887c-66f3-445e-9e39-0d23e43cc9be-kube-api-access-hf7db\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:45:26.120975 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:26.120942    5304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4cce806c-5164-485a-a790-519176d79e26" path="/var/lib/kubelet/pods/4cce806c-5164-485a-a790-519176d79e26/volumes"
Nov 02 15:45:26.121469 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:26.121454    5304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="647ad5be-64be-4aff-bff6-d0ac2943671c" path="/var/lib/kubelet/pods/647ad5be-64be-4aff-bff6-d0ac2943671c/volumes"
Nov 02 15:45:26.121912 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:26.121898    5304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8ea83c17-d3cb-4d7e-a6ef-cbd731415216" path="/var/lib/kubelet/pods/8ea83c17-d3cb-4d7e-a6ef-cbd731415216/volumes"
Nov 02 15:45:26.122612 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:26.122596    5304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9846a62a-c4aa-4937-b1dd-2d9a342dd72a" path="/var/lib/kubelet/pods/9846a62a-c4aa-4937-b1dd-2d9a342dd72a/volumes"
Nov 02 15:45:26.123033 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:26.123020    5304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a07ca06a-c083-4fe9-bca4-1614646e80fb" path="/var/lib/kubelet/pods/a07ca06a-c083-4fe9-bca4-1614646e80fb/volumes"
Nov 02 15:45:26.209016 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:26.208988    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5113c2-29893905-bnvks"
Nov 02 15:45:26.209236 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:26.208969    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5113c2-29893905-bnvks" event={"ID":"5efd887c-66f3-445e-9e39-0d23e43cc9be","Type":"ContainerDied","Data":"e7114e07c946ae8292b720c549f55b6bf25e5cfab7c6474741235bea40983442"}
Nov 02 15:45:26.209236 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:26.209154    5304 scope.go:122] "RemoveContainer" containerID="01a29ee5a8a29880062f14bf9d5237f19e808147c0448414092c8d2bce9799af"
Nov 02 15:45:26.272488 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:26.272443    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5113c2-29893905-bnvks"]
Nov 02 15:45:26.277052 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:26.277018    5304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5113c2-29893905-bnvks"]
Nov 02 15:45:26.468429 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:26.468171    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bbd4709-nbnqr"
Nov 02 15:45:26.495471 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:26.495445    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-1cc1b2-29893905-whfr9"
Nov 02 15:45:26.581649 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:26.581612    5304 patch_prober.go:29] interesting pod/e2e-ipp-b-3805b8-openshift-default-6c9f89869-wr624 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused" start-of-body=
Nov 02 15:45:26.581905 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:26.581689    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-wr624" podUID="20517696-bd32-4370-a351-67c838f8ccf6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused"
Nov 02 15:45:26.661785 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:26.661706    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/96b6b761-e07b-41be-8460-f0b68cd55d4a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96b6b761-e07b-41be-8460-f0b68cd55d4a-maas-api-openshift-ca\") pod \"96b6b761-e07b-41be-8460-f0b68cd55d4a\" (UID: \"96b6b761-e07b-41be-8460-f0b68cd55d4a\") "
Nov 02 15:45:26.661785 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:26.661773    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ea2c4ec8-2ace-49dd-a172-99ced21dbc4f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea2c4ec8-2ace-49dd-a172-99ced21dbc4f-maas-api-service-ca\") pod \"ea2c4ec8-2ace-49dd-a172-99ced21dbc4f\" (UID: \"ea2c4ec8-2ace-49dd-a172-99ced21dbc4f\") "
Nov 02 15:45:26.662005 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:26.661978    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/96b6b761-e07b-41be-8460-f0b68cd55d4a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96b6b761-e07b-41be-8460-f0b68cd55d4a-maas-api-xks-ca\") pod \"96b6b761-e07b-41be-8460-f0b68cd55d4a\" (UID: \"96b6b761-e07b-41be-8460-f0b68cd55d4a\") "
Nov 02 15:45:26.662395 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:26.662108    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea2c4ec8-2ace-49dd-a172-99ced21dbc4f-kube-api-access-mq9sq\" (UniqueName: \"kubernetes.io/projected/ea2c4ec8-2ace-49dd-a172-99ced21dbc4f-kube-api-access-mq9sq\") pod \"ea2c4ec8-2ace-49dd-a172-99ced21dbc4f\" (UID: \"ea2c4ec8-2ace-49dd-a172-99ced21dbc4f\") "
Nov 02 15:45:26.662395 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:26.662183    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96b6b761-e07b-41be-8460-f0b68cd55d4a-kube-api-access-99tmq\" (UniqueName: \"kubernetes.io/projected/96b6b761-e07b-41be-8460-f0b68cd55d4a-kube-api-access-99tmq\") pod \"96b6b761-e07b-41be-8460-f0b68cd55d4a\" (UID: \"96b6b761-e07b-41be-8460-f0b68cd55d4a\") "
Nov 02 15:45:26.662395 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:26.662192    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96b6b761-e07b-41be-8460-f0b68cd55d4a-maas-api-openshift-ca" pod "96b6b761-e07b-41be-8460-f0b68cd55d4a" (UID: "96b6b761-e07b-41be-8460-f0b68cd55d4a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:45:26.662395 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:26.662215    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea2c4ec8-2ace-49dd-a172-99ced21dbc4f-maas-api-service-ca" pod "ea2c4ec8-2ace-49dd-a172-99ced21dbc4f" (UID: "ea2c4ec8-2ace-49dd-a172-99ced21dbc4f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:45:26.662738 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:26.662718    5304 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96b6b761-e07b-41be-8460-f0b68cd55d4a-maas-api-openshift-ca\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:45:26.662796 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:26.662744    5304 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea2c4ec8-2ace-49dd-a172-99ced21dbc4f-maas-api-service-ca\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:45:26.664232 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:26.664201    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea2c4ec8-2ace-49dd-a172-99ced21dbc4f-kube-api-access-mq9sq" pod "ea2c4ec8-2ace-49dd-a172-99ced21dbc4f" (UID: "ea2c4ec8-2ace-49dd-a172-99ced21dbc4f"). InnerVolumeSpecName "kube-api-access-mq9sq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:45:26.664749 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:26.664732    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96b6b761-e07b-41be-8460-f0b68cd55d4a-kube-api-access-99tmq" pod "96b6b761-e07b-41be-8460-f0b68cd55d4a" (UID: "96b6b761-e07b-41be-8460-f0b68cd55d4a"). InnerVolumeSpecName "kube-api-access-99tmq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:45:26.664819 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:26.664796    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/96b6b761-e07b-41be-8460-f0b68cd55d4a-maas-api-xks-ca" pod "96b6b761-e07b-41be-8460-f0b68cd55d4a" (UID: "96b6b761-e07b-41be-8460-f0b68cd55d4a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:45:26.778695 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:26.778596    5304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mq9sq\" (UniqueName: \"kubernetes.io/projected/ea2c4ec8-2ace-49dd-a172-99ced21dbc4f-kube-api-access-mq9sq\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:45:26.778695 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:26.778631    5304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-99tmq\" (UniqueName: \"kubernetes.io/projected/96b6b761-e07b-41be-8460-f0b68cd55d4a-kube-api-access-99tmq\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:45:26.778695 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:26.778646    5304 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96b6b761-e07b-41be-8460-f0b68cd55d4a-maas-api-xks-ca\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:45:27.221731 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:27.221703    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bbd4709-nbnqr"
Nov 02 15:45:27.221899 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:27.221736    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bbd4709-nbnqr" event={"ID":"96b6b761-e07b-41be-8460-f0b68cd55d4a","Type":"ContainerDied","Data":"c55d534d729a465a8673b3653376eb526e7240c225071fd91670ba976cecc8b5"}
Nov 02 15:45:27.221899 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:27.221788    5304 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c55d534d729a465a8673b3653376eb526e7240c225071fd91670ba976cecc8b5"
Nov 02 15:45:27.224394 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:27.224367    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-1cc1b2-29893905-whfr9" event={"ID":"ea2c4ec8-2ace-49dd-a172-99ced21dbc4f","Type":"ContainerDied","Data":"33ba14be7000721034bb3b3740721f00c59b7c570f9754744178679b1b85d442"}
Nov 02 15:45:27.224394 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:27.224390    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-1cc1b2-29893905-whfr9"
Nov 02 15:45:27.224603 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:27.224421    5304 scope.go:122] "RemoveContainer" containerID="21987302651eeabb9267207030c7a2aded621a4f3a2c46e57ed7c45e2c00dc73"
Nov 02 15:45:27.278745 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:27.278710    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-1cc1b2-29893905-whfr9"]
Nov 02 15:45:27.290110 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:27.290061    5304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-1cc1b2-29893905-whfr9"]
Nov 02 15:45:27.586682 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:27.586648    5304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-wr624"
Nov 02 15:45:27.587087 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:27.587041    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-wr624"
Nov 02 15:45:27.587927 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:27.587904    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-wr624"
Nov 02 15:45:28.121271 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:28.121228    5304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5efd887c-66f3-445e-9e39-0d23e43cc9be" path="/var/lib/kubelet/pods/5efd887c-66f3-445e-9e39-0d23e43cc9be/volumes"
Nov 02 15:45:28.121861 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:28.121837    5304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ea2c4ec8-2ace-49dd-a172-99ced21dbc4f" path="/var/lib/kubelet/pods/ea2c4ec8-2ace-49dd-a172-99ced21dbc4f/volumes"
Nov 02 15:45:40.792915 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:40.792876    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-9cfd1f-77499c8864-p9ccn"]
Nov 02 15:45:40.793483 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:40.793094    5304 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-9cfd1f-77499c8864-p9ccn" podUID="c85057ae-7026-4b45-8ce9-2568c3027565" containerName="payload-processing" containerID="cri-o://c5133aa6e14ffbf1455468aa960943e2b2647f981ad4a90b3f7ab890eb10b079" gracePeriod=30
Nov 02 15:45:40.828636 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:40.828600    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-9cfd1f-9bf597b4c-6fbz9"]
Nov 02 15:45:40.828865 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:40.828837    5304 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9cfd1f-9bf597b4c-6fbz9" podUID="d8479a68-3948-4660-a797-8aca350dcd67" containerName="payload-pre-processing" containerID="cri-o://e11a96e2f8769c7790258581d5fdd1625e9ad831d7762a360956df550b3b22df" gracePeriod=30
Nov 02 15:45:41.270848 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:41.259581    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-9cfd1f-77499c8864-p9ccn"
Nov 02 15:45:41.293203 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:41.281804    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9cfd1f-9bf597b4c-6fbz9"
Nov 02 15:45:41.336203 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:41.331817    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c85057ae-7026-4b45-8ce9-2568c3027565-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c85057ae-7026-4b45-8ce9-2568c3027565-plugins-config-volume\") pod \"c85057ae-7026-4b45-8ce9-2568c3027565\" (UID: \"c85057ae-7026-4b45-8ce9-2568c3027565\") "
Nov 02 15:45:41.336203 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:41.331876    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c85057ae-7026-4b45-8ce9-2568c3027565-kube-api-access-7ffnt\" (UniqueName: \"kubernetes.io/projected/c85057ae-7026-4b45-8ce9-2568c3027565-kube-api-access-7ffnt\") pod \"c85057ae-7026-4b45-8ce9-2568c3027565\" (UID: \"c85057ae-7026-4b45-8ce9-2568c3027565\") "
Nov 02 15:45:41.340146 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:41.339422    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c85057ae-7026-4b45-8ce9-2568c3027565-plugins-config-volume" pod "c85057ae-7026-4b45-8ce9-2568c3027565" (UID: "c85057ae-7026-4b45-8ce9-2568c3027565"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:45:41.356628 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:41.356591    5304 generic.go:356] "Generic (PLEG): container finished" podID="c85057ae-7026-4b45-8ce9-2568c3027565" containerID="c5133aa6e14ffbf1455468aa960943e2b2647f981ad4a90b3f7ab890eb10b079" exitCode=0
Nov 02 15:45:41.358253 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:41.358220    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-9cfd1f-77499c8864-p9ccn" event={"ID":"c85057ae-7026-4b45-8ce9-2568c3027565","Type":"ContainerDied","Data":"c5133aa6e14ffbf1455468aa960943e2b2647f981ad4a90b3f7ab890eb10b079"}
Nov 02 15:45:41.358936 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:41.358460    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-9cfd1f-77499c8864-p9ccn" event={"ID":"c85057ae-7026-4b45-8ce9-2568c3027565","Type":"ContainerDied","Data":"42a582dffc6f3e8af274e5901bba56b6a8ae00af6ce32b5c5ac891c5154ffa4d"}
Nov 02 15:45:41.358936 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:41.358499    5304 scope.go:122] "RemoveContainer" containerID="c5133aa6e14ffbf1455468aa960943e2b2647f981ad4a90b3f7ab890eb10b079"
Nov 02 15:45:41.358936 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:41.358677    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-9cfd1f-77499c8864-p9ccn"
Nov 02 15:45:41.362703 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:41.362664    5304 generic.go:356] "Generic (PLEG): container finished" podID="d8479a68-3948-4660-a797-8aca350dcd67" containerID="e11a96e2f8769c7790258581d5fdd1625e9ad831d7762a360956df550b3b22df" exitCode=0
Nov 02 15:45:41.362927 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:41.362903    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9cfd1f-9bf597b4c-6fbz9" event={"ID":"d8479a68-3948-4660-a797-8aca350dcd67","Type":"ContainerDied","Data":"e11a96e2f8769c7790258581d5fdd1625e9ad831d7762a360956df550b3b22df"}
Nov 02 15:45:41.363069 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:41.363052    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9cfd1f-9bf597b4c-6fbz9" event={"ID":"d8479a68-3948-4660-a797-8aca350dcd67","Type":"ContainerDied","Data":"5536dc9304299982ecba1927285b9143b9d58102995b9a8b3f8940726a5c6b2a"}
Nov 02 15:45:41.363416 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:41.363398    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9cfd1f-9bf597b4c-6fbz9"
Nov 02 15:45:41.421571 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:41.421194    5304 scope.go:122] "RemoveContainer" containerID="c5133aa6e14ffbf1455468aa960943e2b2647f981ad4a90b3f7ab890eb10b079"
Nov 02 15:45:41.422226 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:45:41.421837    5304 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c5133aa6e14ffbf1455468aa960943e2b2647f981ad4a90b3f7ab890eb10b079\": container with ID starting with c5133aa6e14ffbf1455468aa960943e2b2647f981ad4a90b3f7ab890eb10b079 not found: ID does not exist" containerID="c5133aa6e14ffbf1455468aa960943e2b2647f981ad4a90b3f7ab890eb10b079"
Nov 02 15:45:41.422226 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:41.421876    5304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5133aa6e14ffbf1455468aa960943e2b2647f981ad4a90b3f7ab890eb10b079"} err="failed to get container status \"c5133aa6e14ffbf1455468aa960943e2b2647f981ad4a90b3f7ab890eb10b079\": rpc error: code = NotFound desc = could not find container \"c5133aa6e14ffbf1455468aa960943e2b2647f981ad4a90b3f7ab890eb10b079\": container with ID starting with c5133aa6e14ffbf1455468aa960943e2b2647f981ad4a90b3f7ab890eb10b079 not found: ID does not exist"
Nov 02 15:45:41.422226 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:41.421901    5304 scope.go:122] "RemoveContainer" containerID="e11a96e2f8769c7790258581d5fdd1625e9ad831d7762a360956df550b3b22df"
Nov 02 15:45:41.445894 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:41.436326    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d8479a68-3948-4660-a797-8aca350dcd67-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d8479a68-3948-4660-a797-8aca350dcd67-plugins-config-volume\") pod \"d8479a68-3948-4660-a797-8aca350dcd67\" (UID: \"d8479a68-3948-4660-a797-8aca350dcd67\") "
Nov 02 15:45:41.445894 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:41.436411    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d8479a68-3948-4660-a797-8aca350dcd67-kube-api-access-vw4z6\" (UniqueName: \"kubernetes.io/projected/d8479a68-3948-4660-a797-8aca350dcd67-kube-api-access-vw4z6\") pod \"d8479a68-3948-4660-a797-8aca350dcd67\" (UID: \"d8479a68-3948-4660-a797-8aca350dcd67\") "
Nov 02 15:45:41.445894 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:41.437283    5304 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c85057ae-7026-4b45-8ce9-2568c3027565-plugins-config-volume\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:45:41.445894 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:41.437680    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c85057ae-7026-4b45-8ce9-2568c3027565-kube-api-access-7ffnt" pod "c85057ae-7026-4b45-8ce9-2568c3027565" (UID: "c85057ae-7026-4b45-8ce9-2568c3027565"). InnerVolumeSpecName "kube-api-access-7ffnt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:45:41.445894 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:41.437988    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8479a68-3948-4660-a797-8aca350dcd67-plugins-config-volume" pod "d8479a68-3948-4660-a797-8aca350dcd67" (UID: "d8479a68-3948-4660-a797-8aca350dcd67"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:45:41.445894 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:41.442196    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8479a68-3948-4660-a797-8aca350dcd67-kube-api-access-vw4z6" pod "d8479a68-3948-4660-a797-8aca350dcd67" (UID: "d8479a68-3948-4660-a797-8aca350dcd67"). InnerVolumeSpecName "kube-api-access-vw4z6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:45:41.547715 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:41.547550    5304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vw4z6\" (UniqueName: \"kubernetes.io/projected/d8479a68-3948-4660-a797-8aca350dcd67-kube-api-access-vw4z6\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:45:41.547715 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:41.547592    5304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7ffnt\" (UniqueName: \"kubernetes.io/projected/c85057ae-7026-4b45-8ce9-2568c3027565-kube-api-access-7ffnt\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:45:41.547715 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:41.547607    5304 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d8479a68-3948-4660-a797-8aca350dcd67-plugins-config-volume\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:45:41.550006 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:41.549971    5304 scope.go:122] "RemoveContainer" containerID="e11a96e2f8769c7790258581d5fdd1625e9ad831d7762a360956df550b3b22df"
Nov 02 15:45:41.551233 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:45:41.551076    5304 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e11a96e2f8769c7790258581d5fdd1625e9ad831d7762a360956df550b3b22df\": container with ID starting with e11a96e2f8769c7790258581d5fdd1625e9ad831d7762a360956df550b3b22df not found: ID does not exist" containerID="e11a96e2f8769c7790258581d5fdd1625e9ad831d7762a360956df550b3b22df"
Nov 02 15:45:41.551392 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:41.551116    5304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e11a96e2f8769c7790258581d5fdd1625e9ad831d7762a360956df550b3b22df"} err="failed to get container status \"e11a96e2f8769c7790258581d5fdd1625e9ad831d7762a360956df550b3b22df\": rpc error: code = NotFound desc = could not find container \"e11a96e2f8769c7790258581d5fdd1625e9ad831d7762a360956df550b3b22df\": container with ID starting with e11a96e2f8769c7790258581d5fdd1625e9ad831d7762a360956df550b3b22df not found: ID does not exist"
Nov 02 15:45:41.705623 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:41.705592    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-9cfd1f-77499c8864-p9ccn"]
Nov 02 15:45:41.717375 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:41.717347    5304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-9cfd1f-77499c8864-p9ccn"]
Nov 02 15:45:41.745610 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:41.745573    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-9cfd1f-9bf597b4c-6fbz9"]
Nov 02 15:45:41.750069 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:41.750040    5304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-9cfd1f-9bf597b4c-6fbz9"]
Nov 02 15:45:42.122349 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:42.122036    5304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c85057ae-7026-4b45-8ce9-2568c3027565" path="/var/lib/kubelet/pods/c85057ae-7026-4b45-8ce9-2568c3027565/volumes"
Nov 02 15:45:42.122913 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:42.122889    5304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d8479a68-3948-4660-a797-8aca350dcd67" path="/var/lib/kubelet/pods/d8479a68-3948-4660-a797-8aca350dcd67/volumes"
Nov 02 15:45:48.630179 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:48.619363    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-9cfd1f-openshift-default-6fbb7b57d9-hh28t"]
Nov 02 15:45:48.630179 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:48.619631    5304 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-9cfd1f-openshift-default-6fbb7b57d9-hh28t" podUID="d665d588-97c3-4797-be57-c426b3d905ff" containerName="istio-proxy" containerID="cri-o://c775141644658fa89c11450fa81952fd56da0b264110c7fc3655b74f6763e0dc" gracePeriod=30
Nov 02 15:45:50.385059 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:50.385026    5304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-24255b-openshift-default-6687484f49-992gq"]
Nov 02 15:45:50.385836 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:50.385818    5304 cpu_manager.go:414] "containerMap: removing container" podUID="ea2c4ec8-2ace-49dd-a172-99ced21dbc4f" containerName="cleanup"
Nov 02 15:45:50.385900 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:50.385839    5304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea2c4ec8-2ace-49dd-a172-99ced21dbc4f" containerName="cleanup"
Nov 02 15:45:50.385900 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:50.385849    5304 cpu_manager.go:414] "containerMap: removing container" podUID="4cce806c-5164-485a-a790-519176d79e26" containerName="cleanup"
Nov 02 15:45:50.385900 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:50.385854    5304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4cce806c-5164-485a-a790-519176d79e26" containerName="cleanup"
Nov 02 15:45:50.385900 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:50.385866    5304 cpu_manager.go:414] "containerMap: removing container" podUID="5efd887c-66f3-445e-9e39-0d23e43cc9be" containerName="cleanup"
Nov 02 15:45:50.385900 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:50.385874    5304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5efd887c-66f3-445e-9e39-0d23e43cc9be" containerName="cleanup"
Nov 02 15:45:50.385900 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:50.385888    5304 cpu_manager.go:414] "containerMap: removing container" podUID="8ea83c17-d3cb-4d7e-a6ef-cbd731415216" containerName="cleanup"
Nov 02 15:45:50.385900 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:50.385893    5304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ea83c17-d3cb-4d7e-a6ef-cbd731415216" containerName="cleanup"
Nov 02 15:45:50.385900 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:50.385899    5304 cpu_manager.go:414] "containerMap: removing container" podUID="ea2c4ec8-2ace-49dd-a172-99ced21dbc4f" containerName="cleanup"
Nov 02 15:45:50.386187 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:50.385905    5304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea2c4ec8-2ace-49dd-a172-99ced21dbc4f" containerName="cleanup"
Nov 02 15:45:50.386187 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:50.385915    5304 cpu_manager.go:414] "containerMap: removing container" podUID="1d05aa2a-2dfb-47be-9573-a8f32cf05ab7" containerName="cleanup"
Nov 02 15:45:50.386187 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:50.385920    5304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d05aa2a-2dfb-47be-9573-a8f32cf05ab7" containerName="cleanup"
Nov 02 15:45:50.386187 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:50.385927    5304 cpu_manager.go:414] "containerMap: removing container" podUID="8ea83c17-d3cb-4d7e-a6ef-cbd731415216" containerName="cleanup"
Nov 02 15:45:50.386187 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:50.385931    5304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ea83c17-d3cb-4d7e-a6ef-cbd731415216" containerName="cleanup"
Nov 02 15:45:50.386187 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:50.385941    5304 cpu_manager.go:414] "containerMap: removing container" podUID="9846a62a-c4aa-4937-b1dd-2d9a342dd72a" containerName="cleanup"
Nov 02 15:45:50.386187 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:50.385947    5304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9846a62a-c4aa-4937-b1dd-2d9a342dd72a" containerName="cleanup"
Nov 02 15:45:50.386187 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:50.385959    5304 cpu_manager.go:414] "containerMap: removing container" podUID="647ad5be-64be-4aff-bff6-d0ac2943671c" containerName="cleanup"
Nov 02 15:45:50.386187 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:50.385964    5304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="647ad5be-64be-4aff-bff6-d0ac2943671c" containerName="cleanup"
Nov 02 15:45:50.386187 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:50.385971    5304 cpu_manager.go:414] "containerMap: removing container" podUID="a07ca06a-c083-4fe9-bca4-1614646e80fb" containerName="cleanup"
Nov 02 15:45:50.386187 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:50.385976    5304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a07ca06a-c083-4fe9-bca4-1614646e80fb" containerName="cleanup"
Nov 02 15:45:50.386187 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:50.385983    5304 cpu_manager.go:414] "containerMap: removing container" podUID="4cce806c-5164-485a-a790-519176d79e26" containerName="cleanup"
Nov 02 15:45:50.386187 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:50.385987    5304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4cce806c-5164-485a-a790-519176d79e26" containerName="cleanup"
Nov 02 15:45:50.386187 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:50.385996    5304 cpu_manager.go:414] "containerMap: removing container" podUID="9846a62a-c4aa-4937-b1dd-2d9a342dd72a" containerName="cleanup"
Nov 02 15:45:50.386187 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:50.386000    5304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9846a62a-c4aa-4937-b1dd-2d9a342dd72a" containerName="cleanup"
Nov 02 15:45:50.386187 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:50.386007    5304 cpu_manager.go:414] "containerMap: removing container" podUID="4cce806c-5164-485a-a790-519176d79e26" containerName="cleanup"
Nov 02 15:45:50.386187 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:50.386011    5304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4cce806c-5164-485a-a790-519176d79e26" containerName="cleanup"
Nov 02 15:45:50.386187 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:50.386018    5304 cpu_manager.go:414] "containerMap: removing container" podUID="ada2b1b0-e598-4d38-b995-becab65c7e82" containerName="cleanup"
Nov 02 15:45:50.386187 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:50.386023    5304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ada2b1b0-e598-4d38-b995-becab65c7e82" containerName="cleanup"
Nov 02 15:45:50.386187 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:50.386032    5304 cpu_manager.go:414] "containerMap: removing container" podUID="5efd887c-66f3-445e-9e39-0d23e43cc9be" containerName="cleanup"
Nov 02 15:45:50.386187 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:50.386039    5304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5efd887c-66f3-445e-9e39-0d23e43cc9be" containerName="cleanup"
Nov 02 15:45:50.386187 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:50.386047    5304 cpu_manager.go:414] "containerMap: removing container" podUID="5efd887c-66f3-445e-9e39-0d23e43cc9be" containerName="cleanup"
Nov 02 15:45:50.386187 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:50.386052    5304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5efd887c-66f3-445e-9e39-0d23e43cc9be" containerName="cleanup"
Nov 02 15:45:50.386187 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:50.386059    5304 cpu_manager.go:414] "containerMap: removing container" podUID="c85057ae-7026-4b45-8ce9-2568c3027565" containerName="payload-processing"
Nov 02 15:45:50.386187 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:50.386064    5304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c85057ae-7026-4b45-8ce9-2568c3027565" containerName="payload-processing"
Nov 02 15:45:50.386187 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:50.386073    5304 cpu_manager.go:414] "containerMap: removing container" podUID="9846a62a-c4aa-4937-b1dd-2d9a342dd72a" containerName="cleanup"
Nov 02 15:45:50.386187 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:50.386077    5304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9846a62a-c4aa-4937-b1dd-2d9a342dd72a" containerName="cleanup"
Nov 02 15:45:50.386187 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:50.386087    5304 cpu_manager.go:414] "containerMap: removing container" podUID="a07ca06a-c083-4fe9-bca4-1614646e80fb" containerName="cleanup"
Nov 02 15:45:50.386187 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:50.386092    5304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a07ca06a-c083-4fe9-bca4-1614646e80fb" containerName="cleanup"
Nov 02 15:45:50.386187 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:50.386102    5304 cpu_manager.go:414] "containerMap: removing container" podUID="a07ca06a-c083-4fe9-bca4-1614646e80fb" containerName="cleanup"
Nov 02 15:45:50.386187 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:50.386109    5304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a07ca06a-c083-4fe9-bca4-1614646e80fb" containerName="cleanup"
Nov 02 15:45:50.386187 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:50.386121    5304 cpu_manager.go:414] "containerMap: removing container" podUID="ada2b1b0-e598-4d38-b995-becab65c7e82" containerName="cleanup"
Nov 02 15:45:50.386187 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:50.386142    5304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ada2b1b0-e598-4d38-b995-becab65c7e82" containerName="cleanup"
Nov 02 15:45:50.386187 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:50.386152    5304 cpu_manager.go:414] "containerMap: removing container" podUID="ada2b1b0-e598-4d38-b995-becab65c7e82" containerName="cleanup"
Nov 02 15:45:50.386187 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:50.386156    5304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ada2b1b0-e598-4d38-b995-becab65c7e82" containerName="cleanup"
Nov 02 15:45:50.386187 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:50.386164    5304 cpu_manager.go:414] "containerMap: removing container" podUID="96b6b761-e07b-41be-8460-f0b68cd55d4a" containerName="revoke-keys"
Nov 02 15:45:50.386187 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:50.386168    5304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96b6b761-e07b-41be-8460-f0b68cd55d4a" containerName="revoke-keys"
Nov 02 15:45:50.386187 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:50.386179    5304 cpu_manager.go:414] "containerMap: removing container" podUID="d8479a68-3948-4660-a797-8aca350dcd67" containerName="payload-pre-processing"
Nov 02 15:45:50.386187 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:50.386187    5304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d8479a68-3948-4660-a797-8aca350dcd67" containerName="payload-pre-processing"
Nov 02 15:45:50.386187 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:50.386200    5304 cpu_manager.go:414] "containerMap: removing container" podUID="647ad5be-64be-4aff-bff6-d0ac2943671c" containerName="cleanup"
Nov 02 15:45:50.386187 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:50.386204    5304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="647ad5be-64be-4aff-bff6-d0ac2943671c" containerName="cleanup"
Nov 02 15:45:50.387687 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:50.386301    5304 memory_manager.go:365] "RemoveStaleState removing state" podUID="9846a62a-c4aa-4937-b1dd-2d9a342dd72a" containerName="cleanup"
Nov 02 15:45:50.387687 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:50.386310    5304 memory_manager.go:365] "RemoveStaleState removing state" podUID="a07ca06a-c083-4fe9-bca4-1614646e80fb" containerName="cleanup"
Nov 02 15:45:50.387687 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:50.386318    5304 memory_manager.go:365] "RemoveStaleState removing state" podUID="647ad5be-64be-4aff-bff6-d0ac2943671c" containerName="cleanup"
Nov 02 15:45:50.387687 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:50.386324    5304 memory_manager.go:365] "RemoveStaleState removing state" podUID="a07ca06a-c083-4fe9-bca4-1614646e80fb" containerName="cleanup"
Nov 02 15:45:50.387687 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:50.386330    5304 memory_manager.go:365] "RemoveStaleState removing state" podUID="ada2b1b0-e598-4d38-b995-becab65c7e82" containerName="cleanup"
Nov 02 15:45:50.387687 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:50.386339    5304 memory_manager.go:365] "RemoveStaleState removing state" podUID="ada2b1b0-e598-4d38-b995-becab65c7e82" containerName="cleanup"
Nov 02 15:45:50.387687 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:50.386346    5304 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ea83c17-d3cb-4d7e-a6ef-cbd731415216" containerName="cleanup"
Nov 02 15:45:50.387687 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:50.386353    5304 memory_manager.go:365] "RemoveStaleState removing state" podUID="96b6b761-e07b-41be-8460-f0b68cd55d4a" containerName="revoke-keys"
Nov 02 15:45:50.387687 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:50.386359    5304 memory_manager.go:365] "RemoveStaleState removing state" podUID="d8479a68-3948-4660-a797-8aca350dcd67" containerName="payload-pre-processing"
Nov 02 15:45:50.387687 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:50.386365    5304 memory_manager.go:365] "RemoveStaleState removing state" podUID="647ad5be-64be-4aff-bff6-d0ac2943671c" containerName="cleanup"
Nov 02 15:45:50.387687 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:50.386371    5304 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea2c4ec8-2ace-49dd-a172-99ced21dbc4f" containerName="cleanup"
Nov 02 15:45:50.387687 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:50.386376    5304 memory_manager.go:365] "RemoveStaleState removing state" podUID="4cce806c-5164-485a-a790-519176d79e26" containerName="cleanup"
Nov 02 15:45:50.387687 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:50.386382    5304 memory_manager.go:365] "RemoveStaleState removing state" podUID="5efd887c-66f3-445e-9e39-0d23e43cc9be" containerName="cleanup"
Nov 02 15:45:50.387687 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:50.386387    5304 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ea83c17-d3cb-4d7e-a6ef-cbd731415216" containerName="cleanup"
Nov 02 15:45:50.387687 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:50.386392    5304 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea2c4ec8-2ace-49dd-a172-99ced21dbc4f" containerName="cleanup"
Nov 02 15:45:50.387687 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:50.386397    5304 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d05aa2a-2dfb-47be-9573-a8f32cf05ab7" containerName="cleanup"
Nov 02 15:45:50.387687 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:50.386403    5304 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ea83c17-d3cb-4d7e-a6ef-cbd731415216" containerName="cleanup"
Nov 02 15:45:50.387687 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:50.386414    5304 memory_manager.go:365] "RemoveStaleState removing state" podUID="647ad5be-64be-4aff-bff6-d0ac2943671c" containerName="cleanup"
Nov 02 15:45:50.387687 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:50.386423    5304 memory_manager.go:365] "RemoveStaleState removing state" podUID="e0819318-3392-48ed-a0e4-4d7961144393" containerName="cleanup"
Nov 02 15:45:50.387687 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:50.386432    5304 memory_manager.go:365] "RemoveStaleState removing state" podUID="4cce806c-5164-485a-a790-519176d79e26" containerName="cleanup"
Nov 02 15:45:50.387687 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:50.386442    5304 memory_manager.go:365] "RemoveStaleState removing state" podUID="9846a62a-c4aa-4937-b1dd-2d9a342dd72a" containerName="cleanup"
Nov 02 15:45:50.387687 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:50.386450    5304 memory_manager.go:365] "RemoveStaleState removing state" podUID="4cce806c-5164-485a-a790-519176d79e26" containerName="cleanup"
Nov 02 15:45:50.387687 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:50.386458    5304 memory_manager.go:365] "RemoveStaleState removing state" podUID="5efd887c-66f3-445e-9e39-0d23e43cc9be" containerName="cleanup"
Nov 02 15:45:50.387687 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:50.386467    5304 memory_manager.go:365] "RemoveStaleState removing state" podUID="5efd887c-66f3-445e-9e39-0d23e43cc9be" containerName="cleanup"
Nov 02 15:45:50.387687 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:50.386473    5304 memory_manager.go:365] "RemoveStaleState removing state" podUID="c85057ae-7026-4b45-8ce9-2568c3027565" containerName="payload-processing"
Nov 02 15:45:50.392104 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:50.392080    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-24255b-openshift-default-6687484f49-992gq"
Nov 02 15:45:50.423608 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:50.423575    5304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-24255b-openshift-default-dockercfg-pqcsc\""
Nov 02 15:45:50.427930 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:50.427900    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-24255b-openshift-default-6687484f49-992gq"]
Nov 02 15:45:50.455254 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:50.455214    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c19acd3a-5dbf-4da3-b88f-e36284b22243-workload-certs\") pod \"e2e-ipp-b-24255b-openshift-default-6687484f49-992gq\" (UID: \"c19acd3a-5dbf-4da3-b88f-e36284b22243\") " pod="openshift-ingress/e2e-ipp-b-24255b-openshift-default-6687484f49-992gq"
Nov 02 15:45:50.455511 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:50.455329    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c19acd3a-5dbf-4da3-b88f-e36284b22243-istio-envoy\") pod \"e2e-ipp-b-24255b-openshift-default-6687484f49-992gq\" (UID: \"c19acd3a-5dbf-4da3-b88f-e36284b22243\") " pod="openshift-ingress/e2e-ipp-b-24255b-openshift-default-6687484f49-992gq"
Nov 02 15:45:50.455511 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:50.455358    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cn7fd\" (UniqueName: \"kubernetes.io/projected/c19acd3a-5dbf-4da3-b88f-e36284b22243-kube-api-access-cn7fd\") pod \"e2e-ipp-b-24255b-openshift-default-6687484f49-992gq\" (UID: \"c19acd3a-5dbf-4da3-b88f-e36284b22243\") " pod="openshift-ingress/e2e-ipp-b-24255b-openshift-default-6687484f49-992gq"
Nov 02 15:45:50.455511 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:50.455404    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c19acd3a-5dbf-4da3-b88f-e36284b22243-istiod-ca-cert\") pod \"e2e-ipp-b-24255b-openshift-default-6687484f49-992gq\" (UID: \"c19acd3a-5dbf-4da3-b88f-e36284b22243\") " pod="openshift-ingress/e2e-ipp-b-24255b-openshift-default-6687484f49-992gq"
Nov 02 15:45:50.455511 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:50.455446    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c19acd3a-5dbf-4da3-b88f-e36284b22243-istio-data\") pod \"e2e-ipp-b-24255b-openshift-default-6687484f49-992gq\" (UID: \"c19acd3a-5dbf-4da3-b88f-e36284b22243\") " pod="openshift-ingress/e2e-ipp-b-24255b-openshift-default-6687484f49-992gq"
Nov 02 15:45:50.455511 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:50.455479    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c19acd3a-5dbf-4da3-b88f-e36284b22243-istio-token\") pod \"e2e-ipp-b-24255b-openshift-default-6687484f49-992gq\" (UID: \"c19acd3a-5dbf-4da3-b88f-e36284b22243\") " pod="openshift-ingress/e2e-ipp-b-24255b-openshift-default-6687484f49-992gq"
Nov 02 15:45:50.455819 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:50.455542    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c19acd3a-5dbf-4da3-b88f-e36284b22243-workload-socket\") pod \"e2e-ipp-b-24255b-openshift-default-6687484f49-992gq\" (UID: \"c19acd3a-5dbf-4da3-b88f-e36284b22243\") " pod="openshift-ingress/e2e-ipp-b-24255b-openshift-default-6687484f49-992gq"
Nov 02 15:45:50.455819 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:50.455775    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c19acd3a-5dbf-4da3-b88f-e36284b22243-credential-socket\") pod \"e2e-ipp-b-24255b-openshift-default-6687484f49-992gq\" (UID: \"c19acd3a-5dbf-4da3-b88f-e36284b22243\") " pod="openshift-ingress/e2e-ipp-b-24255b-openshift-default-6687484f49-992gq"
Nov 02 15:45:50.455937 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:50.455844    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c19acd3a-5dbf-4da3-b88f-e36284b22243-istio-podinfo\") pod \"e2e-ipp-b-24255b-openshift-default-6687484f49-992gq\" (UID: \"c19acd3a-5dbf-4da3-b88f-e36284b22243\") " pod="openshift-ingress/e2e-ipp-b-24255b-openshift-default-6687484f49-992gq"
Nov 02 15:45:50.562714 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:50.562676    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c19acd3a-5dbf-4da3-b88f-e36284b22243-credential-socket\") pod \"e2e-ipp-b-24255b-openshift-default-6687484f49-992gq\" (UID: \"c19acd3a-5dbf-4da3-b88f-e36284b22243\") " pod="openshift-ingress/e2e-ipp-b-24255b-openshift-default-6687484f49-992gq"
Nov 02 15:45:50.582334 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:50.582296    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c19acd3a-5dbf-4da3-b88f-e36284b22243-istio-podinfo\") pod \"e2e-ipp-b-24255b-openshift-default-6687484f49-992gq\" (UID: \"c19acd3a-5dbf-4da3-b88f-e36284b22243\") " pod="openshift-ingress/e2e-ipp-b-24255b-openshift-default-6687484f49-992gq"
Nov 02 15:45:50.583372 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:50.583335    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c19acd3a-5dbf-4da3-b88f-e36284b22243-workload-certs\") pod \"e2e-ipp-b-24255b-openshift-default-6687484f49-992gq\" (UID: \"c19acd3a-5dbf-4da3-b88f-e36284b22243\") " pod="openshift-ingress/e2e-ipp-b-24255b-openshift-default-6687484f49-992gq"
Nov 02 15:45:50.584025 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:50.584004    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c19acd3a-5dbf-4da3-b88f-e36284b22243-istio-envoy\") pod \"e2e-ipp-b-24255b-openshift-default-6687484f49-992gq\" (UID: \"c19acd3a-5dbf-4da3-b88f-e36284b22243\") " pod="openshift-ingress/e2e-ipp-b-24255b-openshift-default-6687484f49-992gq"
Nov 02 15:45:50.584548 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:50.584529    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cn7fd\" (UniqueName: \"kubernetes.io/projected/c19acd3a-5dbf-4da3-b88f-e36284b22243-kube-api-access-cn7fd\") pod \"e2e-ipp-b-24255b-openshift-default-6687484f49-992gq\" (UID: \"c19acd3a-5dbf-4da3-b88f-e36284b22243\") " pod="openshift-ingress/e2e-ipp-b-24255b-openshift-default-6687484f49-992gq"
Nov 02 15:45:50.585112 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:50.585090    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c19acd3a-5dbf-4da3-b88f-e36284b22243-istiod-ca-cert\") pod \"e2e-ipp-b-24255b-openshift-default-6687484f49-992gq\" (UID: \"c19acd3a-5dbf-4da3-b88f-e36284b22243\") " pod="openshift-ingress/e2e-ipp-b-24255b-openshift-default-6687484f49-992gq"
Nov 02 15:45:50.586066 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:50.586046    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c19acd3a-5dbf-4da3-b88f-e36284b22243-istio-data\") pod \"e2e-ipp-b-24255b-openshift-default-6687484f49-992gq\" (UID: \"c19acd3a-5dbf-4da3-b88f-e36284b22243\") " pod="openshift-ingress/e2e-ipp-b-24255b-openshift-default-6687484f49-992gq"
Nov 02 15:45:50.586547 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:50.586529    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c19acd3a-5dbf-4da3-b88f-e36284b22243-istio-token\") pod \"e2e-ipp-b-24255b-openshift-default-6687484f49-992gq\" (UID: \"c19acd3a-5dbf-4da3-b88f-e36284b22243\") " pod="openshift-ingress/e2e-ipp-b-24255b-openshift-default-6687484f49-992gq"
Nov 02 15:45:50.587020 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:50.587000    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c19acd3a-5dbf-4da3-b88f-e36284b22243-workload-socket\") pod \"e2e-ipp-b-24255b-openshift-default-6687484f49-992gq\" (UID: \"c19acd3a-5dbf-4da3-b88f-e36284b22243\") " pod="openshift-ingress/e2e-ipp-b-24255b-openshift-default-6687484f49-992gq"
Nov 02 15:45:50.587688 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:50.586470    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c19acd3a-5dbf-4da3-b88f-e36284b22243-istio-data\") pod \"e2e-ipp-b-24255b-openshift-default-6687484f49-992gq\" (UID: \"c19acd3a-5dbf-4da3-b88f-e36284b22243\") " pod="openshift-ingress/e2e-ipp-b-24255b-openshift-default-6687484f49-992gq"
Nov 02 15:45:50.587836 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:50.583866    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c19acd3a-5dbf-4da3-b88f-e36284b22243-workload-certs\") pod \"e2e-ipp-b-24255b-openshift-default-6687484f49-992gq\" (UID: \"c19acd3a-5dbf-4da3-b88f-e36284b22243\") " pod="openshift-ingress/e2e-ipp-b-24255b-openshift-default-6687484f49-992gq"
Nov 02 15:45:50.587959 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:50.585946    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c19acd3a-5dbf-4da3-b88f-e36284b22243-istiod-ca-cert\") pod \"e2e-ipp-b-24255b-openshift-default-6687484f49-992gq\" (UID: \"c19acd3a-5dbf-4da3-b88f-e36284b22243\") " pod="openshift-ingress/e2e-ipp-b-24255b-openshift-default-6687484f49-992gq"
Nov 02 15:45:50.588064 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:50.587399    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c19acd3a-5dbf-4da3-b88f-e36284b22243-workload-socket\") pod \"e2e-ipp-b-24255b-openshift-default-6687484f49-992gq\" (UID: \"c19acd3a-5dbf-4da3-b88f-e36284b22243\") " pod="openshift-ingress/e2e-ipp-b-24255b-openshift-default-6687484f49-992gq"
Nov 02 15:45:50.588185 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:50.563473    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c19acd3a-5dbf-4da3-b88f-e36284b22243-credential-socket\") pod \"e2e-ipp-b-24255b-openshift-default-6687484f49-992gq\" (UID: \"c19acd3a-5dbf-4da3-b88f-e36284b22243\") " pod="openshift-ingress/e2e-ipp-b-24255b-openshift-default-6687484f49-992gq"
Nov 02 15:45:50.595484 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:50.595456    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c19acd3a-5dbf-4da3-b88f-e36284b22243-istio-podinfo\") pod \"e2e-ipp-b-24255b-openshift-default-6687484f49-992gq\" (UID: \"c19acd3a-5dbf-4da3-b88f-e36284b22243\") " pod="openshift-ingress/e2e-ipp-b-24255b-openshift-default-6687484f49-992gq"
Nov 02 15:45:50.621018 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:50.620911    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c19acd3a-5dbf-4da3-b88f-e36284b22243-istio-envoy\") pod \"e2e-ipp-b-24255b-openshift-default-6687484f49-992gq\" (UID: \"c19acd3a-5dbf-4da3-b88f-e36284b22243\") " pod="openshift-ingress/e2e-ipp-b-24255b-openshift-default-6687484f49-992gq"
Nov 02 15:45:50.634151 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:50.625460    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cn7fd\" (UniqueName: \"kubernetes.io/projected/c19acd3a-5dbf-4da3-b88f-e36284b22243-kube-api-access-cn7fd\") pod \"e2e-ipp-b-24255b-openshift-default-6687484f49-992gq\" (UID: \"c19acd3a-5dbf-4da3-b88f-e36284b22243\") " pod="openshift-ingress/e2e-ipp-b-24255b-openshift-default-6687484f49-992gq"
Nov 02 15:45:50.634151 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:50.629672    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c19acd3a-5dbf-4da3-b88f-e36284b22243-istio-token\") pod \"e2e-ipp-b-24255b-openshift-default-6687484f49-992gq\" (UID: \"c19acd3a-5dbf-4da3-b88f-e36284b22243\") " pod="openshift-ingress/e2e-ipp-b-24255b-openshift-default-6687484f49-992gq"
Nov 02 15:45:50.720530 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:50.720434    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-24255b-openshift-default-6687484f49-992gq"
Nov 02 15:45:51.156345 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:51.156317    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-24255b-openshift-default-6687484f49-992gq"]
Nov 02 15:45:51.158971 ip-10-0-135-138 kubenswrapper[5304]: W1102 15:45:51.158940    5304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc19acd3a_5dbf_4da3_b88f_e36284b22243.slice/crio-4f8dd28ee9dc312e3de29e0ec68a15de89453d8c7272e98b0882723acd7e26e2 WatchSource:0}: Error finding container 4f8dd28ee9dc312e3de29e0ec68a15de89453d8c7272e98b0882723acd7e26e2: Status 404 returned error can't find the container with id 4f8dd28ee9dc312e3de29e0ec68a15de89453d8c7272e98b0882723acd7e26e2
Nov 02 15:45:51.162566 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:51.162503    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:45:51.162708 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:51.162683    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:45:51.169807 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:51.162790    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:45:51.454215 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:51.454090    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-24255b-openshift-default-6687484f49-992gq" event={"ID":"c19acd3a-5dbf-4da3-b88f-e36284b22243","Type":"ContainerStarted","Data":"2a334a4d6be2f27cefd44840ccc5c33c95658bbdb551ba895da7970f96112d02"}
Nov 02 15:45:51.454725 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:51.454239    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-24255b-openshift-default-6687484f49-992gq" event={"ID":"c19acd3a-5dbf-4da3-b88f-e36284b22243","Type":"ContainerStarted","Data":"4f8dd28ee9dc312e3de29e0ec68a15de89453d8c7272e98b0882723acd7e26e2"}
Nov 02 15:45:51.503347 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:51.503287    5304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-24255b-openshift-default-6687484f49-992gq" podStartSLOduration=1.503272399 podStartE2EDuration="1.503272399s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:45:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:45:51.498781074 +0000 UTC m=+2951.492605483" watchObservedRunningTime="2026-11-02 15:45:51.503272399 +0000 UTC m=+2951.497096808"
Nov 02 15:45:51.721454 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:51.721353    5304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-24255b-openshift-default-6687484f49-992gq"
Nov 02 15:45:52.728409 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:52.728373    5304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-24255b-openshift-default-6687484f49-992gq"
Nov 02 15:45:53.468101 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:53.468070    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-24255b-openshift-default-6687484f49-992gq"
Nov 02 15:45:53.469073 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:53.469049    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-24255b-openshift-default-6687484f49-992gq"
Nov 02 15:45:53.926597 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:53.926569    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-9cfd1f-openshift-default-6fbb7b57d9-hh28t"
Nov 02 15:45:54.072203 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:54.072112    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d665d588-97c3-4797-be57-c426b3d905ff-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d665d588-97c3-4797-be57-c426b3d905ff-istio-podinfo\") pod \"d665d588-97c3-4797-be57-c426b3d905ff\" (UID: \"d665d588-97c3-4797-be57-c426b3d905ff\") "
Nov 02 15:45:54.072379 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:54.072208    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d665d588-97c3-4797-be57-c426b3d905ff-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d665d588-97c3-4797-be57-c426b3d905ff-istio-data\") pod \"d665d588-97c3-4797-be57-c426b3d905ff\" (UID: \"d665d588-97c3-4797-be57-c426b3d905ff\") "
Nov 02 15:45:54.072379 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:54.072263    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d665d588-97c3-4797-be57-c426b3d905ff-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d665d588-97c3-4797-be57-c426b3d905ff-workload-certs\") pod \"d665d588-97c3-4797-be57-c426b3d905ff\" (UID: \"d665d588-97c3-4797-be57-c426b3d905ff\") "
Nov 02 15:45:54.072542 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:54.072443    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d665d588-97c3-4797-be57-c426b3d905ff-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d665d588-97c3-4797-be57-c426b3d905ff-istiod-ca-cert\") pod \"d665d588-97c3-4797-be57-c426b3d905ff\" (UID: \"d665d588-97c3-4797-be57-c426b3d905ff\") "
Nov 02 15:45:54.072542 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:54.072483    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d665d588-97c3-4797-be57-c426b3d905ff-kube-api-access-dj54j\" (UniqueName: \"kubernetes.io/projected/d665d588-97c3-4797-be57-c426b3d905ff-kube-api-access-dj54j\") pod \"d665d588-97c3-4797-be57-c426b3d905ff\" (UID: \"d665d588-97c3-4797-be57-c426b3d905ff\") "
Nov 02 15:45:54.072542 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:54.072518    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d665d588-97c3-4797-be57-c426b3d905ff-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d665d588-97c3-4797-be57-c426b3d905ff-credential-socket\") pod \"d665d588-97c3-4797-be57-c426b3d905ff\" (UID: \"d665d588-97c3-4797-be57-c426b3d905ff\") "
Nov 02 15:45:54.072721 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:54.072546    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d665d588-97c3-4797-be57-c426b3d905ff-istio-data" pod "d665d588-97c3-4797-be57-c426b3d905ff" (UID: "d665d588-97c3-4797-be57-c426b3d905ff"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:45:54.072721 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:54.072650    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d665d588-97c3-4797-be57-c426b3d905ff-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d665d588-97c3-4797-be57-c426b3d905ff-istio-envoy\") pod \"d665d588-97c3-4797-be57-c426b3d905ff\" (UID: \"d665d588-97c3-4797-be57-c426b3d905ff\") "
Nov 02 15:45:54.072721 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:54.072685    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d665d588-97c3-4797-be57-c426b3d905ff-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d665d588-97c3-4797-be57-c426b3d905ff-workload-socket\") pod \"d665d588-97c3-4797-be57-c426b3d905ff\" (UID: \"d665d588-97c3-4797-be57-c426b3d905ff\") "
Nov 02 15:45:54.072893 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:54.072712    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d665d588-97c3-4797-be57-c426b3d905ff-workload-certs" pod "d665d588-97c3-4797-be57-c426b3d905ff" (UID: "d665d588-97c3-4797-be57-c426b3d905ff"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:45:54.072893 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:54.072741    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d665d588-97c3-4797-be57-c426b3d905ff-istio-token\" (UniqueName: \"kubernetes.io/projected/d665d588-97c3-4797-be57-c426b3d905ff-istio-token\") pod \"d665d588-97c3-4797-be57-c426b3d905ff\" (UID: \"d665d588-97c3-4797-be57-c426b3d905ff\") "
Nov 02 15:45:54.073020 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:54.072908    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d665d588-97c3-4797-be57-c426b3d905ff-credential-socket" pod "d665d588-97c3-4797-be57-c426b3d905ff" (UID: "d665d588-97c3-4797-be57-c426b3d905ff"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:45:54.073020 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:54.072958    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d665d588-97c3-4797-be57-c426b3d905ff-istiod-ca-cert" pod "d665d588-97c3-4797-be57-c426b3d905ff" (UID: "d665d588-97c3-4797-be57-c426b3d905ff"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:45:54.073177 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:54.073139    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d665d588-97c3-4797-be57-c426b3d905ff-workload-socket" pod "d665d588-97c3-4797-be57-c426b3d905ff" (UID: "d665d588-97c3-4797-be57-c426b3d905ff"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:45:54.073426 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:54.073406    5304 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d665d588-97c3-4797-be57-c426b3d905ff-istio-data\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:45:54.073511 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:54.073432    5304 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d665d588-97c3-4797-be57-c426b3d905ff-workload-certs\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:45:54.073511 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:54.073447    5304 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d665d588-97c3-4797-be57-c426b3d905ff-istiod-ca-cert\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:45:54.073511 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:54.073461    5304 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d665d588-97c3-4797-be57-c426b3d905ff-credential-socket\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:45:54.073511 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:54.073476    5304 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d665d588-97c3-4797-be57-c426b3d905ff-workload-socket\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:45:54.074995 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:54.074964    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d665d588-97c3-4797-be57-c426b3d905ff-istio-podinfo" pod "d665d588-97c3-4797-be57-c426b3d905ff" (UID: "d665d588-97c3-4797-be57-c426b3d905ff"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 15:45:54.075120 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:54.075012    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d665d588-97c3-4797-be57-c426b3d905ff-kube-api-access-dj54j" pod "d665d588-97c3-4797-be57-c426b3d905ff" (UID: "d665d588-97c3-4797-be57-c426b3d905ff"). InnerVolumeSpecName "kube-api-access-dj54j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:45:54.075120 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:54.075064    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d665d588-97c3-4797-be57-c426b3d905ff-istio-envoy" pod "d665d588-97c3-4797-be57-c426b3d905ff" (UID: "d665d588-97c3-4797-be57-c426b3d905ff"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:45:54.075253 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:54.075194    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d665d588-97c3-4797-be57-c426b3d905ff-istio-token" pod "d665d588-97c3-4797-be57-c426b3d905ff" (UID: "d665d588-97c3-4797-be57-c426b3d905ff"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:45:54.174999 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:54.174959    5304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dj54j\" (UniqueName: \"kubernetes.io/projected/d665d588-97c3-4797-be57-c426b3d905ff-kube-api-access-dj54j\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:45:54.174999 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:54.174994    5304 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d665d588-97c3-4797-be57-c426b3d905ff-istio-envoy\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:45:54.174999 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:54.175004    5304 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d665d588-97c3-4797-be57-c426b3d905ff-istio-token\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:45:54.174999 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:54.175013    5304 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d665d588-97c3-4797-be57-c426b3d905ff-istio-podinfo\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:45:54.476766 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:54.476729    5304 generic.go:356] "Generic (PLEG): container finished" podID="d665d588-97c3-4797-be57-c426b3d905ff" containerID="c775141644658fa89c11450fa81952fd56da0b264110c7fc3655b74f6763e0dc" exitCode=0
Nov 02 15:45:54.476956 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:54.476805    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-9cfd1f-openshift-default-6fbb7b57d9-hh28t"
Nov 02 15:45:54.476956 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:54.476813    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-9cfd1f-openshift-default-6fbb7b57d9-hh28t" event={"ID":"d665d588-97c3-4797-be57-c426b3d905ff","Type":"ContainerDied","Data":"c775141644658fa89c11450fa81952fd56da0b264110c7fc3655b74f6763e0dc"}
Nov 02 15:45:54.476956 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:54.476869    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-9cfd1f-openshift-default-6fbb7b57d9-hh28t" event={"ID":"d665d588-97c3-4797-be57-c426b3d905ff","Type":"ContainerDied","Data":"352deb3e37dd4d1391328f659027d810c3d22db470f34c40a7fdced015cac2d3"}
Nov 02 15:45:54.476956 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:54.476890    5304 scope.go:122] "RemoveContainer" containerID="c775141644658fa89c11450fa81952fd56da0b264110c7fc3655b74f6763e0dc"
Nov 02 15:45:54.488267 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:54.488248    5304 scope.go:122] "RemoveContainer" containerID="844d78510fb78a43c3c7c69d7bafedfd7f55e584f99d15174806ac946b6c56a3"
Nov 02 15:45:54.498437 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:54.498417    5304 scope.go:122] "RemoveContainer" containerID="c775141644658fa89c11450fa81952fd56da0b264110c7fc3655b74f6763e0dc"
Nov 02 15:45:54.498721 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:45:54.498699    5304 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c775141644658fa89c11450fa81952fd56da0b264110c7fc3655b74f6763e0dc\": container with ID starting with c775141644658fa89c11450fa81952fd56da0b264110c7fc3655b74f6763e0dc not found: ID does not exist" containerID="c775141644658fa89c11450fa81952fd56da0b264110c7fc3655b74f6763e0dc"
Nov 02 15:45:54.498821 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:54.498729    5304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c775141644658fa89c11450fa81952fd56da0b264110c7fc3655b74f6763e0dc"} err="failed to get container status \"c775141644658fa89c11450fa81952fd56da0b264110c7fc3655b74f6763e0dc\": rpc error: code = NotFound desc = could not find container \"c775141644658fa89c11450fa81952fd56da0b264110c7fc3655b74f6763e0dc\": container with ID starting with c775141644658fa89c11450fa81952fd56da0b264110c7fc3655b74f6763e0dc not found: ID does not exist"
Nov 02 15:45:54.498821 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:54.498748    5304 scope.go:122] "RemoveContainer" containerID="844d78510fb78a43c3c7c69d7bafedfd7f55e584f99d15174806ac946b6c56a3"
Nov 02 15:45:54.499011 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:45:54.498980    5304 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"844d78510fb78a43c3c7c69d7bafedfd7f55e584f99d15174806ac946b6c56a3\": container with ID starting with 844d78510fb78a43c3c7c69d7bafedfd7f55e584f99d15174806ac946b6c56a3 not found: ID does not exist" containerID="844d78510fb78a43c3c7c69d7bafedfd7f55e584f99d15174806ac946b6c56a3"
Nov 02 15:45:54.499087 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:54.499015    5304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"844d78510fb78a43c3c7c69d7bafedfd7f55e584f99d15174806ac946b6c56a3"} err="failed to get container status \"844d78510fb78a43c3c7c69d7bafedfd7f55e584f99d15174806ac946b6c56a3\": rpc error: code = NotFound desc = could not find container \"844d78510fb78a43c3c7c69d7bafedfd7f55e584f99d15174806ac946b6c56a3\": container with ID starting with 844d78510fb78a43c3c7c69d7bafedfd7f55e584f99d15174806ac946b6c56a3 not found: ID does not exist"
Nov 02 15:45:54.530911 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:54.530881    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-9cfd1f-openshift-default-6fbb7b57d9-hh28t"]
Nov 02 15:45:54.535777 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:54.535752    5304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-9cfd1f-openshift-default-6fbb7b57d9-hh28t"]
Nov 02 15:45:54.801900 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:45:54.801818    5304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k6fjk" containerName="extractor"
Nov 02 15:45:56.198041 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:56.198001    5304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d665d588-97c3-4797-be57-c426b3d905ff" path="/var/lib/kubelet/pods/d665d588-97c3-4797-be57-c426b3d905ff/volumes"
Nov 02 15:45:58.182793 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:58.182756    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f84ad8-64f97947fb-zppnf"]
Nov 02 15:45:58.183346 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:58.182980    5304 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-f84ad8-64f97947fb-zppnf" podUID="5386e95b-8eef-49a6-a84e-80c8119c27fa" containerName="payload-processing" containerID="cri-o://25f8e82eec34ba10a101448d30de932c80726a73af3327c7b2d2f39069f290f3" gracePeriod=30
Nov 02 15:45:58.218731 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:58.218695    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f84ad8-789796656c-g97z7"]
Nov 02 15:45:58.219005 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:58.218979    5304 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f84ad8-789796656c-g97z7" podUID="fc34fbb9-6804-4ad8-a007-25a6551a0960" containerName="payload-pre-processing" containerID="cri-o://eabbaf604a20bcbdc58b08d2331287047ee01c0e0f8c516efdb2c97625677a87" gracePeriod=30
Nov 02 15:45:58.598151 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:58.591703    5304 generic.go:356] "Generic (PLEG): container finished" podID="fc34fbb9-6804-4ad8-a007-25a6551a0960" containerID="eabbaf604a20bcbdc58b08d2331287047ee01c0e0f8c516efdb2c97625677a87" exitCode=0
Nov 02 15:45:58.598151 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:58.591854    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f84ad8-789796656c-g97z7" event={"ID":"fc34fbb9-6804-4ad8-a007-25a6551a0960","Type":"ContainerDied","Data":"eabbaf604a20bcbdc58b08d2331287047ee01c0e0f8c516efdb2c97625677a87"}
Nov 02 15:45:58.604148 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:58.601921    5304 generic.go:356] "Generic (PLEG): container finished" podID="5386e95b-8eef-49a6-a84e-80c8119c27fa" containerID="25f8e82eec34ba10a101448d30de932c80726a73af3327c7b2d2f39069f290f3" exitCode=0
Nov 02 15:45:58.604148 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:58.602187    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-f84ad8-64f97947fb-zppnf" event={"ID":"5386e95b-8eef-49a6-a84e-80c8119c27fa","Type":"ContainerDied","Data":"25f8e82eec34ba10a101448d30de932c80726a73af3327c7b2d2f39069f290f3"}
Nov 02 15:45:58.637204 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:58.628303    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-f84ad8-64f97947fb-zppnf"
Nov 02 15:45:58.757672 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:58.757582    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5386e95b-8eef-49a6-a84e-80c8119c27fa-kube-api-access-jwstw\" (UniqueName: \"kubernetes.io/projected/5386e95b-8eef-49a6-a84e-80c8119c27fa-kube-api-access-jwstw\") pod \"5386e95b-8eef-49a6-a84e-80c8119c27fa\" (UID: \"5386e95b-8eef-49a6-a84e-80c8119c27fa\") "
Nov 02 15:45:58.758026 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:58.757951    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5386e95b-8eef-49a6-a84e-80c8119c27fa-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5386e95b-8eef-49a6-a84e-80c8119c27fa-plugins-config-volume\") pod \"5386e95b-8eef-49a6-a84e-80c8119c27fa\" (UID: \"5386e95b-8eef-49a6-a84e-80c8119c27fa\") "
Nov 02 15:45:58.765022 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:58.764976    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5386e95b-8eef-49a6-a84e-80c8119c27fa-plugins-config-volume" pod "5386e95b-8eef-49a6-a84e-80c8119c27fa" (UID: "5386e95b-8eef-49a6-a84e-80c8119c27fa"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:45:58.775201 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:58.774319    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5386e95b-8eef-49a6-a84e-80c8119c27fa-kube-api-access-jwstw" pod "5386e95b-8eef-49a6-a84e-80c8119c27fa" (UID: "5386e95b-8eef-49a6-a84e-80c8119c27fa"). InnerVolumeSpecName "kube-api-access-jwstw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:45:58.833162 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:58.830957    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f84ad8-nn4jm"]
Nov 02 15:45:58.836073 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:58.835272    5304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f84ad8-nn4jm"]
Nov 02 15:45:58.846651 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:58.846623    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f84ad8-789796656c-g97z7"
Nov 02 15:45:58.860721 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:58.860693    5304 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5386e95b-8eef-49a6-a84e-80c8119c27fa-plugins-config-volume\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:45:58.860721 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:58.860717    5304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jwstw\" (UniqueName: \"kubernetes.io/projected/5386e95b-8eef-49a6-a84e-80c8119c27fa-kube-api-access-jwstw\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:45:58.962112 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:58.962081    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc34fbb9-6804-4ad8-a007-25a6551a0960-kube-api-access-l5sh9\" (UniqueName: \"kubernetes.io/projected/fc34fbb9-6804-4ad8-a007-25a6551a0960-kube-api-access-l5sh9\") pod \"fc34fbb9-6804-4ad8-a007-25a6551a0960\" (UID: \"fc34fbb9-6804-4ad8-a007-25a6551a0960\") "
Nov 02 15:45:58.962308 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:58.962247    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fc34fbb9-6804-4ad8-a007-25a6551a0960-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc34fbb9-6804-4ad8-a007-25a6551a0960-plugins-config-volume\") pod \"fc34fbb9-6804-4ad8-a007-25a6551a0960\" (UID: \"fc34fbb9-6804-4ad8-a007-25a6551a0960\") "
Nov 02 15:45:58.962711 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:58.962677    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc34fbb9-6804-4ad8-a007-25a6551a0960-plugins-config-volume" pod "fc34fbb9-6804-4ad8-a007-25a6551a0960" (UID: "fc34fbb9-6804-4ad8-a007-25a6551a0960"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:45:58.962915 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:58.962891    5304 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc34fbb9-6804-4ad8-a007-25a6551a0960-plugins-config-volume\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:45:58.965066 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:58.965040    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc34fbb9-6804-4ad8-a007-25a6551a0960-kube-api-access-l5sh9" pod "fc34fbb9-6804-4ad8-a007-25a6551a0960" (UID: "fc34fbb9-6804-4ad8-a007-25a6551a0960"). InnerVolumeSpecName "kube-api-access-l5sh9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:45:59.064835 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:59.064749    5304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l5sh9\" (UniqueName: \"kubernetes.io/projected/fc34fbb9-6804-4ad8-a007-25a6551a0960-kube-api-access-l5sh9\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:45:59.610424 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:59.610393    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f84ad8-789796656c-g97z7"
Nov 02 15:45:59.610914 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:59.610433    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f84ad8-789796656c-g97z7" event={"ID":"fc34fbb9-6804-4ad8-a007-25a6551a0960","Type":"ContainerDied","Data":"1b7f6022b164e933c06198063b4a6841724dd32514f10008831710d42db8ae98"}
Nov 02 15:45:59.610914 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:59.610501    5304 scope.go:122] "RemoveContainer" containerID="eabbaf604a20bcbdc58b08d2331287047ee01c0e0f8c516efdb2c97625677a87"
Nov 02 15:45:59.612097 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:59.611906    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-f84ad8-64f97947fb-zppnf"
Nov 02 15:45:59.612097 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:59.611923    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-f84ad8-64f97947fb-zppnf" event={"ID":"5386e95b-8eef-49a6-a84e-80c8119c27fa","Type":"ContainerDied","Data":"34e392631a5f0fda92fbb33f52c4bba19b7a2199ad34d80dd7ccd6f992dc3862"}
Nov 02 15:45:59.623433 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:59.623415    5304 scope.go:122] "RemoveContainer" containerID="25f8e82eec34ba10a101448d30de932c80726a73af3327c7b2d2f39069f290f3"
Nov 02 15:45:59.674480 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:59.674447    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f84ad8-64f97947fb-zppnf"]
Nov 02 15:45:59.686378 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:59.686343    5304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f84ad8-64f97947fb-zppnf"]
Nov 02 15:45:59.728026 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:59.727984    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f84ad8-789796656c-g97z7"]
Nov 02 15:45:59.732945 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:45:59.732916    5304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f84ad8-789796656c-g97z7"]
Nov 02 15:46:00.122056 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:00.122023    5304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5386e95b-8eef-49a6-a84e-80c8119c27fa" path="/var/lib/kubelet/pods/5386e95b-8eef-49a6-a84e-80c8119c27fa/volumes"
Nov 02 15:46:00.122618 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:00.122601    5304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="97f24cb0-604f-4969-ba8b-90dd2373c61a" path="/var/lib/kubelet/pods/97f24cb0-604f-4969-ba8b-90dd2373c61a/volumes"
Nov 02 15:46:00.123028 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:00.123014    5304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fc34fbb9-6804-4ad8-a007-25a6551a0960" path="/var/lib/kubelet/pods/fc34fbb9-6804-4ad8-a007-25a6551a0960/volumes"
Nov 02 15:46:04.419247 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:04.419205    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-3805b8-6c6d4cd55b-fbstm"]
Nov 02 15:46:04.419747 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:04.419431    5304 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-3805b8-6c6d4cd55b-fbstm" podUID="53a0141a-7572-475e-adac-56ed64e21bcc" containerName="payload-processing" containerID="cri-o://dfe92a495a2297d5e256eb7e3f4ff821895bca20955e66434b3ecd2582b4ceba" gracePeriod=30
Nov 02 15:46:04.445240 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:04.445206    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-3805b8-68df779859-spzxb"]
Nov 02 15:46:04.445481 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:04.445452    5304 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3805b8-68df779859-spzxb" podUID="a7a7782c-d47c-4ee8-b352-9fe2c6b2ee45" containerName="payload-pre-processing" containerID="cri-o://361cad4c2af48e0b49b06a72fe2c4445417d03263acf67aefd55ec05c54f04d3" gracePeriod=30
Nov 02 15:46:04.862221 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:04.853385    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-3805b8-6c6d4cd55b-fbstm"
Nov 02 15:46:04.871163 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:04.863970    5304 generic.go:356] "Generic (PLEG): container finished" podID="a7a7782c-d47c-4ee8-b352-9fe2c6b2ee45" containerID="361cad4c2af48e0b49b06a72fe2c4445417d03263acf67aefd55ec05c54f04d3" exitCode=0
Nov 02 15:46:04.871163 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:04.864260    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3805b8-68df779859-spzxb" event={"ID":"a7a7782c-d47c-4ee8-b352-9fe2c6b2ee45","Type":"ContainerDied","Data":"361cad4c2af48e0b49b06a72fe2c4445417d03263acf67aefd55ec05c54f04d3"}
Nov 02 15:46:04.871163 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:04.866262    5304 generic.go:356] "Generic (PLEG): container finished" podID="53a0141a-7572-475e-adac-56ed64e21bcc" containerID="dfe92a495a2297d5e256eb7e3f4ff821895bca20955e66434b3ecd2582b4ceba" exitCode=0
Nov 02 15:46:04.871163 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:04.866358    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-3805b8-6c6d4cd55b-fbstm" event={"ID":"53a0141a-7572-475e-adac-56ed64e21bcc","Type":"ContainerDied","Data":"dfe92a495a2297d5e256eb7e3f4ff821895bca20955e66434b3ecd2582b4ceba"}
Nov 02 15:46:04.871163 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:04.866426    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-3805b8-6c6d4cd55b-fbstm" event={"ID":"53a0141a-7572-475e-adac-56ed64e21bcc","Type":"ContainerDied","Data":"3d86a46009c32d20d7f4edef81f2cffd73ca12545c906ecd12f51b1878fa4ee3"}
Nov 02 15:46:04.871163 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:04.866470    5304 scope.go:122] "RemoveContainer" containerID="dfe92a495a2297d5e256eb7e3f4ff821895bca20955e66434b3ecd2582b4ceba"
Nov 02 15:46:04.885301 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:04.874979    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-3805b8-6c6d4cd55b-fbstm"
Nov 02 15:46:05.196169 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:05.175864    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/53a0141a-7572-475e-adac-56ed64e21bcc-kube-api-access-xxqh5\" (UniqueName: \"kubernetes.io/projected/53a0141a-7572-475e-adac-56ed64e21bcc-kube-api-access-xxqh5\") pod \"53a0141a-7572-475e-adac-56ed64e21bcc\" (UID: \"53a0141a-7572-475e-adac-56ed64e21bcc\") "
Nov 02 15:46:05.196169 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:05.175949    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/53a0141a-7572-475e-adac-56ed64e21bcc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53a0141a-7572-475e-adac-56ed64e21bcc-plugins-config-volume\") pod \"53a0141a-7572-475e-adac-56ed64e21bcc\" (UID: \"53a0141a-7572-475e-adac-56ed64e21bcc\") "
Nov 02 15:46:05.197537 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:05.196913    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53a0141a-7572-475e-adac-56ed64e21bcc-plugins-config-volume" pod "53a0141a-7572-475e-adac-56ed64e21bcc" (UID: "53a0141a-7572-475e-adac-56ed64e21bcc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:46:05.215009 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:05.214979    5304 scope.go:122] "RemoveContainer" containerID="dfe92a495a2297d5e256eb7e3f4ff821895bca20955e66434b3ecd2582b4ceba"
Nov 02 15:46:05.215414 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:46:05.215376    5304 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dfe92a495a2297d5e256eb7e3f4ff821895bca20955e66434b3ecd2582b4ceba\": container with ID starting with dfe92a495a2297d5e256eb7e3f4ff821895bca20955e66434b3ecd2582b4ceba not found: ID does not exist" containerID="dfe92a495a2297d5e256eb7e3f4ff821895bca20955e66434b3ecd2582b4ceba"
Nov 02 15:46:05.215531 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:05.215426    5304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dfe92a495a2297d5e256eb7e3f4ff821895bca20955e66434b3ecd2582b4ceba"} err="failed to get container status \"dfe92a495a2297d5e256eb7e3f4ff821895bca20955e66434b3ecd2582b4ceba\": rpc error: code = NotFound desc = could not find container \"dfe92a495a2297d5e256eb7e3f4ff821895bca20955e66434b3ecd2582b4ceba\": container with ID starting with dfe92a495a2297d5e256eb7e3f4ff821895bca20955e66434b3ecd2582b4ceba not found: ID does not exist"
Nov 02 15:46:05.220791 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:05.220723    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53a0141a-7572-475e-adac-56ed64e21bcc-kube-api-access-xxqh5" pod "53a0141a-7572-475e-adac-56ed64e21bcc" (UID: "53a0141a-7572-475e-adac-56ed64e21bcc"). InnerVolumeSpecName "kube-api-access-xxqh5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:46:05.231885 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:05.231863    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3805b8-68df779859-spzxb"
Nov 02 15:46:05.279481 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:05.279431    5304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xxqh5\" (UniqueName: \"kubernetes.io/projected/53a0141a-7572-475e-adac-56ed64e21bcc-kube-api-access-xxqh5\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:46:05.279759 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:05.279738    5304 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53a0141a-7572-475e-adac-56ed64e21bcc-plugins-config-volume\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:46:05.381006 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:05.380973    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a7a7782c-d47c-4ee8-b352-9fe2c6b2ee45-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7a7782c-d47c-4ee8-b352-9fe2c6b2ee45-plugins-config-volume\") pod \"a7a7782c-d47c-4ee8-b352-9fe2c6b2ee45\" (UID: \"a7a7782c-d47c-4ee8-b352-9fe2c6b2ee45\") "
Nov 02 15:46:05.381264 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:05.381247    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a7a7782c-d47c-4ee8-b352-9fe2c6b2ee45-kube-api-access-tk9mf\" (UniqueName: \"kubernetes.io/projected/a7a7782c-d47c-4ee8-b352-9fe2c6b2ee45-kube-api-access-tk9mf\") pod \"a7a7782c-d47c-4ee8-b352-9fe2c6b2ee45\" (UID: \"a7a7782c-d47c-4ee8-b352-9fe2c6b2ee45\") "
Nov 02 15:46:05.381427 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:05.381403    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7a7782c-d47c-4ee8-b352-9fe2c6b2ee45-plugins-config-volume" pod "a7a7782c-d47c-4ee8-b352-9fe2c6b2ee45" (UID: "a7a7782c-d47c-4ee8-b352-9fe2c6b2ee45"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:46:05.381904 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:05.381884    5304 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7a7782c-d47c-4ee8-b352-9fe2c6b2ee45-plugins-config-volume\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:46:05.383336 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:05.383316    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7a7782c-d47c-4ee8-b352-9fe2c6b2ee45-kube-api-access-tk9mf" pod "a7a7782c-d47c-4ee8-b352-9fe2c6b2ee45" (UID: "a7a7782c-d47c-4ee8-b352-9fe2c6b2ee45"). InnerVolumeSpecName "kube-api-access-tk9mf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:46:05.507823 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:05.507734    5304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tk9mf\" (UniqueName: \"kubernetes.io/projected/a7a7782c-d47c-4ee8-b352-9fe2c6b2ee45-kube-api-access-tk9mf\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:46:05.739515 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:05.739477    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-3805b8-6c6d4cd55b-fbstm"]
Nov 02 15:46:05.748733 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:05.748697    5304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-3805b8-6c6d4cd55b-fbstm"]
Nov 02 15:46:05.874896 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:05.874868    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3805b8-68df779859-spzxb"
Nov 02 15:46:05.874896 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:05.874881    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3805b8-68df779859-spzxb" event={"ID":"a7a7782c-d47c-4ee8-b352-9fe2c6b2ee45","Type":"ContainerDied","Data":"dbb82366148a06cdaac0b5e218638357c284a2405ec94134d3eb6a41acb1da2a"}
Nov 02 15:46:05.875187 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:05.874967    5304 scope.go:122] "RemoveContainer" containerID="361cad4c2af48e0b49b06a72fe2c4445417d03263acf67aefd55ec05c54f04d3"
Nov 02 15:46:05.994212 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:05.994176    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-3805b8-68df779859-spzxb"]
Nov 02 15:46:06.070222 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:06.050518    5304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-3805b8-68df779859-spzxb"]
Nov 02 15:46:06.126260 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:06.126172    5304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="53a0141a-7572-475e-adac-56ed64e21bcc" path="/var/lib/kubelet/pods/53a0141a-7572-475e-adac-56ed64e21bcc/volumes"
Nov 02 15:46:06.126829 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:06.126812    5304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a7a7782c-d47c-4ee8-b352-9fe2c6b2ee45" path="/var/lib/kubelet/pods/a7a7782c-d47c-4ee8-b352-9fe2c6b2ee45/volumes"
Nov 02 15:46:07.030651 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:07.030619    5304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-f0039c-openshift-default-5f656fc965-rjf2w"]
Nov 02 15:46:07.031236 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:07.031221    5304 cpu_manager.go:414] "containerMap: removing container" podUID="a7a7782c-d47c-4ee8-b352-9fe2c6b2ee45" containerName="payload-pre-processing"
Nov 02 15:46:07.031290 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:07.031237    5304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7a7782c-d47c-4ee8-b352-9fe2c6b2ee45" containerName="payload-pre-processing"
Nov 02 15:46:07.031290 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:07.031248    5304 cpu_manager.go:414] "containerMap: removing container" podUID="d665d588-97c3-4797-be57-c426b3d905ff" containerName="istio-proxy"
Nov 02 15:46:07.031290 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:07.031253    5304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d665d588-97c3-4797-be57-c426b3d905ff" containerName="istio-proxy"
Nov 02 15:46:07.031290 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:07.031267    5304 cpu_manager.go:414] "containerMap: removing container" podUID="647ad5be-64be-4aff-bff6-d0ac2943671c" containerName="cleanup"
Nov 02 15:46:07.031290 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:07.031272    5304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="647ad5be-64be-4aff-bff6-d0ac2943671c" containerName="cleanup"
Nov 02 15:46:07.031290 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:07.031282    5304 cpu_manager.go:414] "containerMap: removing container" podUID="5386e95b-8eef-49a6-a84e-80c8119c27fa" containerName="payload-processing"
Nov 02 15:46:07.031290 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:07.031286    5304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5386e95b-8eef-49a6-a84e-80c8119c27fa" containerName="payload-processing"
Nov 02 15:46:07.031511 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:07.031296    5304 cpu_manager.go:414] "containerMap: removing container" podUID="d665d588-97c3-4797-be57-c426b3d905ff" containerName="istio-proxy"
Nov 02 15:46:07.031511 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:07.031301    5304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d665d588-97c3-4797-be57-c426b3d905ff" containerName="istio-proxy"
Nov 02 15:46:07.031511 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:07.031308    5304 cpu_manager.go:414] "containerMap: removing container" podUID="ea2c4ec8-2ace-49dd-a172-99ced21dbc4f" containerName="cleanup"
Nov 02 15:46:07.031511 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:07.031312    5304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea2c4ec8-2ace-49dd-a172-99ced21dbc4f" containerName="cleanup"
Nov 02 15:46:07.031511 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:07.031322    5304 cpu_manager.go:414] "containerMap: removing container" podUID="8ea83c17-d3cb-4d7e-a6ef-cbd731415216" containerName="cleanup"
Nov 02 15:46:07.031511 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:07.031326    5304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ea83c17-d3cb-4d7e-a6ef-cbd731415216" containerName="cleanup"
Nov 02 15:46:07.031511 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:07.031339    5304 cpu_manager.go:414] "containerMap: removing container" podUID="53a0141a-7572-475e-adac-56ed64e21bcc" containerName="payload-processing"
Nov 02 15:46:07.031511 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:07.031344    5304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53a0141a-7572-475e-adac-56ed64e21bcc" containerName="payload-processing"
Nov 02 15:46:07.031511 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:07.031354    5304 cpu_manager.go:414] "containerMap: removing container" podUID="fc34fbb9-6804-4ad8-a007-25a6551a0960" containerName="payload-pre-processing"
Nov 02 15:46:07.031511 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:07.031359    5304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc34fbb9-6804-4ad8-a007-25a6551a0960" containerName="payload-pre-processing"
Nov 02 15:46:07.031511 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:07.031464    5304 memory_manager.go:365] "RemoveStaleState removing state" podUID="d665d588-97c3-4797-be57-c426b3d905ff" containerName="istio-proxy"
Nov 02 15:46:07.031511 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:07.031473    5304 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea2c4ec8-2ace-49dd-a172-99ced21dbc4f" containerName="cleanup"
Nov 02 15:46:07.031511 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:07.031483    5304 memory_manager.go:365] "RemoveStaleState removing state" podUID="53a0141a-7572-475e-adac-56ed64e21bcc" containerName="payload-processing"
Nov 02 15:46:07.031511 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:07.031489    5304 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc34fbb9-6804-4ad8-a007-25a6551a0960" containerName="payload-pre-processing"
Nov 02 15:46:07.031511 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:07.031496    5304 memory_manager.go:365] "RemoveStaleState removing state" podUID="9846a62a-c4aa-4937-b1dd-2d9a342dd72a" containerName="cleanup"
Nov 02 15:46:07.031511 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:07.031505    5304 memory_manager.go:365] "RemoveStaleState removing state" podUID="a07ca06a-c083-4fe9-bca4-1614646e80fb" containerName="cleanup"
Nov 02 15:46:07.031511 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:07.031511    5304 memory_manager.go:365] "RemoveStaleState removing state" podUID="a7a7782c-d47c-4ee8-b352-9fe2c6b2ee45" containerName="payload-pre-processing"
Nov 02 15:46:07.031511 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:07.031517    5304 memory_manager.go:365] "RemoveStaleState removing state" podUID="ada2b1b0-e598-4d38-b995-becab65c7e82" containerName="cleanup"
Nov 02 15:46:07.032092 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:07.031525    5304 memory_manager.go:365] "RemoveStaleState removing state" podUID="5386e95b-8eef-49a6-a84e-80c8119c27fa" containerName="payload-processing"
Nov 02 15:46:07.036635 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:07.036605    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-f0039c-openshift-default-5f656fc965-rjf2w"
Nov 02 15:46:07.056569 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:07.056545    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9178b9f8-6fda-40a9-8850-fa0097f2d39a-workload-certs\") pod \"e2e-ipp-a-f0039c-openshift-default-5f656fc965-rjf2w\" (UID: \"9178b9f8-6fda-40a9-8850-fa0097f2d39a\") " pod="openshift-ingress/e2e-ipp-a-f0039c-openshift-default-5f656fc965-rjf2w"
Nov 02 15:46:07.056704 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:07.056594    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9178b9f8-6fda-40a9-8850-fa0097f2d39a-istio-data\") pod \"e2e-ipp-a-f0039c-openshift-default-5f656fc965-rjf2w\" (UID: \"9178b9f8-6fda-40a9-8850-fa0097f2d39a\") " pod="openshift-ingress/e2e-ipp-a-f0039c-openshift-default-5f656fc965-rjf2w"
Nov 02 15:46:07.056752 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:07.056695    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9178b9f8-6fda-40a9-8850-fa0097f2d39a-istio-podinfo\") pod \"e2e-ipp-a-f0039c-openshift-default-5f656fc965-rjf2w\" (UID: \"9178b9f8-6fda-40a9-8850-fa0097f2d39a\") " pod="openshift-ingress/e2e-ipp-a-f0039c-openshift-default-5f656fc965-rjf2w"
Nov 02 15:46:07.056752 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:07.056738    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9178b9f8-6fda-40a9-8850-fa0097f2d39a-istio-token\") pod \"e2e-ipp-a-f0039c-openshift-default-5f656fc965-rjf2w\" (UID: \"9178b9f8-6fda-40a9-8850-fa0097f2d39a\") " pod="openshift-ingress/e2e-ipp-a-f0039c-openshift-default-5f656fc965-rjf2w"
Nov 02 15:46:07.056832 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:07.056813    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9178b9f8-6fda-40a9-8850-fa0097f2d39a-workload-socket\") pod \"e2e-ipp-a-f0039c-openshift-default-5f656fc965-rjf2w\" (UID: \"9178b9f8-6fda-40a9-8850-fa0097f2d39a\") " pod="openshift-ingress/e2e-ipp-a-f0039c-openshift-default-5f656fc965-rjf2w"
Nov 02 15:46:07.056870 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:07.056843    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zxl5t\" (UniqueName: \"kubernetes.io/projected/9178b9f8-6fda-40a9-8850-fa0097f2d39a-kube-api-access-zxl5t\") pod \"e2e-ipp-a-f0039c-openshift-default-5f656fc965-rjf2w\" (UID: \"9178b9f8-6fda-40a9-8850-fa0097f2d39a\") " pod="openshift-ingress/e2e-ipp-a-f0039c-openshift-default-5f656fc965-rjf2w"
Nov 02 15:46:07.056970 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:07.056955    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9178b9f8-6fda-40a9-8850-fa0097f2d39a-istio-envoy\") pod \"e2e-ipp-a-f0039c-openshift-default-5f656fc965-rjf2w\" (UID: \"9178b9f8-6fda-40a9-8850-fa0097f2d39a\") " pod="openshift-ingress/e2e-ipp-a-f0039c-openshift-default-5f656fc965-rjf2w"
Nov 02 15:46:07.057017 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:07.057006    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9178b9f8-6fda-40a9-8850-fa0097f2d39a-istiod-ca-cert\") pod \"e2e-ipp-a-f0039c-openshift-default-5f656fc965-rjf2w\" (UID: \"9178b9f8-6fda-40a9-8850-fa0097f2d39a\") " pod="openshift-ingress/e2e-ipp-a-f0039c-openshift-default-5f656fc965-rjf2w"
Nov 02 15:46:07.057054 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:07.057042    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9178b9f8-6fda-40a9-8850-fa0097f2d39a-credential-socket\") pod \"e2e-ipp-a-f0039c-openshift-default-5f656fc965-rjf2w\" (UID: \"9178b9f8-6fda-40a9-8850-fa0097f2d39a\") " pod="openshift-ingress/e2e-ipp-a-f0039c-openshift-default-5f656fc965-rjf2w"
Nov 02 15:46:07.058377 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:07.058358    5304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-f0039c-openshift-default-dockercfg-gsgsb\""
Nov 02 15:46:07.073538 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:07.073506    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-f0039c-openshift-default-5f656fc965-rjf2w"]
Nov 02 15:46:07.158400 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:07.158360    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9178b9f8-6fda-40a9-8850-fa0097f2d39a-istio-envoy\") pod \"e2e-ipp-a-f0039c-openshift-default-5f656fc965-rjf2w\" (UID: \"9178b9f8-6fda-40a9-8850-fa0097f2d39a\") " pod="openshift-ingress/e2e-ipp-a-f0039c-openshift-default-5f656fc965-rjf2w"
Nov 02 15:46:07.158607 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:07.158417    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9178b9f8-6fda-40a9-8850-fa0097f2d39a-istiod-ca-cert\") pod \"e2e-ipp-a-f0039c-openshift-default-5f656fc965-rjf2w\" (UID: \"9178b9f8-6fda-40a9-8850-fa0097f2d39a\") " pod="openshift-ingress/e2e-ipp-a-f0039c-openshift-default-5f656fc965-rjf2w"
Nov 02 15:46:07.158607 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:07.158476    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9178b9f8-6fda-40a9-8850-fa0097f2d39a-credential-socket\") pod \"e2e-ipp-a-f0039c-openshift-default-5f656fc965-rjf2w\" (UID: \"9178b9f8-6fda-40a9-8850-fa0097f2d39a\") " pod="openshift-ingress/e2e-ipp-a-f0039c-openshift-default-5f656fc965-rjf2w"
Nov 02 15:46:07.158924 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:07.158888    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9178b9f8-6fda-40a9-8850-fa0097f2d39a-credential-socket\") pod \"e2e-ipp-a-f0039c-openshift-default-5f656fc965-rjf2w\" (UID: \"9178b9f8-6fda-40a9-8850-fa0097f2d39a\") " pod="openshift-ingress/e2e-ipp-a-f0039c-openshift-default-5f656fc965-rjf2w"
Nov 02 15:46:07.159047 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:07.158980    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9178b9f8-6fda-40a9-8850-fa0097f2d39a-workload-certs\") pod \"e2e-ipp-a-f0039c-openshift-default-5f656fc965-rjf2w\" (UID: \"9178b9f8-6fda-40a9-8850-fa0097f2d39a\") " pod="openshift-ingress/e2e-ipp-a-f0039c-openshift-default-5f656fc965-rjf2w"
Nov 02 15:46:07.159047 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:07.159042    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9178b9f8-6fda-40a9-8850-fa0097f2d39a-istio-data\") pod \"e2e-ipp-a-f0039c-openshift-default-5f656fc965-rjf2w\" (UID: \"9178b9f8-6fda-40a9-8850-fa0097f2d39a\") " pod="openshift-ingress/e2e-ipp-a-f0039c-openshift-default-5f656fc965-rjf2w"
Nov 02 15:46:07.159234 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:07.159103    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9178b9f8-6fda-40a9-8850-fa0097f2d39a-istio-podinfo\") pod \"e2e-ipp-a-f0039c-openshift-default-5f656fc965-rjf2w\" (UID: \"9178b9f8-6fda-40a9-8850-fa0097f2d39a\") " pod="openshift-ingress/e2e-ipp-a-f0039c-openshift-default-5f656fc965-rjf2w"
Nov 02 15:46:07.159234 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:07.159171    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9178b9f8-6fda-40a9-8850-fa0097f2d39a-istio-token\") pod \"e2e-ipp-a-f0039c-openshift-default-5f656fc965-rjf2w\" (UID: \"9178b9f8-6fda-40a9-8850-fa0097f2d39a\") " pod="openshift-ingress/e2e-ipp-a-f0039c-openshift-default-5f656fc965-rjf2w"
Nov 02 15:46:07.159234 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:07.159202    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9178b9f8-6fda-40a9-8850-fa0097f2d39a-istiod-ca-cert\") pod \"e2e-ipp-a-f0039c-openshift-default-5f656fc965-rjf2w\" (UID: \"9178b9f8-6fda-40a9-8850-fa0097f2d39a\") " pod="openshift-ingress/e2e-ipp-a-f0039c-openshift-default-5f656fc965-rjf2w"
Nov 02 15:46:07.159402 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:07.159239    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9178b9f8-6fda-40a9-8850-fa0097f2d39a-workload-socket\") pod \"e2e-ipp-a-f0039c-openshift-default-5f656fc965-rjf2w\" (UID: \"9178b9f8-6fda-40a9-8850-fa0097f2d39a\") " pod="openshift-ingress/e2e-ipp-a-f0039c-openshift-default-5f656fc965-rjf2w"
Nov 02 15:46:07.159402 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:07.159270    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zxl5t\" (UniqueName: \"kubernetes.io/projected/9178b9f8-6fda-40a9-8850-fa0097f2d39a-kube-api-access-zxl5t\") pod \"e2e-ipp-a-f0039c-openshift-default-5f656fc965-rjf2w\" (UID: \"9178b9f8-6fda-40a9-8850-fa0097f2d39a\") " pod="openshift-ingress/e2e-ipp-a-f0039c-openshift-default-5f656fc965-rjf2w"
Nov 02 15:46:07.159402 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:07.159312    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9178b9f8-6fda-40a9-8850-fa0097f2d39a-workload-certs\") pod \"e2e-ipp-a-f0039c-openshift-default-5f656fc965-rjf2w\" (UID: \"9178b9f8-6fda-40a9-8850-fa0097f2d39a\") " pod="openshift-ingress/e2e-ipp-a-f0039c-openshift-default-5f656fc965-rjf2w"
Nov 02 15:46:07.159589 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:07.159423    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9178b9f8-6fda-40a9-8850-fa0097f2d39a-istio-data\") pod \"e2e-ipp-a-f0039c-openshift-default-5f656fc965-rjf2w\" (UID: \"9178b9f8-6fda-40a9-8850-fa0097f2d39a\") " pod="openshift-ingress/e2e-ipp-a-f0039c-openshift-default-5f656fc965-rjf2w"
Nov 02 15:46:07.159721 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:07.159699    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9178b9f8-6fda-40a9-8850-fa0097f2d39a-workload-socket\") pod \"e2e-ipp-a-f0039c-openshift-default-5f656fc965-rjf2w\" (UID: \"9178b9f8-6fda-40a9-8850-fa0097f2d39a\") " pod="openshift-ingress/e2e-ipp-a-f0039c-openshift-default-5f656fc965-rjf2w"
Nov 02 15:46:07.160917 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:07.160888    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9178b9f8-6fda-40a9-8850-fa0097f2d39a-istio-envoy\") pod \"e2e-ipp-a-f0039c-openshift-default-5f656fc965-rjf2w\" (UID: \"9178b9f8-6fda-40a9-8850-fa0097f2d39a\") " pod="openshift-ingress/e2e-ipp-a-f0039c-openshift-default-5f656fc965-rjf2w"
Nov 02 15:46:07.161420 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:07.161401    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9178b9f8-6fda-40a9-8850-fa0097f2d39a-istio-podinfo\") pod \"e2e-ipp-a-f0039c-openshift-default-5f656fc965-rjf2w\" (UID: \"9178b9f8-6fda-40a9-8850-fa0097f2d39a\") " pod="openshift-ingress/e2e-ipp-a-f0039c-openshift-default-5f656fc965-rjf2w"
Nov 02 15:46:07.231983 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:07.225565    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zxl5t\" (UniqueName: \"kubernetes.io/projected/9178b9f8-6fda-40a9-8850-fa0097f2d39a-kube-api-access-zxl5t\") pod \"e2e-ipp-a-f0039c-openshift-default-5f656fc965-rjf2w\" (UID: \"9178b9f8-6fda-40a9-8850-fa0097f2d39a\") " pod="openshift-ingress/e2e-ipp-a-f0039c-openshift-default-5f656fc965-rjf2w"
Nov 02 15:46:07.254529 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:07.246687    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9178b9f8-6fda-40a9-8850-fa0097f2d39a-istio-token\") pod \"e2e-ipp-a-f0039c-openshift-default-5f656fc965-rjf2w\" (UID: \"9178b9f8-6fda-40a9-8850-fa0097f2d39a\") " pod="openshift-ingress/e2e-ipp-a-f0039c-openshift-default-5f656fc965-rjf2w"
Nov 02 15:46:07.361153 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:07.357221    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-f0039c-openshift-default-5f656fc965-rjf2w"
Nov 02 15:46:07.560436 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:07.556855    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-f84ad8-openshift-default-779f5d9f49-mstqq"]
Nov 02 15:46:07.560436 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:07.557058    5304 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-f84ad8-openshift-default-779f5d9f49-mstqq" podUID="b96fbb28-3d8d-4871-8895-ab78dfab160d" containerName="istio-proxy" containerID="cri-o://016be369cc2fd35f4dad3d3ed3dba1cc3085d1b59257ffaeb55db4a43fd57c90" gracePeriod=30
Nov 02 15:46:08.075648 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:08.075621    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-f0039c-openshift-default-5f656fc965-rjf2w"]
Nov 02 15:46:08.077999 ip-10-0-135-138 kubenswrapper[5304]: W1102 15:46:08.077971    5304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9178b9f8_6fda_40a9_8850_fa0097f2d39a.slice/crio-7c63663b5576566f525d1dc79c17c5bd9cca7ceb738a6219a05b00c237e80aed WatchSource:0}: Error finding container 7c63663b5576566f525d1dc79c17c5bd9cca7ceb738a6219a05b00c237e80aed: Status 404 returned error can't find the container with id 7c63663b5576566f525d1dc79c17c5bd9cca7ceb738a6219a05b00c237e80aed
Nov 02 15:46:08.081026 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:08.080985    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:46:08.081115 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:08.081058    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:46:08.081115 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:08.081087    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:46:08.899644 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:08.899612    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f0039c-openshift-default-5f656fc965-rjf2w" event={"ID":"9178b9f8-6fda-40a9-8850-fa0097f2d39a","Type":"ContainerStarted","Data":"7f93fabea8b9718b8713ce2ac008f397f62178433855a93d33fdd815291278ee"}
Nov 02 15:46:08.899849 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:08.899673    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f0039c-openshift-default-5f656fc965-rjf2w" event={"ID":"9178b9f8-6fda-40a9-8850-fa0097f2d39a","Type":"ContainerStarted","Data":"7c63663b5576566f525d1dc79c17c5bd9cca7ceb738a6219a05b00c237e80aed"}
Nov 02 15:46:08.961930 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:08.961880    5304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-f0039c-openshift-default-5f656fc965-rjf2w" podStartSLOduration=2.9618641500000003 podStartE2EDuration="2.96186415s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:46:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:46:08.957393923 +0000 UTC m=+2968.951218344" watchObservedRunningTime="2026-11-02 15:46:08.96186415 +0000 UTC m=+2968.955688558"
Nov 02 15:46:09.358644 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:09.358609    5304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-f0039c-openshift-default-5f656fc965-rjf2w"
Nov 02 15:46:09.364399 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:09.364372    5304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-f0039c-openshift-default-5f656fc965-rjf2w"
Nov 02 15:46:09.906420 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:09.906382    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-f0039c-openshift-default-5f656fc965-rjf2w"
Nov 02 15:46:09.907410 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:09.907388    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-f0039c-openshift-default-5f656fc965-rjf2w"
Nov 02 15:46:12.341959 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:12.341920    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-cspmf"]
Nov 02 15:46:12.342539 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:12.342195    5304 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-cspmf" podUID="529d1f83-57ea-433f-aa66-ee16157950ca" containerName="istio-proxy" containerID="cri-o://0f9dd4cca6d1f8bfba9d564ba424fd7c53f2bbc5d2e6bfcdbed7eeafd8ec2ce0" gracePeriod=30
Nov 02 15:46:12.351211 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:12.351179    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-kgbpj"]
Nov 02 15:46:12.351513 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:12.351387    5304 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-kgbpj" podUID="24182bc4-1f65-4ddf-8e20-0d5e7bb2bc29" containerName="istio-proxy" containerID="cri-o://77d58210cbb297d5ba13193f55f76b17a9e94243126efd9051a88b97200745f9" gracePeriod=30
Nov 02 15:46:12.357076 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:12.357050    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-qcms9"]
Nov 02 15:46:12.357340 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:12.357318    5304 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-qcms9" podUID="f75de3cc-f69f-4dfc-9351-9e75a08723b3" containerName="istio-proxy" containerID="cri-o://a9889cf15f06da4d089e1f321314b5f4b7903d51460a5dc49ebfa13b542b8c0d" gracePeriod=30
Nov 02 15:46:12.362597 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:12.362572    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-wr624"]
Nov 02 15:46:12.362785 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:12.362751    5304 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-wr624" podUID="20517696-bd32-4370-a351-67c838f8ccf6" containerName="istio-proxy" containerID="cri-o://58b6c2093d5f573e612edbf1a557dce96641aac727276c87303fb92c48097a92" gracePeriod=30
Nov 02 15:46:12.616474 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:12.613351    5304 patch_prober.go:29] interesting pod/e2e-ipp-b-3805b8-openshift-default-6c9f89869-wr624 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 15:46:12.616474 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:12.613421    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-wr624" podUID="20517696-bd32-4370-a351-67c838f8ccf6" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 15:46:12.857084 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:12.857019    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-f84ad8-openshift-default-779f5d9f49-mstqq"
Nov 02 15:46:12.898178 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:12.898116    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b96fbb28-3d8d-4871-8895-ab78dfab160d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b96fbb28-3d8d-4871-8895-ab78dfab160d-istio-podinfo\") pod \"b96fbb28-3d8d-4871-8895-ab78dfab160d\" (UID: \"b96fbb28-3d8d-4871-8895-ab78dfab160d\") "
Nov 02 15:46:12.898383 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:12.898249    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b96fbb28-3d8d-4871-8895-ab78dfab160d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b96fbb28-3d8d-4871-8895-ab78dfab160d-istiod-ca-cert\") pod \"b96fbb28-3d8d-4871-8895-ab78dfab160d\" (UID: \"b96fbb28-3d8d-4871-8895-ab78dfab160d\") "
Nov 02 15:46:12.898383 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:12.898324    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b96fbb28-3d8d-4871-8895-ab78dfab160d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b96fbb28-3d8d-4871-8895-ab78dfab160d-istio-envoy\") pod \"b96fbb28-3d8d-4871-8895-ab78dfab160d\" (UID: \"b96fbb28-3d8d-4871-8895-ab78dfab160d\") "
Nov 02 15:46:12.898383 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:12.898345    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b96fbb28-3d8d-4871-8895-ab78dfab160d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b96fbb28-3d8d-4871-8895-ab78dfab160d-workload-certs\") pod \"b96fbb28-3d8d-4871-8895-ab78dfab160d\" (UID: \"b96fbb28-3d8d-4871-8895-ab78dfab160d\") "
Nov 02 15:46:12.898578 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:12.898416    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b96fbb28-3d8d-4871-8895-ab78dfab160d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b96fbb28-3d8d-4871-8895-ab78dfab160d-workload-socket\") pod \"b96fbb28-3d8d-4871-8895-ab78dfab160d\" (UID: \"b96fbb28-3d8d-4871-8895-ab78dfab160d\") "
Nov 02 15:46:12.898578 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:12.898439    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b96fbb28-3d8d-4871-8895-ab78dfab160d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b96fbb28-3d8d-4871-8895-ab78dfab160d-credential-socket\") pod \"b96fbb28-3d8d-4871-8895-ab78dfab160d\" (UID: \"b96fbb28-3d8d-4871-8895-ab78dfab160d\") "
Nov 02 15:46:12.898578 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:12.898459    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b96fbb28-3d8d-4871-8895-ab78dfab160d-istio-token\" (UniqueName: \"kubernetes.io/projected/b96fbb28-3d8d-4871-8895-ab78dfab160d-istio-token\") pod \"b96fbb28-3d8d-4871-8895-ab78dfab160d\" (UID: \"b96fbb28-3d8d-4871-8895-ab78dfab160d\") "
Nov 02 15:46:12.898719 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:12.898598    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b96fbb28-3d8d-4871-8895-ab78dfab160d-kube-api-access-wx7sn\" (UniqueName: \"kubernetes.io/projected/b96fbb28-3d8d-4871-8895-ab78dfab160d-kube-api-access-wx7sn\") pod \"b96fbb28-3d8d-4871-8895-ab78dfab160d\" (UID: \"b96fbb28-3d8d-4871-8895-ab78dfab160d\") "
Nov 02 15:46:12.898719 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:12.898647    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b96fbb28-3d8d-4871-8895-ab78dfab160d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b96fbb28-3d8d-4871-8895-ab78dfab160d-istio-data\") pod \"b96fbb28-3d8d-4871-8895-ab78dfab160d\" (UID: \"b96fbb28-3d8d-4871-8895-ab78dfab160d\") "
Nov 02 15:46:12.898902 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:12.898656    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b96fbb28-3d8d-4871-8895-ab78dfab160d-istiod-ca-cert" pod "b96fbb28-3d8d-4871-8895-ab78dfab160d" (UID: "b96fbb28-3d8d-4871-8895-ab78dfab160d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:46:12.898902 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:12.898750    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b96fbb28-3d8d-4871-8895-ab78dfab160d-workload-certs" pod "b96fbb28-3d8d-4871-8895-ab78dfab160d" (UID: "b96fbb28-3d8d-4871-8895-ab78dfab160d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:46:12.899027 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:12.899002    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b96fbb28-3d8d-4871-8895-ab78dfab160d-istio-data" pod "b96fbb28-3d8d-4871-8895-ab78dfab160d" (UID: "b96fbb28-3d8d-4871-8895-ab78dfab160d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:46:12.899027 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:12.899014    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b96fbb28-3d8d-4871-8895-ab78dfab160d-workload-socket" pod "b96fbb28-3d8d-4871-8895-ab78dfab160d" (UID: "b96fbb28-3d8d-4871-8895-ab78dfab160d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:46:12.899170 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:12.899034    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b96fbb28-3d8d-4871-8895-ab78dfab160d-credential-socket" pod "b96fbb28-3d8d-4871-8895-ab78dfab160d" (UID: "b96fbb28-3d8d-4871-8895-ab78dfab160d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:46:12.899481 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:12.899450    5304 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b96fbb28-3d8d-4871-8895-ab78dfab160d-workload-certs\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:46:12.899481 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:12.899477    5304 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b96fbb28-3d8d-4871-8895-ab78dfab160d-workload-socket\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:46:12.899630 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:12.899494    5304 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b96fbb28-3d8d-4871-8895-ab78dfab160d-credential-socket\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:46:12.899630 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:12.899505    5304 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b96fbb28-3d8d-4871-8895-ab78dfab160d-istio-data\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:46:12.899630 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:12.899514    5304 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b96fbb28-3d8d-4871-8895-ab78dfab160d-istiod-ca-cert\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:46:12.900640 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:12.900558    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b96fbb28-3d8d-4871-8895-ab78dfab160d-istio-podinfo" pod "b96fbb28-3d8d-4871-8895-ab78dfab160d" (UID: "b96fbb28-3d8d-4871-8895-ab78dfab160d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 15:46:12.900703 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:12.900692    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b96fbb28-3d8d-4871-8895-ab78dfab160d-istio-envoy" pod "b96fbb28-3d8d-4871-8895-ab78dfab160d" (UID: "b96fbb28-3d8d-4871-8895-ab78dfab160d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:46:12.901452 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:12.901427    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b96fbb28-3d8d-4871-8895-ab78dfab160d-istio-token" pod "b96fbb28-3d8d-4871-8895-ab78dfab160d" (UID: "b96fbb28-3d8d-4871-8895-ab78dfab160d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:46:12.901581 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:12.901462    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b96fbb28-3d8d-4871-8895-ab78dfab160d-kube-api-access-wx7sn" pod "b96fbb28-3d8d-4871-8895-ab78dfab160d" (UID: "b96fbb28-3d8d-4871-8895-ab78dfab160d"). InnerVolumeSpecName "kube-api-access-wx7sn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:46:12.929390 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:12.929353    5304 generic.go:356] "Generic (PLEG): container finished" podID="b96fbb28-3d8d-4871-8895-ab78dfab160d" containerID="016be369cc2fd35f4dad3d3ed3dba1cc3085d1b59257ffaeb55db4a43fd57c90" exitCode=0
Nov 02 15:46:12.929539 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:12.929401    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f84ad8-openshift-default-779f5d9f49-mstqq" event={"ID":"b96fbb28-3d8d-4871-8895-ab78dfab160d","Type":"ContainerDied","Data":"016be369cc2fd35f4dad3d3ed3dba1cc3085d1b59257ffaeb55db4a43fd57c90"}
Nov 02 15:46:12.929539 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:12.929419    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-f84ad8-openshift-default-779f5d9f49-mstqq"
Nov 02 15:46:12.929539 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:12.929479    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f84ad8-openshift-default-779f5d9f49-mstqq" event={"ID":"b96fbb28-3d8d-4871-8895-ab78dfab160d","Type":"ContainerDied","Data":"89b538dae2afd2ad0fb66e7480d6afab632929aa075ff4dfbaa8c4877fb2415d"}
Nov 02 15:46:12.929539 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:12.929503    5304 scope.go:122] "RemoveContainer" containerID="016be369cc2fd35f4dad3d3ed3dba1cc3085d1b59257ffaeb55db4a43fd57c90"
Nov 02 15:46:12.943847 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:12.943749    5304 scope.go:122] "RemoveContainer" containerID="8dd5dd14b116e03a35e9b9218038cdacfc7c086554d20c52616e125e269e744c"
Nov 02 15:46:12.956391 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:12.956374    5304 scope.go:122] "RemoveContainer" containerID="016be369cc2fd35f4dad3d3ed3dba1cc3085d1b59257ffaeb55db4a43fd57c90"
Nov 02 15:46:12.956726 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:46:12.956706    5304 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"016be369cc2fd35f4dad3d3ed3dba1cc3085d1b59257ffaeb55db4a43fd57c90\": container with ID starting with 016be369cc2fd35f4dad3d3ed3dba1cc3085d1b59257ffaeb55db4a43fd57c90 not found: ID does not exist" containerID="016be369cc2fd35f4dad3d3ed3dba1cc3085d1b59257ffaeb55db4a43fd57c90"
Nov 02 15:46:12.956798 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:12.956735    5304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"016be369cc2fd35f4dad3d3ed3dba1cc3085d1b59257ffaeb55db4a43fd57c90"} err="failed to get container status \"016be369cc2fd35f4dad3d3ed3dba1cc3085d1b59257ffaeb55db4a43fd57c90\": rpc error: code = NotFound desc = could not find container \"016be369cc2fd35f4dad3d3ed3dba1cc3085d1b59257ffaeb55db4a43fd57c90\": container with ID starting with 016be369cc2fd35f4dad3d3ed3dba1cc3085d1b59257ffaeb55db4a43fd57c90 not found: ID does not exist"
Nov 02 15:46:12.956798 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:12.956756    5304 scope.go:122] "RemoveContainer" containerID="8dd5dd14b116e03a35e9b9218038cdacfc7c086554d20c52616e125e269e744c"
Nov 02 15:46:12.957070 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:46:12.957047    5304 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8dd5dd14b116e03a35e9b9218038cdacfc7c086554d20c52616e125e269e744c\": container with ID starting with 8dd5dd14b116e03a35e9b9218038cdacfc7c086554d20c52616e125e269e744c not found: ID does not exist" containerID="8dd5dd14b116e03a35e9b9218038cdacfc7c086554d20c52616e125e269e744c"
Nov 02 15:46:12.957162 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:12.957080    5304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8dd5dd14b116e03a35e9b9218038cdacfc7c086554d20c52616e125e269e744c"} err="failed to get container status \"8dd5dd14b116e03a35e9b9218038cdacfc7c086554d20c52616e125e269e744c\": rpc error: code = NotFound desc = could not find container \"8dd5dd14b116e03a35e9b9218038cdacfc7c086554d20c52616e125e269e744c\": container with ID starting with 8dd5dd14b116e03a35e9b9218038cdacfc7c086554d20c52616e125e269e744c not found: ID does not exist"
Nov 02 15:46:12.984822 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:12.984795    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-f84ad8-openshift-default-779f5d9f49-mstqq"]
Nov 02 15:46:12.990486 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:12.990460    5304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-f84ad8-openshift-default-779f5d9f49-mstqq"]
Nov 02 15:46:13.001480 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:13.001451    5304 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b96fbb28-3d8d-4871-8895-ab78dfab160d-istio-envoy\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:46:13.001480 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:13.001476    5304 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b96fbb28-3d8d-4871-8895-ab78dfab160d-istio-token\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:46:13.001678 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:13.001490    5304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wx7sn\" (UniqueName: \"kubernetes.io/projected/b96fbb28-3d8d-4871-8895-ab78dfab160d-kube-api-access-wx7sn\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:46:13.001678 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:13.001504    5304 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b96fbb28-3d8d-4871-8895-ab78dfab160d-istio-podinfo\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:46:13.008810 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:13.008788    5304 patch_prober.go:29] interesting pod/e2e-ipp-b-3805b8-openshift-default-6c9f89869-cspmf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 15:46:13.008912 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:13.008859    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-cspmf" podUID="529d1f83-57ea-433f-aa66-ee16157950ca" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 15:46:13.458657 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:13.458624    5304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f0039c-6c5bf99c7f-62jkr"]
Nov 02 15:46:13.459277 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:13.459262    5304 cpu_manager.go:414] "containerMap: removing container" podUID="b96fbb28-3d8d-4871-8895-ab78dfab160d" containerName="istio-proxy"
Nov 02 15:46:13.459277 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:13.459278    5304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b96fbb28-3d8d-4871-8895-ab78dfab160d" containerName="istio-proxy"
Nov 02 15:46:13.459371 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:13.459294    5304 cpu_manager.go:414] "containerMap: removing container" podUID="b96fbb28-3d8d-4871-8895-ab78dfab160d" containerName="istio-proxy"
Nov 02 15:46:13.459371 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:13.459301    5304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b96fbb28-3d8d-4871-8895-ab78dfab160d" containerName="istio-proxy"
Nov 02 15:46:13.459434 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:13.459387    5304 memory_manager.go:365] "RemoveStaleState removing state" podUID="b96fbb28-3d8d-4871-8895-ab78dfab160d" containerName="istio-proxy"
Nov 02 15:46:13.459434 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:13.459398    5304 memory_manager.go:365] "RemoveStaleState removing state" podUID="d665d588-97c3-4797-be57-c426b3d905ff" containerName="istio-proxy"
Nov 02 15:46:13.459434 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:13.459409    5304 memory_manager.go:365] "RemoveStaleState removing state" podUID="b96fbb28-3d8d-4871-8895-ab78dfab160d" containerName="istio-proxy"
Nov 02 15:46:13.464313 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:13.464295    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f0039c-6c5bf99c7f-62jkr"
Nov 02 15:46:13.484677 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:13.484647    5304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-f0039c-dockercfg-2zlwp\""
Nov 02 15:46:13.484997 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:13.484973    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f0039c-6c5bf99c7f-62jkr"]
Nov 02 15:46:13.485947 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:13.485931    5304 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-f0039c\""
Nov 02 15:46:13.509822 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:13.509792    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rtgv6\" (UniqueName: \"kubernetes.io/projected/6d2ec092-d84c-4335-99b9-6b14712f57ed-kube-api-access-rtgv6\") pod \"payload-pre-processing-e2e-ipp-a-f0039c-6c5bf99c7f-62jkr\" (UID: \"6d2ec092-d84c-4335-99b9-6b14712f57ed\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f0039c-6c5bf99c7f-62jkr"
Nov 02 15:46:13.510047 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:13.510016    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d2ec092-d84c-4335-99b9-6b14712f57ed-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-f0039c-6c5bf99c7f-62jkr\" (UID: \"6d2ec092-d84c-4335-99b9-6b14712f57ed\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f0039c-6c5bf99c7f-62jkr"
Nov 02 15:46:13.539868 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:13.539829    5304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f0039c-b78956c9-vm85j"]
Nov 02 15:46:13.545385 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:13.545357    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-f0039c-b78956c9-vm85j"
Nov 02 15:46:13.688599 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:13.688262    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f0039c-b78956c9-vm85j"]
Nov 02 15:46:13.717614 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:13.717204    5304 patch_prober.go:29] interesting pod/e2e-ipp-b-3805b8-openshift-default-6c9f89869-kgbpj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 15:46:13.717614 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:13.717266    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-kgbpj" podUID="24182bc4-1f65-4ddf-8e20-0d5e7bb2bc29" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 15:46:13.740904 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:13.740866    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97dde426-8f56-4bdd-a2f8-cb78c00e27a6-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-f0039c-b78956c9-vm85j\" (UID: \"97dde426-8f56-4bdd-a2f8-cb78c00e27a6\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f0039c-b78956c9-vm85j"
Nov 02 15:46:13.741101 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:13.741023    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w8mdb\" (UniqueName: \"kubernetes.io/projected/97dde426-8f56-4bdd-a2f8-cb78c00e27a6-kube-api-access-w8mdb\") pod \"payload-processing-e2e-ipp-a-f0039c-b78956c9-vm85j\" (UID: \"97dde426-8f56-4bdd-a2f8-cb78c00e27a6\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f0039c-b78956c9-vm85j"
Nov 02 15:46:13.741101 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:13.741095    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d2ec092-d84c-4335-99b9-6b14712f57ed-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-f0039c-6c5bf99c7f-62jkr\" (UID: \"6d2ec092-d84c-4335-99b9-6b14712f57ed\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f0039c-6c5bf99c7f-62jkr"
Nov 02 15:46:13.742151 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:13.741282    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rtgv6\" (UniqueName: \"kubernetes.io/projected/6d2ec092-d84c-4335-99b9-6b14712f57ed-kube-api-access-rtgv6\") pod \"payload-pre-processing-e2e-ipp-a-f0039c-6c5bf99c7f-62jkr\" (UID: \"6d2ec092-d84c-4335-99b9-6b14712f57ed\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f0039c-6c5bf99c7f-62jkr"
Nov 02 15:46:13.743391 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:13.742832    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d2ec092-d84c-4335-99b9-6b14712f57ed-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-f0039c-6c5bf99c7f-62jkr\" (UID: \"6d2ec092-d84c-4335-99b9-6b14712f57ed\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f0039c-6c5bf99c7f-62jkr"
Nov 02 15:46:13.784417 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:13.783090    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rtgv6\" (UniqueName: \"kubernetes.io/projected/6d2ec092-d84c-4335-99b9-6b14712f57ed-kube-api-access-rtgv6\") pod \"payload-pre-processing-e2e-ipp-a-f0039c-6c5bf99c7f-62jkr\" (UID: \"6d2ec092-d84c-4335-99b9-6b14712f57ed\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f0039c-6c5bf99c7f-62jkr"
Nov 02 15:46:13.784417 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:13.783353    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f0039c-6c5bf99c7f-62jkr"
Nov 02 15:46:13.843236 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:13.843195    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w8mdb\" (UniqueName: \"kubernetes.io/projected/97dde426-8f56-4bdd-a2f8-cb78c00e27a6-kube-api-access-w8mdb\") pod \"payload-processing-e2e-ipp-a-f0039c-b78956c9-vm85j\" (UID: \"97dde426-8f56-4bdd-a2f8-cb78c00e27a6\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f0039c-b78956c9-vm85j"
Nov 02 15:46:13.843462 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:13.843444    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97dde426-8f56-4bdd-a2f8-cb78c00e27a6-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-f0039c-b78956c9-vm85j\" (UID: \"97dde426-8f56-4bdd-a2f8-cb78c00e27a6\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f0039c-b78956c9-vm85j"
Nov 02 15:46:13.844365 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:13.844338    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97dde426-8f56-4bdd-a2f8-cb78c00e27a6-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-f0039c-b78956c9-vm85j\" (UID: \"97dde426-8f56-4bdd-a2f8-cb78c00e27a6\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f0039c-b78956c9-vm85j"
Nov 02 15:46:13.868297 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:13.868258    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w8mdb\" (UniqueName: \"kubernetes.io/projected/97dde426-8f56-4bdd-a2f8-cb78c00e27a6-kube-api-access-w8mdb\") pod \"payload-processing-e2e-ipp-a-f0039c-b78956c9-vm85j\" (UID: \"97dde426-8f56-4bdd-a2f8-cb78c00e27a6\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f0039c-b78956c9-vm85j"
Nov 02 15:46:13.989849 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:13.989760    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-f0039c-b78956c9-vm85j"
Nov 02 15:46:14.120975 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:14.120930    5304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b96fbb28-3d8d-4871-8895-ab78dfab160d" path="/var/lib/kubelet/pods/b96fbb28-3d8d-4871-8895-ab78dfab160d/volumes"
Nov 02 15:46:14.122725 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:14.122703    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f0039c-6c5bf99c7f-62jkr"]
Nov 02 15:46:14.127216 ip-10-0-135-138 kubenswrapper[5304]: W1102 15:46:14.127185    5304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d2ec092_d84c_4335_99b9_6b14712f57ed.slice/crio-3338303cf3fdbf856359b2e81a0b5dae9e4857c1c917f9e9669e1e34672def1e WatchSource:0}: Error finding container 3338303cf3fdbf856359b2e81a0b5dae9e4857c1c917f9e9669e1e34672def1e: Status 404 returned error can't find the container with id 3338303cf3fdbf856359b2e81a0b5dae9e4857c1c917f9e9669e1e34672def1e
Nov 02 15:46:14.297010 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:14.296928    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f0039c-b78956c9-vm85j"]
Nov 02 15:46:14.300369 ip-10-0-135-138 kubenswrapper[5304]: W1102 15:46:14.300343    5304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97dde426_8f56_4bdd_a2f8_cb78c00e27a6.slice/crio-f2f15c11268dec6b6dd0394a9c56b8ae89820f9efa2b82d082ab2ff517c25602 WatchSource:0}: Error finding container f2f15c11268dec6b6dd0394a9c56b8ae89820f9efa2b82d082ab2ff517c25602: Status 404 returned error can't find the container with id f2f15c11268dec6b6dd0394a9c56b8ae89820f9efa2b82d082ab2ff517c25602
Nov 02 15:46:14.948985 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:14.948946    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f0039c-6c5bf99c7f-62jkr" event={"ID":"6d2ec092-d84c-4335-99b9-6b14712f57ed","Type":"ContainerStarted","Data":"702a6227dd8fed3d4853cbba6c8d431c3de634078f8847c887db5c3778399290"}
Nov 02 15:46:14.949467 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:14.949018    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f0039c-6c5bf99c7f-62jkr"
Nov 02 15:46:14.949467 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:14.949042    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f0039c-6c5bf99c7f-62jkr" event={"ID":"6d2ec092-d84c-4335-99b9-6b14712f57ed","Type":"ContainerStarted","Data":"3338303cf3fdbf856359b2e81a0b5dae9e4857c1c917f9e9669e1e34672def1e"}
Nov 02 15:46:14.950646 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:14.950618    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-f0039c-b78956c9-vm85j" event={"ID":"97dde426-8f56-4bdd-a2f8-cb78c00e27a6","Type":"ContainerStarted","Data":"a24d3aa92683081a4e12edb7e7c80257264e7d3f80f1c1548b9dee328972327c"}
Nov 02 15:46:14.950727 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:14.950668    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-f0039c-b78956c9-vm85j" event={"ID":"97dde426-8f56-4bdd-a2f8-cb78c00e27a6","Type":"ContainerStarted","Data":"f2f15c11268dec6b6dd0394a9c56b8ae89820f9efa2b82d082ab2ff517c25602"}
Nov 02 15:46:14.950727 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:14.950702    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-f0039c-b78956c9-vm85j"
Nov 02 15:46:14.992029 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:14.991980    5304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f0039c-6c5bf99c7f-62jkr" podStartSLOduration=1.991967659 podStartE2EDuration="1.991967659s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:46:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:46:14.991183545 +0000 UTC m=+2974.985007952" watchObservedRunningTime="2026-11-02 15:46:14.991967659 +0000 UTC m=+2974.985792067"
Nov 02 15:46:15.034546 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:15.034492    5304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-f0039c-b78956c9-vm85j" podStartSLOduration=2.034477469 podStartE2EDuration="2.034477469s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:46:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:46:15.032436368 +0000 UTC m=+2975.026260776" watchObservedRunningTime="2026-11-02 15:46:15.034477469 +0000 UTC m=+2975.028301878"
Nov 02 15:46:17.660178 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:17.656156    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-kgbpj"
Nov 02 15:46:17.704318 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:17.704287    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-cspmf"
Nov 02 15:46:17.706357 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:17.706331    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-qcms9"
Nov 02 15:46:17.710600 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:17.710580    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-wr624"
Nov 02 15:46:17.841248 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:17.841216    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f75de3cc-f69f-4dfc-9351-9e75a08723b3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f75de3cc-f69f-4dfc-9351-9e75a08723b3-istio-envoy\") pod \"f75de3cc-f69f-4dfc-9351-9e75a08723b3\" (UID: \"f75de3cc-f69f-4dfc-9351-9e75a08723b3\") "
Nov 02 15:46:17.841447 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:17.841283    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/24182bc4-1f65-4ddf-8e20-0d5e7bb2bc29-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24182bc4-1f65-4ddf-8e20-0d5e7bb2bc29-istio-podinfo\") pod \"24182bc4-1f65-4ddf-8e20-0d5e7bb2bc29\" (UID: \"24182bc4-1f65-4ddf-8e20-0d5e7bb2bc29\") "
Nov 02 15:46:17.841447 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:17.841306    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20517696-bd32-4370-a351-67c838f8ccf6-istio-token\" (UniqueName: \"kubernetes.io/projected/20517696-bd32-4370-a351-67c838f8ccf6-istio-token\") pod \"20517696-bd32-4370-a351-67c838f8ccf6\" (UID: \"20517696-bd32-4370-a351-67c838f8ccf6\") "
Nov 02 15:46:17.841447 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:17.841348    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/529d1f83-57ea-433f-aa66-ee16157950ca-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/529d1f83-57ea-433f-aa66-ee16157950ca-workload-certs\") pod \"529d1f83-57ea-433f-aa66-ee16157950ca\" (UID: \"529d1f83-57ea-433f-aa66-ee16157950ca\") "
Nov 02 15:46:17.841447 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:17.841374    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20517696-bd32-4370-a351-67c838f8ccf6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20517696-bd32-4370-a351-67c838f8ccf6-credential-socket\") pod \"20517696-bd32-4370-a351-67c838f8ccf6\" (UID: \"20517696-bd32-4370-a351-67c838f8ccf6\") "
Nov 02 15:46:17.841447 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:17.841407    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/529d1f83-57ea-433f-aa66-ee16157950ca-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/529d1f83-57ea-433f-aa66-ee16157950ca-istio-envoy\") pod \"529d1f83-57ea-433f-aa66-ee16157950ca\" (UID: \"529d1f83-57ea-433f-aa66-ee16157950ca\") "
Nov 02 15:46:17.841447 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:17.841443    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f75de3cc-f69f-4dfc-9351-9e75a08723b3-istio-token\" (UniqueName: \"kubernetes.io/projected/f75de3cc-f69f-4dfc-9351-9e75a08723b3-istio-token\") pod \"f75de3cc-f69f-4dfc-9351-9e75a08723b3\" (UID: \"f75de3cc-f69f-4dfc-9351-9e75a08723b3\") "
Nov 02 15:46:17.841889 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:17.841478    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/529d1f83-57ea-433f-aa66-ee16157950ca-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/529d1f83-57ea-433f-aa66-ee16157950ca-istio-podinfo\") pod \"529d1f83-57ea-433f-aa66-ee16157950ca\" (UID: \"529d1f83-57ea-433f-aa66-ee16157950ca\") "
Nov 02 15:46:17.841889 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:17.841520    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24182bc4-1f65-4ddf-8e20-0d5e7bb2bc29-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24182bc4-1f65-4ddf-8e20-0d5e7bb2bc29-istio-envoy\") pod \"24182bc4-1f65-4ddf-8e20-0d5e7bb2bc29\" (UID: \"24182bc4-1f65-4ddf-8e20-0d5e7bb2bc29\") "
Nov 02 15:46:17.841889 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:17.841549    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f75de3cc-f69f-4dfc-9351-9e75a08723b3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f75de3cc-f69f-4dfc-9351-9e75a08723b3-workload-socket\") pod \"f75de3cc-f69f-4dfc-9351-9e75a08723b3\" (UID: \"f75de3cc-f69f-4dfc-9351-9e75a08723b3\") "
Nov 02 15:46:17.841889 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:17.841586    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f75de3cc-f69f-4dfc-9351-9e75a08723b3-kube-api-access-jn78x\" (UniqueName: \"kubernetes.io/projected/f75de3cc-f69f-4dfc-9351-9e75a08723b3-kube-api-access-jn78x\") pod \"f75de3cc-f69f-4dfc-9351-9e75a08723b3\" (UID: \"f75de3cc-f69f-4dfc-9351-9e75a08723b3\") "
Nov 02 15:46:17.841889 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:17.841620    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f75de3cc-f69f-4dfc-9351-9e75a08723b3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f75de3cc-f69f-4dfc-9351-9e75a08723b3-istiod-ca-cert\") pod \"f75de3cc-f69f-4dfc-9351-9e75a08723b3\" (UID: \"f75de3cc-f69f-4dfc-9351-9e75a08723b3\") "
Nov 02 15:46:17.841889 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:17.841670    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/529d1f83-57ea-433f-aa66-ee16157950ca-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/529d1f83-57ea-433f-aa66-ee16157950ca-credential-socket\") pod \"529d1f83-57ea-433f-aa66-ee16157950ca\" (UID: \"529d1f83-57ea-433f-aa66-ee16157950ca\") "
Nov 02 15:46:17.841889 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:17.841723    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/529d1f83-57ea-433f-aa66-ee16157950ca-istio-token\" (UniqueName: \"kubernetes.io/projected/529d1f83-57ea-433f-aa66-ee16157950ca-istio-token\") pod \"529d1f83-57ea-433f-aa66-ee16157950ca\" (UID: \"529d1f83-57ea-433f-aa66-ee16157950ca\") "
Nov 02 15:46:17.841889 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:17.841767    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/529d1f83-57ea-433f-aa66-ee16157950ca-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/529d1f83-57ea-433f-aa66-ee16157950ca-workload-socket\") pod \"529d1f83-57ea-433f-aa66-ee16157950ca\" (UID: \"529d1f83-57ea-433f-aa66-ee16157950ca\") "
Nov 02 15:46:17.841889 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:17.841793    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24182bc4-1f65-4ddf-8e20-0d5e7bb2bc29-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24182bc4-1f65-4ddf-8e20-0d5e7bb2bc29-workload-certs\") pod \"24182bc4-1f65-4ddf-8e20-0d5e7bb2bc29\" (UID: \"24182bc4-1f65-4ddf-8e20-0d5e7bb2bc29\") "
Nov 02 15:46:17.841889 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:17.841824    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20517696-bd32-4370-a351-67c838f8ccf6-credential-socket" pod "20517696-bd32-4370-a351-67c838f8ccf6" (UID: "20517696-bd32-4370-a351-67c838f8ccf6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:46:17.841889 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:17.841842    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24182bc4-1f65-4ddf-8e20-0d5e7bb2bc29-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24182bc4-1f65-4ddf-8e20-0d5e7bb2bc29-credential-socket\") pod \"24182bc4-1f65-4ddf-8e20-0d5e7bb2bc29\" (UID: \"24182bc4-1f65-4ddf-8e20-0d5e7bb2bc29\") "
Nov 02 15:46:17.841889 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:17.841872    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f75de3cc-f69f-4dfc-9351-9e75a08723b3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f75de3cc-f69f-4dfc-9351-9e75a08723b3-workload-certs\") pod \"f75de3cc-f69f-4dfc-9351-9e75a08723b3\" (UID: \"f75de3cc-f69f-4dfc-9351-9e75a08723b3\") "
Nov 02 15:46:17.842590 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:17.841923    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20517696-bd32-4370-a351-67c838f8ccf6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20517696-bd32-4370-a351-67c838f8ccf6-workload-certs\") pod \"20517696-bd32-4370-a351-67c838f8ccf6\" (UID: \"20517696-bd32-4370-a351-67c838f8ccf6\") "
Nov 02 15:46:17.842590 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:17.841961    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/20517696-bd32-4370-a351-67c838f8ccf6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20517696-bd32-4370-a351-67c838f8ccf6-istio-podinfo\") pod \"20517696-bd32-4370-a351-67c838f8ccf6\" (UID: \"20517696-bd32-4370-a351-67c838f8ccf6\") "
Nov 02 15:46:17.842590 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:17.842012    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/529d1f83-57ea-433f-aa66-ee16157950ca-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/529d1f83-57ea-433f-aa66-ee16157950ca-istiod-ca-cert\") pod \"529d1f83-57ea-433f-aa66-ee16157950ca\" (UID: \"529d1f83-57ea-433f-aa66-ee16157950ca\") "
Nov 02 15:46:17.842590 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:17.842046    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20517696-bd32-4370-a351-67c838f8ccf6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20517696-bd32-4370-a351-67c838f8ccf6-istio-data\") pod \"20517696-bd32-4370-a351-67c838f8ccf6\" (UID: \"20517696-bd32-4370-a351-67c838f8ccf6\") "
Nov 02 15:46:17.842590 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:17.842088    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f75de3cc-f69f-4dfc-9351-9e75a08723b3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f75de3cc-f69f-4dfc-9351-9e75a08723b3-credential-socket\") pod \"f75de3cc-f69f-4dfc-9351-9e75a08723b3\" (UID: \"f75de3cc-f69f-4dfc-9351-9e75a08723b3\") "
Nov 02 15:46:17.842590 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:17.842184    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/24182bc4-1f65-4ddf-8e20-0d5e7bb2bc29-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24182bc4-1f65-4ddf-8e20-0d5e7bb2bc29-istiod-ca-cert\") pod \"24182bc4-1f65-4ddf-8e20-0d5e7bb2bc29\" (UID: \"24182bc4-1f65-4ddf-8e20-0d5e7bb2bc29\") "
Nov 02 15:46:17.842590 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:17.842210    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f75de3cc-f69f-4dfc-9351-9e75a08723b3-istiod-ca-cert" pod "f75de3cc-f69f-4dfc-9351-9e75a08723b3" (UID: "f75de3cc-f69f-4dfc-9351-9e75a08723b3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:46:17.842590 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:17.842222    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24182bc4-1f65-4ddf-8e20-0d5e7bb2bc29-istio-token\" (UniqueName: \"kubernetes.io/projected/24182bc4-1f65-4ddf-8e20-0d5e7bb2bc29-istio-token\") pod \"24182bc4-1f65-4ddf-8e20-0d5e7bb2bc29\" (UID: \"24182bc4-1f65-4ddf-8e20-0d5e7bb2bc29\") "
Nov 02 15:46:17.842590 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:17.842249    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/529d1f83-57ea-433f-aa66-ee16157950ca-workload-certs" pod "529d1f83-57ea-433f-aa66-ee16157950ca" (UID: "529d1f83-57ea-433f-aa66-ee16157950ca"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:46:17.842590 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:17.842291    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20517696-bd32-4370-a351-67c838f8ccf6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20517696-bd32-4370-a351-67c838f8ccf6-workload-socket\") pod \"20517696-bd32-4370-a351-67c838f8ccf6\" (UID: \"20517696-bd32-4370-a351-67c838f8ccf6\") "
Nov 02 15:46:17.842590 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:17.842334    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20517696-bd32-4370-a351-67c838f8ccf6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20517696-bd32-4370-a351-67c838f8ccf6-istio-envoy\") pod \"20517696-bd32-4370-a351-67c838f8ccf6\" (UID: \"20517696-bd32-4370-a351-67c838f8ccf6\") "
Nov 02 15:46:17.842590 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:17.842379    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24182bc4-1f65-4ddf-8e20-0d5e7bb2bc29-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24182bc4-1f65-4ddf-8e20-0d5e7bb2bc29-workload-socket\") pod \"24182bc4-1f65-4ddf-8e20-0d5e7bb2bc29\" (UID: \"24182bc4-1f65-4ddf-8e20-0d5e7bb2bc29\") "
Nov 02 15:46:17.842590 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:17.842453    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f75de3cc-f69f-4dfc-9351-9e75a08723b3-workload-certs" pod "f75de3cc-f69f-4dfc-9351-9e75a08723b3" (UID: "f75de3cc-f69f-4dfc-9351-9e75a08723b3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:46:17.842590 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:17.842459    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f75de3cc-f69f-4dfc-9351-9e75a08723b3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f75de3cc-f69f-4dfc-9351-9e75a08723b3-istio-data\") pod \"f75de3cc-f69f-4dfc-9351-9e75a08723b3\" (UID: \"f75de3cc-f69f-4dfc-9351-9e75a08723b3\") "
Nov 02 15:46:17.842590 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:17.842488    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/20517696-bd32-4370-a351-67c838f8ccf6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20517696-bd32-4370-a351-67c838f8ccf6-istiod-ca-cert\") pod \"20517696-bd32-4370-a351-67c838f8ccf6\" (UID: \"20517696-bd32-4370-a351-67c838f8ccf6\") "
Nov 02 15:46:17.842590 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:17.842525    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/529d1f83-57ea-433f-aa66-ee16157950ca-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/529d1f83-57ea-433f-aa66-ee16157950ca-istio-data\") pod \"529d1f83-57ea-433f-aa66-ee16157950ca\" (UID: \"529d1f83-57ea-433f-aa66-ee16157950ca\") "
Nov 02 15:46:17.842590 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:17.842599    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20517696-bd32-4370-a351-67c838f8ccf6-kube-api-access-q9p87\" (UniqueName: \"kubernetes.io/projected/20517696-bd32-4370-a351-67c838f8ccf6-kube-api-access-q9p87\") pod \"20517696-bd32-4370-a351-67c838f8ccf6\" (UID: \"20517696-bd32-4370-a351-67c838f8ccf6\") "
Nov 02 15:46:17.843516 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:17.842612    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/529d1f83-57ea-433f-aa66-ee16157950ca-credential-socket" pod "529d1f83-57ea-433f-aa66-ee16157950ca" (UID: "529d1f83-57ea-433f-aa66-ee16157950ca"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:46:17.843516 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:17.842640    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f75de3cc-f69f-4dfc-9351-9e75a08723b3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f75de3cc-f69f-4dfc-9351-9e75a08723b3-istio-podinfo\") pod \"f75de3cc-f69f-4dfc-9351-9e75a08723b3\" (UID: \"f75de3cc-f69f-4dfc-9351-9e75a08723b3\") "
Nov 02 15:46:17.843516 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:17.842695    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24182bc4-1f65-4ddf-8e20-0d5e7bb2bc29-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24182bc4-1f65-4ddf-8e20-0d5e7bb2bc29-istio-data\") pod \"24182bc4-1f65-4ddf-8e20-0d5e7bb2bc29\" (UID: \"24182bc4-1f65-4ddf-8e20-0d5e7bb2bc29\") "
Nov 02 15:46:17.843516 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:17.842812    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/529d1f83-57ea-433f-aa66-ee16157950ca-kube-api-access-lhjbd\" (UniqueName: \"kubernetes.io/projected/529d1f83-57ea-433f-aa66-ee16157950ca-kube-api-access-lhjbd\") pod \"529d1f83-57ea-433f-aa66-ee16157950ca\" (UID: \"529d1f83-57ea-433f-aa66-ee16157950ca\") "
Nov 02 15:46:17.843516 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:17.842842    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24182bc4-1f65-4ddf-8e20-0d5e7bb2bc29-kube-api-access-rqjxh\" (UniqueName: \"kubernetes.io/projected/24182bc4-1f65-4ddf-8e20-0d5e7bb2bc29-kube-api-access-rqjxh\") pod \"24182bc4-1f65-4ddf-8e20-0d5e7bb2bc29\" (UID: \"24182bc4-1f65-4ddf-8e20-0d5e7bb2bc29\") "
Nov 02 15:46:17.843516 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:17.843256    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24182bc4-1f65-4ddf-8e20-0d5e7bb2bc29-workload-certs" pod "24182bc4-1f65-4ddf-8e20-0d5e7bb2bc29" (UID: "24182bc4-1f65-4ddf-8e20-0d5e7bb2bc29"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:46:17.843922 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:17.843664    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20517696-bd32-4370-a351-67c838f8ccf6-istio-data" pod "20517696-bd32-4370-a351-67c838f8ccf6" (UID: "20517696-bd32-4370-a351-67c838f8ccf6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:46:17.843922 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:17.843716    5304 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/529d1f83-57ea-433f-aa66-ee16157950ca-workload-certs\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:46:17.843922 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:17.843738    5304 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20517696-bd32-4370-a351-67c838f8ccf6-credential-socket\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:46:17.843922 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:17.843755    5304 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f75de3cc-f69f-4dfc-9351-9e75a08723b3-istiod-ca-cert\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:46:17.843922 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:17.843769    5304 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/529d1f83-57ea-433f-aa66-ee16157950ca-credential-socket\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:46:17.843922 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:17.843781    5304 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24182bc4-1f65-4ddf-8e20-0d5e7bb2bc29-workload-certs\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:46:17.843922 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:17.843795    5304 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f75de3cc-f69f-4dfc-9351-9e75a08723b3-workload-certs\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:46:17.843922 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:17.843848    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20517696-bd32-4370-a351-67c838f8ccf6-workload-certs" pod "20517696-bd32-4370-a351-67c838f8ccf6" (UID: "20517696-bd32-4370-a351-67c838f8ccf6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:46:17.844388 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:17.843956    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/529d1f83-57ea-433f-aa66-ee16157950ca-workload-socket" pod "529d1f83-57ea-433f-aa66-ee16157950ca" (UID: "529d1f83-57ea-433f-aa66-ee16157950ca"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:46:17.844388 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:17.844010    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f75de3cc-f69f-4dfc-9351-9e75a08723b3-istio-envoy" pod "f75de3cc-f69f-4dfc-9351-9e75a08723b3" (UID: "f75de3cc-f69f-4dfc-9351-9e75a08723b3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:46:17.844388 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:17.844303    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24182bc4-1f65-4ddf-8e20-0d5e7bb2bc29-istiod-ca-cert" pod "24182bc4-1f65-4ddf-8e20-0d5e7bb2bc29" (UID: "24182bc4-1f65-4ddf-8e20-0d5e7bb2bc29"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:46:17.844571 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:17.844483    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24182bc4-1f65-4ddf-8e20-0d5e7bb2bc29-workload-socket" pod "24182bc4-1f65-4ddf-8e20-0d5e7bb2bc29" (UID: "24182bc4-1f65-4ddf-8e20-0d5e7bb2bc29"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:46:17.844571 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:17.844503    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24182bc4-1f65-4ddf-8e20-0d5e7bb2bc29-credential-socket" pod "24182bc4-1f65-4ddf-8e20-0d5e7bb2bc29" (UID: "24182bc4-1f65-4ddf-8e20-0d5e7bb2bc29"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:46:17.844571 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:17.844558    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f75de3cc-f69f-4dfc-9351-9e75a08723b3-credential-socket" pod "f75de3cc-f69f-4dfc-9351-9e75a08723b3" (UID: "f75de3cc-f69f-4dfc-9351-9e75a08723b3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:46:17.844739 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:17.844684    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f75de3cc-f69f-4dfc-9351-9e75a08723b3-istio-data" pod "f75de3cc-f69f-4dfc-9351-9e75a08723b3" (UID: "f75de3cc-f69f-4dfc-9351-9e75a08723b3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:46:17.844796 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:17.844742    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f75de3cc-f69f-4dfc-9351-9e75a08723b3-workload-socket" pod "f75de3cc-f69f-4dfc-9351-9e75a08723b3" (UID: "f75de3cc-f69f-4dfc-9351-9e75a08723b3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:46:17.845210 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:17.845009    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20517696-bd32-4370-a351-67c838f8ccf6-istiod-ca-cert" pod "20517696-bd32-4370-a351-67c838f8ccf6" (UID: "20517696-bd32-4370-a351-67c838f8ccf6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:46:17.845343 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:17.845253    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20517696-bd32-4370-a351-67c838f8ccf6-workload-socket" pod "20517696-bd32-4370-a351-67c838f8ccf6" (UID: "20517696-bd32-4370-a351-67c838f8ccf6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:46:17.845343 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:17.845296    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24182bc4-1f65-4ddf-8e20-0d5e7bb2bc29-istio-data" pod "24182bc4-1f65-4ddf-8e20-0d5e7bb2bc29" (UID: "24182bc4-1f65-4ddf-8e20-0d5e7bb2bc29"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:46:17.845343 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:17.845303    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/529d1f83-57ea-433f-aa66-ee16157950ca-istiod-ca-cert" pod "529d1f83-57ea-433f-aa66-ee16157950ca" (UID: "529d1f83-57ea-433f-aa66-ee16157950ca"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:46:17.845721 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:17.845690    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/24182bc4-1f65-4ddf-8e20-0d5e7bb2bc29-istio-podinfo" pod "24182bc4-1f65-4ddf-8e20-0d5e7bb2bc29" (UID: "24182bc4-1f65-4ddf-8e20-0d5e7bb2bc29"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 15:46:17.846195 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:17.845939    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/529d1f83-57ea-433f-aa66-ee16157950ca-istio-data" pod "529d1f83-57ea-433f-aa66-ee16157950ca" (UID: "529d1f83-57ea-433f-aa66-ee16157950ca"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:46:17.847783 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:17.847744    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/529d1f83-57ea-433f-aa66-ee16157950ca-istio-podinfo" pod "529d1f83-57ea-433f-aa66-ee16157950ca" (UID: "529d1f83-57ea-433f-aa66-ee16157950ca"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 15:46:17.847902 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:17.847831    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24182bc4-1f65-4ddf-8e20-0d5e7bb2bc29-kube-api-access-rqjxh" pod "24182bc4-1f65-4ddf-8e20-0d5e7bb2bc29" (UID: "24182bc4-1f65-4ddf-8e20-0d5e7bb2bc29"). InnerVolumeSpecName "kube-api-access-rqjxh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:46:17.848619 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:17.848579    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20517696-bd32-4370-a351-67c838f8ccf6-istio-token" pod "20517696-bd32-4370-a351-67c838f8ccf6" (UID: "20517696-bd32-4370-a351-67c838f8ccf6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:46:17.848729 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:17.848664    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/529d1f83-57ea-433f-aa66-ee16157950ca-kube-api-access-lhjbd" pod "529d1f83-57ea-433f-aa66-ee16157950ca" (UID: "529d1f83-57ea-433f-aa66-ee16157950ca"). InnerVolumeSpecName "kube-api-access-lhjbd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:46:17.849173 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:17.849146    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/20517696-bd32-4370-a351-67c838f8ccf6-istio-podinfo" pod "20517696-bd32-4370-a351-67c838f8ccf6" (UID: "20517696-bd32-4370-a351-67c838f8ccf6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 15:46:17.849266 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:17.849190    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20517696-bd32-4370-a351-67c838f8ccf6-istio-envoy" pod "20517696-bd32-4370-a351-67c838f8ccf6" (UID: "20517696-bd32-4370-a351-67c838f8ccf6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:46:17.849431 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:17.849372    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/529d1f83-57ea-433f-aa66-ee16157950ca-istio-token" pod "529d1f83-57ea-433f-aa66-ee16157950ca" (UID: "529d1f83-57ea-433f-aa66-ee16157950ca"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:46:17.849612 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:17.849412    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f75de3cc-f69f-4dfc-9351-9e75a08723b3-istio-token" pod "f75de3cc-f69f-4dfc-9351-9e75a08723b3" (UID: "f75de3cc-f69f-4dfc-9351-9e75a08723b3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:46:17.850609 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:17.850584    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f75de3cc-f69f-4dfc-9351-9e75a08723b3-istio-podinfo" pod "f75de3cc-f69f-4dfc-9351-9e75a08723b3" (UID: "f75de3cc-f69f-4dfc-9351-9e75a08723b3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 15:46:17.850735 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:17.850645    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24182bc4-1f65-4ddf-8e20-0d5e7bb2bc29-istio-token" pod "24182bc4-1f65-4ddf-8e20-0d5e7bb2bc29" (UID: "24182bc4-1f65-4ddf-8e20-0d5e7bb2bc29"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:46:17.850997 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:17.850938    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24182bc4-1f65-4ddf-8e20-0d5e7bb2bc29-istio-envoy" pod "24182bc4-1f65-4ddf-8e20-0d5e7bb2bc29" (UID: "24182bc4-1f65-4ddf-8e20-0d5e7bb2bc29"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:46:17.850997 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:17.850959    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20517696-bd32-4370-a351-67c838f8ccf6-kube-api-access-q9p87" pod "20517696-bd32-4370-a351-67c838f8ccf6" (UID: "20517696-bd32-4370-a351-67c838f8ccf6"). InnerVolumeSpecName "kube-api-access-q9p87". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:46:17.851221 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:17.851193    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f75de3cc-f69f-4dfc-9351-9e75a08723b3-kube-api-access-jn78x" pod "f75de3cc-f69f-4dfc-9351-9e75a08723b3" (UID: "f75de3cc-f69f-4dfc-9351-9e75a08723b3"). InnerVolumeSpecName "kube-api-access-jn78x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:46:17.852065 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:17.852044    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/529d1f83-57ea-433f-aa66-ee16157950ca-istio-envoy" pod "529d1f83-57ea-433f-aa66-ee16157950ca" (UID: "529d1f83-57ea-433f-aa66-ee16157950ca"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:46:17.945046 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:17.944951    5304 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24182bc4-1f65-4ddf-8e20-0d5e7bb2bc29-istiod-ca-cert\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:46:17.945046 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:17.944992    5304 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/24182bc4-1f65-4ddf-8e20-0d5e7bb2bc29-istio-token\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:46:17.945046 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:17.945006    5304 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20517696-bd32-4370-a351-67c838f8ccf6-workload-socket\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:46:17.945046 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:17.945018    5304 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20517696-bd32-4370-a351-67c838f8ccf6-istio-envoy\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:46:17.945046 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:17.945030    5304 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24182bc4-1f65-4ddf-8e20-0d5e7bb2bc29-workload-socket\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:46:17.945046 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:17.945043    5304 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f75de3cc-f69f-4dfc-9351-9e75a08723b3-istio-data\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:46:17.946180 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:17.945056    5304 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20517696-bd32-4370-a351-67c838f8ccf6-istiod-ca-cert\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:46:17.946180 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:17.945067    5304 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/529d1f83-57ea-433f-aa66-ee16157950ca-istio-data\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:46:17.946180 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:17.945079    5304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q9p87\" (UniqueName: \"kubernetes.io/projected/20517696-bd32-4370-a351-67c838f8ccf6-kube-api-access-q9p87\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:46:17.946180 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:17.945092    5304 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f75de3cc-f69f-4dfc-9351-9e75a08723b3-istio-podinfo\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:46:17.946180 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:17.945104    5304 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24182bc4-1f65-4ddf-8e20-0d5e7bb2bc29-istio-data\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:46:17.946180 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:17.945116    5304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lhjbd\" (UniqueName: \"kubernetes.io/projected/529d1f83-57ea-433f-aa66-ee16157950ca-kube-api-access-lhjbd\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:46:17.946180 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:17.945144    5304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rqjxh\" (UniqueName: \"kubernetes.io/projected/24182bc4-1f65-4ddf-8e20-0d5e7bb2bc29-kube-api-access-rqjxh\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:46:17.946180 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:17.945158    5304 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f75de3cc-f69f-4dfc-9351-9e75a08723b3-istio-envoy\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:46:17.946180 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:17.945170    5304 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24182bc4-1f65-4ddf-8e20-0d5e7bb2bc29-istio-podinfo\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:46:17.946180 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:17.945181    5304 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20517696-bd32-4370-a351-67c838f8ccf6-istio-token\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:46:17.946180 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:17.945193    5304 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/529d1f83-57ea-433f-aa66-ee16157950ca-istio-envoy\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:46:17.946180 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:17.945205    5304 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f75de3cc-f69f-4dfc-9351-9e75a08723b3-istio-token\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:46:17.946180 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:17.945216    5304 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/529d1f83-57ea-433f-aa66-ee16157950ca-istio-podinfo\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:46:17.946180 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:17.945229    5304 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24182bc4-1f65-4ddf-8e20-0d5e7bb2bc29-istio-envoy\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:46:17.946180 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:17.945241    5304 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f75de3cc-f69f-4dfc-9351-9e75a08723b3-workload-socket\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:46:17.946180 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:17.945252    5304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jn78x\" (UniqueName: \"kubernetes.io/projected/f75de3cc-f69f-4dfc-9351-9e75a08723b3-kube-api-access-jn78x\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:46:17.946180 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:17.945265    5304 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/529d1f83-57ea-433f-aa66-ee16157950ca-istio-token\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:46:17.946180 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:17.945279    5304 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/529d1f83-57ea-433f-aa66-ee16157950ca-workload-socket\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:46:17.946180 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:17.945292    5304 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24182bc4-1f65-4ddf-8e20-0d5e7bb2bc29-credential-socket\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:46:17.946180 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:17.945305    5304 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20517696-bd32-4370-a351-67c838f8ccf6-workload-certs\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:46:17.946180 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:17.945317    5304 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20517696-bd32-4370-a351-67c838f8ccf6-istio-podinfo\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:46:17.946180 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:17.945328    5304 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/529d1f83-57ea-433f-aa66-ee16157950ca-istiod-ca-cert\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:46:17.946180 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:17.945340    5304 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20517696-bd32-4370-a351-67c838f8ccf6-istio-data\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:46:17.946180 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:17.945351    5304 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f75de3cc-f69f-4dfc-9351-9e75a08723b3-credential-socket\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:46:17.975002 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:17.974968    5304 generic.go:356] "Generic (PLEG): container finished" podID="529d1f83-57ea-433f-aa66-ee16157950ca" containerID="0f9dd4cca6d1f8bfba9d564ba424fd7c53f2bbc5d2e6bfcdbed7eeafd8ec2ce0" exitCode=0
Nov 02 15:46:17.975207 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:17.975038    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-cspmf"
Nov 02 15:46:17.975207 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:17.975046    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-cspmf" event={"ID":"529d1f83-57ea-433f-aa66-ee16157950ca","Type":"ContainerDied","Data":"0f9dd4cca6d1f8bfba9d564ba424fd7c53f2bbc5d2e6bfcdbed7eeafd8ec2ce0"}
Nov 02 15:46:17.975207 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:17.975168    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-cspmf" event={"ID":"529d1f83-57ea-433f-aa66-ee16157950ca","Type":"ContainerDied","Data":"a42710044e5eb273532556517341d653192000c01cfa9e890d30b92096aac8fb"}
Nov 02 15:46:17.975207 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:17.975201    5304 scope.go:122] "RemoveContainer" containerID="0f9dd4cca6d1f8bfba9d564ba424fd7c53f2bbc5d2e6bfcdbed7eeafd8ec2ce0"
Nov 02 15:46:17.976703 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:17.976682    5304 generic.go:356] "Generic (PLEG): container finished" podID="f75de3cc-f69f-4dfc-9351-9e75a08723b3" containerID="a9889cf15f06da4d089e1f321314b5f4b7903d51460a5dc49ebfa13b542b8c0d" exitCode=0
Nov 02 15:46:17.976841 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:17.976742    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-qcms9"
Nov 02 15:46:17.976841 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:17.976760    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-qcms9" event={"ID":"f75de3cc-f69f-4dfc-9351-9e75a08723b3","Type":"ContainerDied","Data":"a9889cf15f06da4d089e1f321314b5f4b7903d51460a5dc49ebfa13b542b8c0d"}
Nov 02 15:46:17.976841 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:17.976822    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-qcms9" event={"ID":"f75de3cc-f69f-4dfc-9351-9e75a08723b3","Type":"ContainerDied","Data":"248579a75b74d73b7c680a64a92d6da161d423f1bd86ce327ce108dd2058bef1"}
Nov 02 15:46:17.982219 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:17.978552    5304 generic.go:356] "Generic (PLEG): container finished" podID="24182bc4-1f65-4ddf-8e20-0d5e7bb2bc29" containerID="77d58210cbb297d5ba13193f55f76b17a9e94243126efd9051a88b97200745f9" exitCode=0
Nov 02 15:46:17.982219 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:17.978594    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-kgbpj" event={"ID":"24182bc4-1f65-4ddf-8e20-0d5e7bb2bc29","Type":"ContainerDied","Data":"77d58210cbb297d5ba13193f55f76b17a9e94243126efd9051a88b97200745f9"}
Nov 02 15:46:17.982219 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:17.978653    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-kgbpj" event={"ID":"24182bc4-1f65-4ddf-8e20-0d5e7bb2bc29","Type":"ContainerDied","Data":"cb277df466b41db1518a1cc36631b2c07ddd7dd2343aff7300f8040eacc82218"}
Nov 02 15:46:17.982219 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:17.978602    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-kgbpj"
Nov 02 15:46:17.982219 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:17.980242    5304 generic.go:356] "Generic (PLEG): container finished" podID="20517696-bd32-4370-a351-67c838f8ccf6" containerID="58b6c2093d5f573e612edbf1a557dce96641aac727276c87303fb92c48097a92" exitCode=0
Nov 02 15:46:17.982219 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:17.980282    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-wr624" event={"ID":"20517696-bd32-4370-a351-67c838f8ccf6","Type":"ContainerDied","Data":"58b6c2093d5f573e612edbf1a557dce96641aac727276c87303fb92c48097a92"}
Nov 02 15:46:17.982219 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:17.980319    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-wr624" event={"ID":"20517696-bd32-4370-a351-67c838f8ccf6","Type":"ContainerDied","Data":"83cce077490e40001fcea35a4c1f1295b5aa01c871b3f964a32ae4e0a3c92ffe"}
Nov 02 15:46:17.982219 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:17.980338    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-wr624"
Nov 02 15:46:17.992468 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:17.992393    5304 scope.go:122] "RemoveContainer" containerID="6774500a1af6c9e7e9c101c43bd4da5839fc192191732fd0aff2eba2c0b62600"
Nov 02 15:46:18.005326 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:18.005304    5304 scope.go:122] "RemoveContainer" containerID="0f9dd4cca6d1f8bfba9d564ba424fd7c53f2bbc5d2e6bfcdbed7eeafd8ec2ce0"
Nov 02 15:46:18.005591 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:46:18.005566    5304 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f9dd4cca6d1f8bfba9d564ba424fd7c53f2bbc5d2e6bfcdbed7eeafd8ec2ce0\": container with ID starting with 0f9dd4cca6d1f8bfba9d564ba424fd7c53f2bbc5d2e6bfcdbed7eeafd8ec2ce0 not found: ID does not exist" containerID="0f9dd4cca6d1f8bfba9d564ba424fd7c53f2bbc5d2e6bfcdbed7eeafd8ec2ce0"
Nov 02 15:46:18.005690 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:18.005605    5304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f9dd4cca6d1f8bfba9d564ba424fd7c53f2bbc5d2e6bfcdbed7eeafd8ec2ce0"} err="failed to get container status \"0f9dd4cca6d1f8bfba9d564ba424fd7c53f2bbc5d2e6bfcdbed7eeafd8ec2ce0\": rpc error: code = NotFound desc = could not find container \"0f9dd4cca6d1f8bfba9d564ba424fd7c53f2bbc5d2e6bfcdbed7eeafd8ec2ce0\": container with ID starting with 0f9dd4cca6d1f8bfba9d564ba424fd7c53f2bbc5d2e6bfcdbed7eeafd8ec2ce0 not found: ID does not exist"
Nov 02 15:46:18.005690 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:18.005629    5304 scope.go:122] "RemoveContainer" containerID="6774500a1af6c9e7e9c101c43bd4da5839fc192191732fd0aff2eba2c0b62600"
Nov 02 15:46:18.005904 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:46:18.005880    5304 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6774500a1af6c9e7e9c101c43bd4da5839fc192191732fd0aff2eba2c0b62600\": container with ID starting with 6774500a1af6c9e7e9c101c43bd4da5839fc192191732fd0aff2eba2c0b62600 not found: ID does not exist" containerID="6774500a1af6c9e7e9c101c43bd4da5839fc192191732fd0aff2eba2c0b62600"
Nov 02 15:46:18.005969 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:18.005911    5304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6774500a1af6c9e7e9c101c43bd4da5839fc192191732fd0aff2eba2c0b62600"} err="failed to get container status \"6774500a1af6c9e7e9c101c43bd4da5839fc192191732fd0aff2eba2c0b62600\": rpc error: code = NotFound desc = could not find container \"6774500a1af6c9e7e9c101c43bd4da5839fc192191732fd0aff2eba2c0b62600\": container with ID starting with 6774500a1af6c9e7e9c101c43bd4da5839fc192191732fd0aff2eba2c0b62600 not found: ID does not exist"
Nov 02 15:46:18.005969 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:18.005928    5304 scope.go:122] "RemoveContainer" containerID="a9889cf15f06da4d089e1f321314b5f4b7903d51460a5dc49ebfa13b542b8c0d"
Nov 02 15:46:18.016724 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:18.016699    5304 scope.go:122] "RemoveContainer" containerID="a9889cf15f06da4d089e1f321314b5f4b7903d51460a5dc49ebfa13b542b8c0d"
Nov 02 15:46:18.016976 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:46:18.016958    5304 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a9889cf15f06da4d089e1f321314b5f4b7903d51460a5dc49ebfa13b542b8c0d\": container with ID starting with a9889cf15f06da4d089e1f321314b5f4b7903d51460a5dc49ebfa13b542b8c0d not found: ID does not exist" containerID="a9889cf15f06da4d089e1f321314b5f4b7903d51460a5dc49ebfa13b542b8c0d"
Nov 02 15:46:18.017033 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:18.016985    5304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a9889cf15f06da4d089e1f321314b5f4b7903d51460a5dc49ebfa13b542b8c0d"} err="failed to get container status \"a9889cf15f06da4d089e1f321314b5f4b7903d51460a5dc49ebfa13b542b8c0d\": rpc error: code = NotFound desc = could not find container \"a9889cf15f06da4d089e1f321314b5f4b7903d51460a5dc49ebfa13b542b8c0d\": container with ID starting with a9889cf15f06da4d089e1f321314b5f4b7903d51460a5dc49ebfa13b542b8c0d not found: ID does not exist"
Nov 02 15:46:18.017033 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:18.017002    5304 scope.go:122] "RemoveContainer" containerID="77d58210cbb297d5ba13193f55f76b17a9e94243126efd9051a88b97200745f9"
Nov 02 15:46:18.026076 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:18.026058    5304 scope.go:122] "RemoveContainer" containerID="77d58210cbb297d5ba13193f55f76b17a9e94243126efd9051a88b97200745f9"
Nov 02 15:46:18.026399 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:46:18.026380    5304 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"77d58210cbb297d5ba13193f55f76b17a9e94243126efd9051a88b97200745f9\": container with ID starting with 77d58210cbb297d5ba13193f55f76b17a9e94243126efd9051a88b97200745f9 not found: ID does not exist" containerID="77d58210cbb297d5ba13193f55f76b17a9e94243126efd9051a88b97200745f9"
Nov 02 15:46:18.026470 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:18.026407    5304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"77d58210cbb297d5ba13193f55f76b17a9e94243126efd9051a88b97200745f9"} err="failed to get container status \"77d58210cbb297d5ba13193f55f76b17a9e94243126efd9051a88b97200745f9\": rpc error: code = NotFound desc = could not find container \"77d58210cbb297d5ba13193f55f76b17a9e94243126efd9051a88b97200745f9\": container with ID starting with 77d58210cbb297d5ba13193f55f76b17a9e94243126efd9051a88b97200745f9 not found: ID does not exist"
Nov 02 15:46:18.026470 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:18.026424    5304 scope.go:122] "RemoveContainer" containerID="58b6c2093d5f573e612edbf1a557dce96641aac727276c87303fb92c48097a92"
Nov 02 15:46:18.032338 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:18.032308    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-cspmf"]
Nov 02 15:46:18.039781 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:18.039433    5304 scope.go:122] "RemoveContainer" containerID="58b6c2093d5f573e612edbf1a557dce96641aac727276c87303fb92c48097a92"
Nov 02 15:46:18.039781 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:18.039437    5304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-cspmf"]
Nov 02 15:46:18.040075 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:46:18.040016    5304 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"58b6c2093d5f573e612edbf1a557dce96641aac727276c87303fb92c48097a92\": container with ID starting with 58b6c2093d5f573e612edbf1a557dce96641aac727276c87303fb92c48097a92 not found: ID does not exist" containerID="58b6c2093d5f573e612edbf1a557dce96641aac727276c87303fb92c48097a92"
Nov 02 15:46:18.040075 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:18.040051    5304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"58b6c2093d5f573e612edbf1a557dce96641aac727276c87303fb92c48097a92"} err="failed to get container status \"58b6c2093d5f573e612edbf1a557dce96641aac727276c87303fb92c48097a92\": rpc error: code = NotFound desc = could not find container \"58b6c2093d5f573e612edbf1a557dce96641aac727276c87303fb92c48097a92\": container with ID starting with 58b6c2093d5f573e612edbf1a557dce96641aac727276c87303fb92c48097a92 not found: ID does not exist"
Nov 02 15:46:18.120954 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:18.120918    5304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="529d1f83-57ea-433f-aa66-ee16157950ca" path="/var/lib/kubelet/pods/529d1f83-57ea-433f-aa66-ee16157950ca/volumes"
Nov 02 15:46:18.124485 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:18.124450    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-qcms9"]
Nov 02 15:46:18.131627 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:18.131595    5304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-qcms9"]
Nov 02 15:46:18.165433 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:18.165400    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-kgbpj"]
Nov 02 15:46:18.171238 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:18.171204    5304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-kgbpj"]
Nov 02 15:46:18.203874 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:18.203788    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-wr624"]
Nov 02 15:46:18.207876 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:18.207850    5304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-wr624"]
Nov 02 15:46:20.146365 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:20.146329    5304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="20517696-bd32-4370-a351-67c838f8ccf6" path="/var/lib/kubelet/pods/20517696-bd32-4370-a351-67c838f8ccf6/volumes"
Nov 02 15:46:20.147369 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:20.147347    5304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="24182bc4-1f65-4ddf-8e20-0d5e7bb2bc29" path="/var/lib/kubelet/pods/24182bc4-1f65-4ddf-8e20-0d5e7bb2bc29/volumes"
Nov 02 15:46:20.147990 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:20.147974    5304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f75de3cc-f69f-4dfc-9351-9e75a08723b3" path="/var/lib/kubelet/pods/f75de3cc-f69f-4dfc-9351-9e75a08723b3/volumes"
Nov 02 15:46:23.110941 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:23.110854    5304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e80c77-2tvs6"]
Nov 02 15:46:23.111686 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:23.111660    5304 cpu_manager.go:414] "containerMap: removing container" podUID="f75de3cc-f69f-4dfc-9351-9e75a08723b3" containerName="istio-proxy"
Nov 02 15:46:23.111786 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:23.111688    5304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f75de3cc-f69f-4dfc-9351-9e75a08723b3" containerName="istio-proxy"
Nov 02 15:46:23.111786 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:23.111714    5304 cpu_manager.go:414] "containerMap: removing container" podUID="24182bc4-1f65-4ddf-8e20-0d5e7bb2bc29" containerName="istio-proxy"
Nov 02 15:46:23.111786 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:23.111723    5304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="24182bc4-1f65-4ddf-8e20-0d5e7bb2bc29" containerName="istio-proxy"
Nov 02 15:46:23.111786 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:23.111740    5304 cpu_manager.go:414] "containerMap: removing container" podUID="529d1f83-57ea-433f-aa66-ee16157950ca" containerName="istio-proxy"
Nov 02 15:46:23.111786 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:23.111748    5304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="529d1f83-57ea-433f-aa66-ee16157950ca" containerName="istio-proxy"
Nov 02 15:46:23.112049 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:23.111796    5304 cpu_manager.go:414] "containerMap: removing container" podUID="529d1f83-57ea-433f-aa66-ee16157950ca" containerName="istio-proxy"
Nov 02 15:46:23.112049 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:23.111803    5304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="529d1f83-57ea-433f-aa66-ee16157950ca" containerName="istio-proxy"
Nov 02 15:46:23.112049 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:23.111830    5304 cpu_manager.go:414] "containerMap: removing container" podUID="20517696-bd32-4370-a351-67c838f8ccf6" containerName="istio-proxy"
Nov 02 15:46:23.112049 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:23.111839    5304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="20517696-bd32-4370-a351-67c838f8ccf6" containerName="istio-proxy"
Nov 02 15:46:23.112049 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:23.111977    5304 memory_manager.go:365] "RemoveStaleState removing state" podUID="529d1f83-57ea-433f-aa66-ee16157950ca" containerName="istio-proxy"
Nov 02 15:46:23.112049 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:23.111991    5304 memory_manager.go:365] "RemoveStaleState removing state" podUID="20517696-bd32-4370-a351-67c838f8ccf6" containerName="istio-proxy"
Nov 02 15:46:23.112049 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:23.112001    5304 memory_manager.go:365] "RemoveStaleState removing state" podUID="f75de3cc-f69f-4dfc-9351-9e75a08723b3" containerName="istio-proxy"
Nov 02 15:46:23.112049 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:23.112013    5304 memory_manager.go:365] "RemoveStaleState removing state" podUID="24182bc4-1f65-4ddf-8e20-0d5e7bb2bc29" containerName="istio-proxy"
Nov 02 15:46:23.117511 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:23.117482    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e80c77-2tvs6"
Nov 02 15:46:23.132559 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:23.132310    5304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 15:46:23.133428 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:23.133405    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e80c77-2tvs6"]
Nov 02 15:46:23.140811 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:23.140786    5304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-m5926\""
Nov 02 15:46:23.232141 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:23.229172    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8bac17da-adf2-45d3-88d2-57c9e56b3db6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-e80c77-2tvs6\" (UID: \"8bac17da-adf2-45d3-88d2-57c9e56b3db6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e80c77-2tvs6"
Nov 02 15:46:23.232648 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:23.232597    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7s6s9\" (UniqueName: \"kubernetes.io/projected/8bac17da-adf2-45d3-88d2-57c9e56b3db6-kube-api-access-7s6s9\") pod \"maas-api-revoke-keys-e2e-ipp-b-e80c77-2tvs6\" (UID: \"8bac17da-adf2-45d3-88d2-57c9e56b3db6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e80c77-2tvs6"
Nov 02 15:46:23.233240 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:23.233217    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8bac17da-adf2-45d3-88d2-57c9e56b3db6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-e80c77-2tvs6\" (UID: \"8bac17da-adf2-45d3-88d2-57c9e56b3db6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e80c77-2tvs6"
Nov 02 15:46:23.335698 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:23.335655    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8bac17da-adf2-45d3-88d2-57c9e56b3db6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-e80c77-2tvs6\" (UID: \"8bac17da-adf2-45d3-88d2-57c9e56b3db6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e80c77-2tvs6"
Nov 02 15:46:23.335906 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:23.335755    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7s6s9\" (UniqueName: \"kubernetes.io/projected/8bac17da-adf2-45d3-88d2-57c9e56b3db6-kube-api-access-7s6s9\") pod \"maas-api-revoke-keys-e2e-ipp-b-e80c77-2tvs6\" (UID: \"8bac17da-adf2-45d3-88d2-57c9e56b3db6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e80c77-2tvs6"
Nov 02 15:46:23.335906 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:23.335844    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8bac17da-adf2-45d3-88d2-57c9e56b3db6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-e80c77-2tvs6\" (UID: \"8bac17da-adf2-45d3-88d2-57c9e56b3db6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e80c77-2tvs6"
Nov 02 15:46:23.336382 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:23.336351    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8bac17da-adf2-45d3-88d2-57c9e56b3db6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-e80c77-2tvs6\" (UID: \"8bac17da-adf2-45d3-88d2-57c9e56b3db6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e80c77-2tvs6"
Nov 02 15:46:23.338223 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:23.338201    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8bac17da-adf2-45d3-88d2-57c9e56b3db6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-e80c77-2tvs6\" (UID: \"8bac17da-adf2-45d3-88d2-57c9e56b3db6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e80c77-2tvs6"
Nov 02 15:46:23.359417 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:23.359391    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7s6s9\" (UniqueName: \"kubernetes.io/projected/8bac17da-adf2-45d3-88d2-57c9e56b3db6-kube-api-access-7s6s9\") pod \"maas-api-revoke-keys-e2e-ipp-b-e80c77-2tvs6\" (UID: \"8bac17da-adf2-45d3-88d2-57c9e56b3db6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e80c77-2tvs6"
Nov 02 15:46:23.431862 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:23.431779    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e80c77-2tvs6"
Nov 02 15:46:23.703275 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:23.703195    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-797488/sub-test-model-797488-kserve-79764959f-k4scd"]
Nov 02 15:46:23.747675 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:23.747645    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e80c77-2tvs6"]
Nov 02 15:46:23.749486 ip-10-0-135-138 kubenswrapper[5304]: W1102 15:46:23.749458    5304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8bac17da_adf2_45d3_88d2_57c9e56b3db6.slice/crio-b3bd29f736609ec32c0d887db52165dd2736e0e0ee47512f4266359af9d6e5da WatchSource:0}: Error finding container b3bd29f736609ec32c0d887db52165dd2736e0e0ee47512f4266359af9d6e5da: Status 404 returned error can't find the container with id b3bd29f736609ec32c0d887db52165dd2736e0e0ee47512f4266359af9d6e5da
Nov 02 15:46:24.024580 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:24.024491    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e80c77-2tvs6" event={"ID":"8bac17da-adf2-45d3-88d2-57c9e56b3db6","Type":"ContainerStarted","Data":"b3bd29f736609ec32c0d887db52165dd2736e0e0ee47512f4266359af9d6e5da"}
Nov 02 15:46:24.751289 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:24.751261    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-501738-767988665-nwzcr"]
Nov 02 15:46:24.787850 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:24.787812    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-501738-864d5f868c-2ssc8"]
Nov 02 15:46:24.800723 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:46:24.800692    5304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k6fjk" containerName="extractor"
Nov 02 15:46:24.861116 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:24.861080    5304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-d79991-openshift-default-74c857f7cc-p2snd"]
Nov 02 15:46:24.862059 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:24.862030    5304 memory_manager.go:365] "RemoveStaleState removing state" podUID="529d1f83-57ea-433f-aa66-ee16157950ca" containerName="istio-proxy"
Nov 02 15:46:24.865815 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:24.865791    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-d79991-openshift-default-74c857f7cc-p2snd"
Nov 02 15:46:24.894406 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:24.888244    5304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-d79991-openshift-default-dockercfg-mm7kr\""
Nov 02 15:46:24.908902 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:24.905356    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-d79991-openshift-default-74c857f7cc-p2snd"]
Nov 02 15:46:24.990966 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:24.990879    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1020808c-e760-46e7-99e8-66932fca1956-istiod-ca-cert\") pod \"e2e-ipp-a-d79991-openshift-default-74c857f7cc-p2snd\" (UID: \"1020808c-e760-46e7-99e8-66932fca1956\") " pod="openshift-ingress/e2e-ipp-a-d79991-openshift-default-74c857f7cc-p2snd"
Nov 02 15:46:24.990966 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:24.990923    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1020808c-e760-46e7-99e8-66932fca1956-istio-envoy\") pod \"e2e-ipp-a-d79991-openshift-default-74c857f7cc-p2snd\" (UID: \"1020808c-e760-46e7-99e8-66932fca1956\") " pod="openshift-ingress/e2e-ipp-a-d79991-openshift-default-74c857f7cc-p2snd"
Nov 02 15:46:24.991222 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:24.990989    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t67w7\" (UniqueName: \"kubernetes.io/projected/1020808c-e760-46e7-99e8-66932fca1956-kube-api-access-t67w7\") pod \"e2e-ipp-a-d79991-openshift-default-74c857f7cc-p2snd\" (UID: \"1020808c-e760-46e7-99e8-66932fca1956\") " pod="openshift-ingress/e2e-ipp-a-d79991-openshift-default-74c857f7cc-p2snd"
Nov 02 15:46:24.991222 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:24.991105    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1020808c-e760-46e7-99e8-66932fca1956-istio-podinfo\") pod \"e2e-ipp-a-d79991-openshift-default-74c857f7cc-p2snd\" (UID: \"1020808c-e760-46e7-99e8-66932fca1956\") " pod="openshift-ingress/e2e-ipp-a-d79991-openshift-default-74c857f7cc-p2snd"
Nov 02 15:46:24.991222 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:24.991199    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1020808c-e760-46e7-99e8-66932fca1956-istio-data\") pod \"e2e-ipp-a-d79991-openshift-default-74c857f7cc-p2snd\" (UID: \"1020808c-e760-46e7-99e8-66932fca1956\") " pod="openshift-ingress/e2e-ipp-a-d79991-openshift-default-74c857f7cc-p2snd"
Nov 02 15:46:24.991355 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:24.991228    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1020808c-e760-46e7-99e8-66932fca1956-istio-token\") pod \"e2e-ipp-a-d79991-openshift-default-74c857f7cc-p2snd\" (UID: \"1020808c-e760-46e7-99e8-66932fca1956\") " pod="openshift-ingress/e2e-ipp-a-d79991-openshift-default-74c857f7cc-p2snd"
Nov 02 15:46:24.991355 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:24.991268    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1020808c-e760-46e7-99e8-66932fca1956-credential-socket\") pod \"e2e-ipp-a-d79991-openshift-default-74c857f7cc-p2snd\" (UID: \"1020808c-e760-46e7-99e8-66932fca1956\") " pod="openshift-ingress/e2e-ipp-a-d79991-openshift-default-74c857f7cc-p2snd"
Nov 02 15:46:24.991355 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:24.991296    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1020808c-e760-46e7-99e8-66932fca1956-workload-certs\") pod \"e2e-ipp-a-d79991-openshift-default-74c857f7cc-p2snd\" (UID: \"1020808c-e760-46e7-99e8-66932fca1956\") " pod="openshift-ingress/e2e-ipp-a-d79991-openshift-default-74c857f7cc-p2snd"
Nov 02 15:46:24.991472 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:24.991387    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1020808c-e760-46e7-99e8-66932fca1956-workload-socket\") pod \"e2e-ipp-a-d79991-openshift-default-74c857f7cc-p2snd\" (UID: \"1020808c-e760-46e7-99e8-66932fca1956\") " pod="openshift-ingress/e2e-ipp-a-d79991-openshift-default-74c857f7cc-p2snd"
Nov 02 15:46:25.033796 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:25.033749    5304 generic.go:356] "Generic (PLEG): container finished" podID="8bac17da-adf2-45d3-88d2-57c9e56b3db6" containerID="c0cd3ff1c2c73e720b00f22565cea7da049e32138f39195f62aa04488adde269" exitCode=0
Nov 02 15:46:25.033974 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:25.033820    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e80c77-2tvs6" event={"ID":"8bac17da-adf2-45d3-88d2-57c9e56b3db6","Type":"ContainerDied","Data":"c0cd3ff1c2c73e720b00f22565cea7da049e32138f39195f62aa04488adde269"}
Nov 02 15:46:25.093272 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:25.093222    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1020808c-e760-46e7-99e8-66932fca1956-istiod-ca-cert\") pod \"e2e-ipp-a-d79991-openshift-default-74c857f7cc-p2snd\" (UID: \"1020808c-e760-46e7-99e8-66932fca1956\") " pod="openshift-ingress/e2e-ipp-a-d79991-openshift-default-74c857f7cc-p2snd"
Nov 02 15:46:25.093754 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:25.093678    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1020808c-e760-46e7-99e8-66932fca1956-istio-envoy\") pod \"e2e-ipp-a-d79991-openshift-default-74c857f7cc-p2snd\" (UID: \"1020808c-e760-46e7-99e8-66932fca1956\") " pod="openshift-ingress/e2e-ipp-a-d79991-openshift-default-74c857f7cc-p2snd"
Nov 02 15:46:25.094790 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:25.094747    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t67w7\" (UniqueName: \"kubernetes.io/projected/1020808c-e760-46e7-99e8-66932fca1956-kube-api-access-t67w7\") pod \"e2e-ipp-a-d79991-openshift-default-74c857f7cc-p2snd\" (UID: \"1020808c-e760-46e7-99e8-66932fca1956\") " pod="openshift-ingress/e2e-ipp-a-d79991-openshift-default-74c857f7cc-p2snd"
Nov 02 15:46:25.095379 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:25.095325    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1020808c-e760-46e7-99e8-66932fca1956-istio-podinfo\") pod \"e2e-ipp-a-d79991-openshift-default-74c857f7cc-p2snd\" (UID: \"1020808c-e760-46e7-99e8-66932fca1956\") " pod="openshift-ingress/e2e-ipp-a-d79991-openshift-default-74c857f7cc-p2snd"
Nov 02 15:46:25.095921 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:25.095822    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1020808c-e760-46e7-99e8-66932fca1956-istio-data\") pod \"e2e-ipp-a-d79991-openshift-default-74c857f7cc-p2snd\" (UID: \"1020808c-e760-46e7-99e8-66932fca1956\") " pod="openshift-ingress/e2e-ipp-a-d79991-openshift-default-74c857f7cc-p2snd"
Nov 02 15:46:25.096176 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:25.096119    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1020808c-e760-46e7-99e8-66932fca1956-istio-token\") pod \"e2e-ipp-a-d79991-openshift-default-74c857f7cc-p2snd\" (UID: \"1020808c-e760-46e7-99e8-66932fca1956\") " pod="openshift-ingress/e2e-ipp-a-d79991-openshift-default-74c857f7cc-p2snd"
Nov 02 15:46:25.096374 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:25.096345    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1020808c-e760-46e7-99e8-66932fca1956-credential-socket\") pod \"e2e-ipp-a-d79991-openshift-default-74c857f7cc-p2snd\" (UID: \"1020808c-e760-46e7-99e8-66932fca1956\") " pod="openshift-ingress/e2e-ipp-a-d79991-openshift-default-74c857f7cc-p2snd"
Nov 02 15:46:25.096665 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:25.096618    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1020808c-e760-46e7-99e8-66932fca1956-workload-certs\") pod \"e2e-ipp-a-d79991-openshift-default-74c857f7cc-p2snd\" (UID: \"1020808c-e760-46e7-99e8-66932fca1956\") " pod="openshift-ingress/e2e-ipp-a-d79991-openshift-default-74c857f7cc-p2snd"
Nov 02 15:46:25.096849 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:25.096824    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1020808c-e760-46e7-99e8-66932fca1956-workload-socket\") pod \"e2e-ipp-a-d79991-openshift-default-74c857f7cc-p2snd\" (UID: \"1020808c-e760-46e7-99e8-66932fca1956\") " pod="openshift-ingress/e2e-ipp-a-d79991-openshift-default-74c857f7cc-p2snd"
Nov 02 15:46:25.117149 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:25.102799    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1020808c-e760-46e7-99e8-66932fca1956-credential-socket\") pod \"e2e-ipp-a-d79991-openshift-default-74c857f7cc-p2snd\" (UID: \"1020808c-e760-46e7-99e8-66932fca1956\") " pod="openshift-ingress/e2e-ipp-a-d79991-openshift-default-74c857f7cc-p2snd"
Nov 02 15:46:25.117149 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:25.103107    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1020808c-e760-46e7-99e8-66932fca1956-workload-certs\") pod \"e2e-ipp-a-d79991-openshift-default-74c857f7cc-p2snd\" (UID: \"1020808c-e760-46e7-99e8-66932fca1956\") " pod="openshift-ingress/e2e-ipp-a-d79991-openshift-default-74c857f7cc-p2snd"
Nov 02 15:46:25.117149 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:25.103631    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1020808c-e760-46e7-99e8-66932fca1956-istio-data\") pod \"e2e-ipp-a-d79991-openshift-default-74c857f7cc-p2snd\" (UID: \"1020808c-e760-46e7-99e8-66932fca1956\") " pod="openshift-ingress/e2e-ipp-a-d79991-openshift-default-74c857f7cc-p2snd"
Nov 02 15:46:25.117791 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:25.117760    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1020808c-e760-46e7-99e8-66932fca1956-workload-socket\") pod \"e2e-ipp-a-d79991-openshift-default-74c857f7cc-p2snd\" (UID: \"1020808c-e760-46e7-99e8-66932fca1956\") " pod="openshift-ingress/e2e-ipp-a-d79991-openshift-default-74c857f7cc-p2snd"
Nov 02 15:46:25.129150 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:25.126207    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1020808c-e760-46e7-99e8-66932fca1956-istiod-ca-cert\") pod \"e2e-ipp-a-d79991-openshift-default-74c857f7cc-p2snd\" (UID: \"1020808c-e760-46e7-99e8-66932fca1956\") " pod="openshift-ingress/e2e-ipp-a-d79991-openshift-default-74c857f7cc-p2snd"
Nov 02 15:46:25.153149 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:25.149635    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1020808c-e760-46e7-99e8-66932fca1956-istio-podinfo\") pod \"e2e-ipp-a-d79991-openshift-default-74c857f7cc-p2snd\" (UID: \"1020808c-e760-46e7-99e8-66932fca1956\") " pod="openshift-ingress/e2e-ipp-a-d79991-openshift-default-74c857f7cc-p2snd"
Nov 02 15:46:25.166450 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:25.166411    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1020808c-e760-46e7-99e8-66932fca1956-istio-envoy\") pod \"e2e-ipp-a-d79991-openshift-default-74c857f7cc-p2snd\" (UID: \"1020808c-e760-46e7-99e8-66932fca1956\") " pod="openshift-ingress/e2e-ipp-a-d79991-openshift-default-74c857f7cc-p2snd"
Nov 02 15:46:25.195306 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:25.195042    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1020808c-e760-46e7-99e8-66932fca1956-istio-token\") pod \"e2e-ipp-a-d79991-openshift-default-74c857f7cc-p2snd\" (UID: \"1020808c-e760-46e7-99e8-66932fca1956\") " pod="openshift-ingress/e2e-ipp-a-d79991-openshift-default-74c857f7cc-p2snd"
Nov 02 15:46:25.196570 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:25.196510    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t67w7\" (UniqueName: \"kubernetes.io/projected/1020808c-e760-46e7-99e8-66932fca1956-kube-api-access-t67w7\") pod \"e2e-ipp-a-d79991-openshift-default-74c857f7cc-p2snd\" (UID: \"1020808c-e760-46e7-99e8-66932fca1956\") " pod="openshift-ingress/e2e-ipp-a-d79991-openshift-default-74c857f7cc-p2snd"
Nov 02 15:46:25.213748 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:25.213710    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-d79991-openshift-default-74c857f7cc-p2snd"
Nov 02 15:46:25.579527 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:25.579447    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-d79991-openshift-default-74c857f7cc-p2snd"]
Nov 02 15:46:25.633174 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:25.631997    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:46:25.633174 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:25.632108    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:46:25.633174 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:25.632179    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:46:25.962494 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:25.962462    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f0039c-6c5bf99c7f-62jkr"
Nov 02 15:46:25.962984 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:25.962618    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-f0039c-b78956c9-vm85j"
Nov 02 15:46:26.049120 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:26.048447    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-d79991-openshift-default-74c857f7cc-p2snd" event={"ID":"1020808c-e760-46e7-99e8-66932fca1956","Type":"ContainerStarted","Data":"554b43614667dd9cb4b1f94d5281367be991545f9ad45480363d8879d52c5001"}
Nov 02 15:46:26.049120 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:26.048509    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-d79991-openshift-default-74c857f7cc-p2snd" event={"ID":"1020808c-e760-46e7-99e8-66932fca1956","Type":"ContainerStarted","Data":"97d29ccda53d96ac36fc9fe56ca9c00fea498d0f8793064b8e700feb2f52268e"}
Nov 02 15:46:26.101724 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:26.101612    5304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-d79991-openshift-default-74c857f7cc-p2snd" podStartSLOduration=2.101591722 podStartE2EDuration="2.101591722s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:46:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:46:26.09291421 +0000 UTC m=+2986.086738620" watchObservedRunningTime="2026-11-02 15:46:26.101591722 +0000 UTC m=+2986.095416133"
Nov 02 15:46:26.223657 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:26.214803    5304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-d79991-openshift-default-74c857f7cc-p2snd"
Nov 02 15:46:26.226239 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:26.225853    5304 patch_prober.go:29] interesting pod/e2e-ipp-a-d79991-openshift-default-74c857f7cc-p2snd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused" start-of-body=
Nov 02 15:46:26.226239 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:26.225935    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-d79991-openshift-default-74c857f7cc-p2snd" podUID="1020808c-e760-46e7-99e8-66932fca1956" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused"
Nov 02 15:46:26.281698 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:26.281664    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e80c77-2tvs6"
Nov 02 15:46:26.445045 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:26.444950    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8bac17da-adf2-45d3-88d2-57c9e56b3db6-kube-api-access-7s6s9\" (UniqueName: \"kubernetes.io/projected/8bac17da-adf2-45d3-88d2-57c9e56b3db6-kube-api-access-7s6s9\") pod \"8bac17da-adf2-45d3-88d2-57c9e56b3db6\" (UID: \"8bac17da-adf2-45d3-88d2-57c9e56b3db6\") "
Nov 02 15:46:26.445262 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:26.445081    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8bac17da-adf2-45d3-88d2-57c9e56b3db6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8bac17da-adf2-45d3-88d2-57c9e56b3db6-maas-api-openshift-ca\") pod \"8bac17da-adf2-45d3-88d2-57c9e56b3db6\" (UID: \"8bac17da-adf2-45d3-88d2-57c9e56b3db6\") "
Nov 02 15:46:26.445262 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:26.445181    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8bac17da-adf2-45d3-88d2-57c9e56b3db6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8bac17da-adf2-45d3-88d2-57c9e56b3db6-maas-api-xks-ca\") pod \"8bac17da-adf2-45d3-88d2-57c9e56b3db6\" (UID: \"8bac17da-adf2-45d3-88d2-57c9e56b3db6\") "
Nov 02 15:46:26.445461 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:26.445435    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8bac17da-adf2-45d3-88d2-57c9e56b3db6-maas-api-openshift-ca" pod "8bac17da-adf2-45d3-88d2-57c9e56b3db6" (UID: "8bac17da-adf2-45d3-88d2-57c9e56b3db6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:46:26.445762 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:26.445748    5304 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8bac17da-adf2-45d3-88d2-57c9e56b3db6-maas-api-openshift-ca\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:46:26.447365 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:26.447339    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8bac17da-adf2-45d3-88d2-57c9e56b3db6-kube-api-access-7s6s9" pod "8bac17da-adf2-45d3-88d2-57c9e56b3db6" (UID: "8bac17da-adf2-45d3-88d2-57c9e56b3db6"). InnerVolumeSpecName "kube-api-access-7s6s9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:46:26.447365 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:26.447339    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8bac17da-adf2-45d3-88d2-57c9e56b3db6-maas-api-xks-ca" pod "8bac17da-adf2-45d3-88d2-57c9e56b3db6" (UID: "8bac17da-adf2-45d3-88d2-57c9e56b3db6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:46:26.547556 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:26.547524    5304 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8bac17da-adf2-45d3-88d2-57c9e56b3db6-maas-api-xks-ca\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:46:26.547556 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:26.547553    5304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7s6s9\" (UniqueName: \"kubernetes.io/projected/8bac17da-adf2-45d3-88d2-57c9e56b3db6-kube-api-access-7s6s9\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:46:27.056079 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:27.056047    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e80c77-2tvs6"
Nov 02 15:46:27.056530 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:27.056087    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e80c77-2tvs6" event={"ID":"8bac17da-adf2-45d3-88d2-57c9e56b3db6","Type":"ContainerDied","Data":"b3bd29f736609ec32c0d887db52165dd2736e0e0ee47512f4266359af9d6e5da"}
Nov 02 15:46:27.056530 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:27.056199    5304 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b3bd29f736609ec32c0d887db52165dd2736e0e0ee47512f4266359af9d6e5da"
Nov 02 15:46:27.219222 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:27.219193    5304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-d79991-openshift-default-74c857f7cc-p2snd"
Nov 02 15:46:28.063046 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:28.062966    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-d79991-openshift-default-74c857f7cc-p2snd"
Nov 02 15:46:28.064121 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:28.064102    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-d79991-openshift-default-74c857f7cc-p2snd"
Nov 02 15:46:29.752607 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:29.752567    5304 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-501738-767988665-nwzcr" podUID="6eab7ac5-dbb2-4b5d-adac-86e3c425c011" containerName="payload-processing" containerID="cri-o://b4997936ebf8111dd7fcd710eee1fdb7909ab32fd4aeb56f3760f614f25f6616" gracePeriod=25
Nov 02 15:46:29.788268 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:29.788215    5304 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-501738-864d5f868c-2ssc8" podUID="5ae97c11-e62c-4022-ace2-03a398ff3854" containerName="payload-pre-processing" containerID="cri-o://4c4d983a44d0823251a3d5f5eb2c8b42b6389365b1627bf268c2d393ebcac1f7" gracePeriod=25
Nov 02 15:46:30.467911 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:30.467880    5304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-d79991-7765697955-29mq6"]
Nov 02 15:46:30.468518 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:30.468503    5304 cpu_manager.go:414] "containerMap: removing container" podUID="8bac17da-adf2-45d3-88d2-57c9e56b3db6" containerName="revoke-keys"
Nov 02 15:46:30.468576 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:30.468519    5304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8bac17da-adf2-45d3-88d2-57c9e56b3db6" containerName="revoke-keys"
Nov 02 15:46:30.468643 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:30.468634    5304 memory_manager.go:365] "RemoveStaleState removing state" podUID="8bac17da-adf2-45d3-88d2-57c9e56b3db6" containerName="revoke-keys"
Nov 02 15:46:30.472271 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:30.472256    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d79991-7765697955-29mq6"
Nov 02 15:46:30.491165 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:30.491118    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-d79991-7765697955-29mq6"]
Nov 02 15:46:30.498759 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:30.498732    5304 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-d79991\""
Nov 02 15:46:30.498759 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:30.498745    5304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-d79991-dockercfg-qqcsf\""
Nov 02 15:46:30.514384 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:30.514347    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dw7cb\" (UniqueName: \"kubernetes.io/projected/ad1b2ff2-4594-4cf3-88a7-199030bae2a5-kube-api-access-dw7cb\") pod \"payload-pre-processing-e2e-ipp-a-d79991-7765697955-29mq6\" (UID: \"ad1b2ff2-4594-4cf3-88a7-199030bae2a5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d79991-7765697955-29mq6"
Nov 02 15:46:30.514539 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:30.514476    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad1b2ff2-4594-4cf3-88a7-199030bae2a5-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-d79991-7765697955-29mq6\" (UID: \"ad1b2ff2-4594-4cf3-88a7-199030bae2a5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d79991-7765697955-29mq6"
Nov 02 15:46:30.616493 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:30.616441    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dw7cb\" (UniqueName: \"kubernetes.io/projected/ad1b2ff2-4594-4cf3-88a7-199030bae2a5-kube-api-access-dw7cb\") pod \"payload-pre-processing-e2e-ipp-a-d79991-7765697955-29mq6\" (UID: \"ad1b2ff2-4594-4cf3-88a7-199030bae2a5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d79991-7765697955-29mq6"
Nov 02 15:46:30.616819 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:30.616797    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad1b2ff2-4594-4cf3-88a7-199030bae2a5-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-d79991-7765697955-29mq6\" (UID: \"ad1b2ff2-4594-4cf3-88a7-199030bae2a5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d79991-7765697955-29mq6"
Nov 02 15:46:30.643655 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:30.643617    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad1b2ff2-4594-4cf3-88a7-199030bae2a5-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-d79991-7765697955-29mq6\" (UID: \"ad1b2ff2-4594-4cf3-88a7-199030bae2a5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d79991-7765697955-29mq6"
Nov 02 15:46:30.655318 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:30.655282    5304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-d79991-56bd8b5955-jkfrr"]
Nov 02 15:46:30.665610 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:30.665578    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-d79991-56bd8b5955-jkfrr"
Nov 02 15:46:30.696168 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:30.686970    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-d79991-56bd8b5955-jkfrr"]
Nov 02 15:46:30.834192 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:30.824586    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dw7cb\" (UniqueName: \"kubernetes.io/projected/ad1b2ff2-4594-4cf3-88a7-199030bae2a5-kube-api-access-dw7cb\") pod \"payload-pre-processing-e2e-ipp-a-d79991-7765697955-29mq6\" (UID: \"ad1b2ff2-4594-4cf3-88a7-199030bae2a5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d79991-7765697955-29mq6"
Nov 02 15:46:30.834192 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:30.825850    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dk45l\" (UniqueName: \"kubernetes.io/projected/588da214-2720-47e6-b537-f90977db6a92-kube-api-access-dk45l\") pod \"payload-processing-e2e-ipp-a-d79991-56bd8b5955-jkfrr\" (UID: \"588da214-2720-47e6-b537-f90977db6a92\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-d79991-56bd8b5955-jkfrr"
Nov 02 15:46:30.834192 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:30.825935    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/588da214-2720-47e6-b537-f90977db6a92-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-d79991-56bd8b5955-jkfrr\" (UID: \"588da214-2720-47e6-b537-f90977db6a92\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-d79991-56bd8b5955-jkfrr"
Nov 02 15:46:30.927573 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:30.927535    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dk45l\" (UniqueName: \"kubernetes.io/projected/588da214-2720-47e6-b537-f90977db6a92-kube-api-access-dk45l\") pod \"payload-processing-e2e-ipp-a-d79991-56bd8b5955-jkfrr\" (UID: \"588da214-2720-47e6-b537-f90977db6a92\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-d79991-56bd8b5955-jkfrr"
Nov 02 15:46:30.927791 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:30.927632    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/588da214-2720-47e6-b537-f90977db6a92-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-d79991-56bd8b5955-jkfrr\" (UID: \"588da214-2720-47e6-b537-f90977db6a92\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-d79991-56bd8b5955-jkfrr"
Nov 02 15:46:30.928492 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:30.928470    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/588da214-2720-47e6-b537-f90977db6a92-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-d79991-56bd8b5955-jkfrr\" (UID: \"588da214-2720-47e6-b537-f90977db6a92\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-d79991-56bd8b5955-jkfrr"
Nov 02 15:46:30.951505 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:30.951472    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dk45l\" (UniqueName: \"kubernetes.io/projected/588da214-2720-47e6-b537-f90977db6a92-kube-api-access-dk45l\") pod \"payload-processing-e2e-ipp-a-d79991-56bd8b5955-jkfrr\" (UID: \"588da214-2720-47e6-b537-f90977db6a92\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-d79991-56bd8b5955-jkfrr"
Nov 02 15:46:31.086200 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:31.086098    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d79991-7765697955-29mq6"
Nov 02 15:46:31.140164 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:31.140097    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-d79991-56bd8b5955-jkfrr"
Nov 02 15:46:31.391864 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:31.391781    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-d79991-7765697955-29mq6"]
Nov 02 15:46:31.394917 ip-10-0-135-138 kubenswrapper[5304]: W1102 15:46:31.394889    5304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad1b2ff2_4594_4cf3_88a7_199030bae2a5.slice/crio-c7fed1f55eff2d8af7f1fd4607b3e829a94acb25ff5344bd23ec54e68c68d89e WatchSource:0}: Error finding container c7fed1f55eff2d8af7f1fd4607b3e829a94acb25ff5344bd23ec54e68c68d89e: Status 404 returned error can't find the container with id c7fed1f55eff2d8af7f1fd4607b3e829a94acb25ff5344bd23ec54e68c68d89e
Nov 02 15:46:31.468083 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:31.468059    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-d79991-56bd8b5955-jkfrr"]
Nov 02 15:46:31.470505 ip-10-0-135-138 kubenswrapper[5304]: W1102 15:46:31.470478    5304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod588da214_2720_47e6_b537_f90977db6a92.slice/crio-ff79915d62e3fc26515b967f41bbd7256f61008815aeea6ecb50ce315a342dc8 WatchSource:0}: Error finding container ff79915d62e3fc26515b967f41bbd7256f61008815aeea6ecb50ce315a342dc8: Status 404 returned error can't find the container with id ff79915d62e3fc26515b967f41bbd7256f61008815aeea6ecb50ce315a342dc8
Nov 02 15:46:32.090595 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:32.090565    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d79991-7765697955-29mq6" event={"ID":"ad1b2ff2-4594-4cf3-88a7-199030bae2a5","Type":"ContainerStarted","Data":"dff1da1027757d871240b7d5650073e55bb49a47b2b39b45a14a23a992fb7afc"}
Nov 02 15:46:32.091094 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:32.090626    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d79991-7765697955-29mq6"
Nov 02 15:46:32.091094 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:32.090648    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d79991-7765697955-29mq6" event={"ID":"ad1b2ff2-4594-4cf3-88a7-199030bae2a5","Type":"ContainerStarted","Data":"c7fed1f55eff2d8af7f1fd4607b3e829a94acb25ff5344bd23ec54e68c68d89e"}
Nov 02 15:46:32.092233 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:32.092209    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-d79991-56bd8b5955-jkfrr" event={"ID":"588da214-2720-47e6-b537-f90977db6a92","Type":"ContainerStarted","Data":"709cc3b9141e57728469a4574d2ab3753af7707308fe8d6b68b464e46bf37910"}
Nov 02 15:46:32.092288 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:32.092252    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-d79991-56bd8b5955-jkfrr" event={"ID":"588da214-2720-47e6-b537-f90977db6a92","Type":"ContainerStarted","Data":"ff79915d62e3fc26515b967f41bbd7256f61008815aeea6ecb50ce315a342dc8"}
Nov 02 15:46:32.092288 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:32.092283    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-d79991-56bd8b5955-jkfrr"
Nov 02 15:46:32.131622 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:32.131564    5304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d79991-7765697955-29mq6" podStartSLOduration=2.131551325 podStartE2EDuration="2.131551325s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:46:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:46:32.130669099 +0000 UTC m=+2992.124493508" watchObservedRunningTime="2026-11-02 15:46:32.131551325 +0000 UTC m=+2992.125375736"
Nov 02 15:46:32.173471 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:32.173119    5304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-d79991-56bd8b5955-jkfrr" podStartSLOduration=2.173099979 podStartE2EDuration="2.173099979s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:46:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:46:32.171620734 +0000 UTC m=+2992.165445142" watchObservedRunningTime="2026-11-02 15:46:32.173099979 +0000 UTC m=+2992.166924392"
Nov 02 15:46:38.290559 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:38.290522    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-501738-864d5f868c-2ssc8" podUID="5ae97c11-e62c-4022-ace2-03a398ff3854" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 15:46:38.290975 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:38.290521    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-501738-767988665-nwzcr" podUID="6eab7ac5-dbb2-4b5d-adac-86e3c425c011" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 15:46:38.565219 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:38.565121    5304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-24255b-5555cd4dc9-jmck4"]
Nov 02 15:46:38.569218 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:38.569196    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-24255b-5555cd4dc9-jmck4"
Nov 02 15:46:38.590594 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:38.590562    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-24255b-5555cd4dc9-jmck4"]
Nov 02 15:46:38.594787 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:38.594762    5304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-24255b-serving-cert\""
Nov 02 15:46:38.640236 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:38.640201    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d6ccq\" (UniqueName: \"kubernetes.io/projected/774e4952-c994-4374-aa62-3d67cc777d5a-kube-api-access-d6ccq\") pod \"maas-api-e2e-ipp-b-24255b-5555cd4dc9-jmck4\" (UID: \"774e4952-c994-4374-aa62-3d67cc777d5a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-24255b-5555cd4dc9-jmck4"
Nov 02 15:46:38.640471 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:38.640279    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/774e4952-c994-4374-aa62-3d67cc777d5a-metrics-certs\") pod \"maas-api-e2e-ipp-b-24255b-5555cd4dc9-jmck4\" (UID: \"774e4952-c994-4374-aa62-3d67cc777d5a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-24255b-5555cd4dc9-jmck4"
Nov 02 15:46:38.640625 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:38.640550    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/774e4952-c994-4374-aa62-3d67cc777d5a-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-24255b-5555cd4dc9-jmck4\" (UID: \"774e4952-c994-4374-aa62-3d67cc777d5a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-24255b-5555cd4dc9-jmck4"
Nov 02 15:46:38.640698 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:38.640645    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/774e4952-c994-4374-aa62-3d67cc777d5a-maas-api-tls\") pod \"maas-api-e2e-ipp-b-24255b-5555cd4dc9-jmck4\" (UID: \"774e4952-c994-4374-aa62-3d67cc777d5a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-24255b-5555cd4dc9-jmck4"
Nov 02 15:46:38.739586 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:38.739542    5304 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w2-797488/sub-test-model-797488-kserve-79764959f-k4scd" podUID="2d6e474d-c5d3-4c69-b12f-993ff23910c1" containerName="main" containerID="cri-o://ed3dfc44a4c9b27ff03ddc35712aa5e43e60362546463f53adf9ea3e0556a7eb" gracePeriod=45
Nov 02 15:46:38.741657 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:38.741630    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/774e4952-c994-4374-aa62-3d67cc777d5a-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-24255b-5555cd4dc9-jmck4\" (UID: \"774e4952-c994-4374-aa62-3d67cc777d5a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-24255b-5555cd4dc9-jmck4"
Nov 02 15:46:38.741783 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:38.741694    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/774e4952-c994-4374-aa62-3d67cc777d5a-maas-api-tls\") pod \"maas-api-e2e-ipp-b-24255b-5555cd4dc9-jmck4\" (UID: \"774e4952-c994-4374-aa62-3d67cc777d5a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-24255b-5555cd4dc9-jmck4"
Nov 02 15:46:38.741848 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:38.741800    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d6ccq\" (UniqueName: \"kubernetes.io/projected/774e4952-c994-4374-aa62-3d67cc777d5a-kube-api-access-d6ccq\") pod \"maas-api-e2e-ipp-b-24255b-5555cd4dc9-jmck4\" (UID: \"774e4952-c994-4374-aa62-3d67cc777d5a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-24255b-5555cd4dc9-jmck4"
Nov 02 15:46:38.741906 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:38.741886    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/774e4952-c994-4374-aa62-3d67cc777d5a-metrics-certs\") pod \"maas-api-e2e-ipp-b-24255b-5555cd4dc9-jmck4\" (UID: \"774e4952-c994-4374-aa62-3d67cc777d5a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-24255b-5555cd4dc9-jmck4"
Nov 02 15:46:38.742645 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:38.742442    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/774e4952-c994-4374-aa62-3d67cc777d5a-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-24255b-5555cd4dc9-jmck4\" (UID: \"774e4952-c994-4374-aa62-3d67cc777d5a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-24255b-5555cd4dc9-jmck4"
Nov 02 15:46:38.744368 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:38.744341    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/774e4952-c994-4374-aa62-3d67cc777d5a-maas-api-tls\") pod \"maas-api-e2e-ipp-b-24255b-5555cd4dc9-jmck4\" (UID: \"774e4952-c994-4374-aa62-3d67cc777d5a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-24255b-5555cd4dc9-jmck4"
Nov 02 15:46:38.744892 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:38.744868    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/774e4952-c994-4374-aa62-3d67cc777d5a-metrics-certs\") pod \"maas-api-e2e-ipp-b-24255b-5555cd4dc9-jmck4\" (UID: \"774e4952-c994-4374-aa62-3d67cc777d5a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-24255b-5555cd4dc9-jmck4"
Nov 02 15:46:38.765266 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:38.765237    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d6ccq\" (UniqueName: \"kubernetes.io/projected/774e4952-c994-4374-aa62-3d67cc777d5a-kube-api-access-d6ccq\") pod \"maas-api-e2e-ipp-b-24255b-5555cd4dc9-jmck4\" (UID: \"774e4952-c994-4374-aa62-3d67cc777d5a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-24255b-5555cd4dc9-jmck4"
Nov 02 15:46:38.887069 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:38.886469    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-24255b-5555cd4dc9-jmck4"
Nov 02 15:46:39.105310 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:39.105281    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-797488/sub-test-model-797488-kserve-79764959f-k4scd"
Nov 02 15:46:39.179704 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:39.179605    5304 generic.go:356] "Generic (PLEG): container finished" podID="2d6e474d-c5d3-4c69-b12f-993ff23910c1" containerID="ed3dfc44a4c9b27ff03ddc35712aa5e43e60362546463f53adf9ea3e0556a7eb" exitCode=0
Nov 02 15:46:39.179704 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:39.179659    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-797488/sub-test-model-797488-kserve-79764959f-k4scd"
Nov 02 15:46:39.179927 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:39.179706    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-797488/sub-test-model-797488-kserve-79764959f-k4scd" event={"ID":"2d6e474d-c5d3-4c69-b12f-993ff23910c1","Type":"ContainerDied","Data":"ed3dfc44a4c9b27ff03ddc35712aa5e43e60362546463f53adf9ea3e0556a7eb"}
Nov 02 15:46:39.179927 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:39.179763    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-797488/sub-test-model-797488-kserve-79764959f-k4scd" event={"ID":"2d6e474d-c5d3-4c69-b12f-993ff23910c1","Type":"ContainerDied","Data":"ea166f3df9e64bfb8ba6de86155dd0052e15c76c65eed7fd98aac1c3e1e7c542"}
Nov 02 15:46:39.179927 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:39.179799    5304 scope.go:122] "RemoveContainer" containerID="ed3dfc44a4c9b27ff03ddc35712aa5e43e60362546463f53adf9ea3e0556a7eb"
Nov 02 15:46:39.190695 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:39.190676    5304 scope.go:122] "RemoveContainer" containerID="ed3dfc44a4c9b27ff03ddc35712aa5e43e60362546463f53adf9ea3e0556a7eb"
Nov 02 15:46:39.191045 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:46:39.191018    5304 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ed3dfc44a4c9b27ff03ddc35712aa5e43e60362546463f53adf9ea3e0556a7eb\": container with ID starting with ed3dfc44a4c9b27ff03ddc35712aa5e43e60362546463f53adf9ea3e0556a7eb not found: ID does not exist" containerID="ed3dfc44a4c9b27ff03ddc35712aa5e43e60362546463f53adf9ea3e0556a7eb"
Nov 02 15:46:39.191204 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:39.191171    5304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed3dfc44a4c9b27ff03ddc35712aa5e43e60362546463f53adf9ea3e0556a7eb"} err="failed to get container status \"ed3dfc44a4c9b27ff03ddc35712aa5e43e60362546463f53adf9ea3e0556a7eb\": rpc error: code = NotFound desc = could not find container \"ed3dfc44a4c9b27ff03ddc35712aa5e43e60362546463f53adf9ea3e0556a7eb\": container with ID starting with ed3dfc44a4c9b27ff03ddc35712aa5e43e60362546463f53adf9ea3e0556a7eb not found: ID does not exist"
Nov 02 15:46:39.205215 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:39.205189    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-24255b-5555cd4dc9-jmck4"]
Nov 02 15:46:39.212934 ip-10-0-135-138 kubenswrapper[5304]: W1102 15:46:39.212896    5304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod774e4952_c994_4374_aa62_3d67cc777d5a.slice/crio-1a5b8f14764643709ecd03c9cd785d37a5367c7666d116105dea2d3aa5522a05 WatchSource:0}: Error finding container 1a5b8f14764643709ecd03c9cd785d37a5367c7666d116105dea2d3aa5522a05: Status 404 returned error can't find the container with id 1a5b8f14764643709ecd03c9cd785d37a5367c7666d116105dea2d3aa5522a05
Nov 02 15:46:39.252535 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:39.252503    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d6e474d-c5d3-4c69-b12f-993ff23910c1-dshm\" (UniqueName: \"kubernetes.io/empty-dir/2d6e474d-c5d3-4c69-b12f-993ff23910c1-dshm\") pod \"2d6e474d-c5d3-4c69-b12f-993ff23910c1\" (UID: \"2d6e474d-c5d3-4c69-b12f-993ff23910c1\") "
Nov 02 15:46:39.252703 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:39.252686    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d6e474d-c5d3-4c69-b12f-993ff23910c1-home\" (UniqueName: \"kubernetes.io/empty-dir/2d6e474d-c5d3-4c69-b12f-993ff23910c1-home\") pod \"2d6e474d-c5d3-4c69-b12f-993ff23910c1\" (UID: \"2d6e474d-c5d3-4c69-b12f-993ff23910c1\") "
Nov 02 15:46:39.252752 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:39.252725    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d6e474d-c5d3-4c69-b12f-993ff23910c1-kube-api-access-2dj2p\" (UniqueName: \"kubernetes.io/projected/2d6e474d-c5d3-4c69-b12f-993ff23910c1-kube-api-access-2dj2p\") pod \"2d6e474d-c5d3-4c69-b12f-993ff23910c1\" (UID: \"2d6e474d-c5d3-4c69-b12f-993ff23910c1\") "
Nov 02 15:46:39.252799 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:39.252772    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2d6e474d-c5d3-4c69-b12f-993ff23910c1-tls-certs\" (UniqueName: \"kubernetes.io/secret/2d6e474d-c5d3-4c69-b12f-993ff23910c1-tls-certs\") pod \"2d6e474d-c5d3-4c69-b12f-993ff23910c1\" (UID: \"2d6e474d-c5d3-4c69-b12f-993ff23910c1\") "
Nov 02 15:46:39.252898 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:39.252868    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d6e474d-c5d3-4c69-b12f-993ff23910c1-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d6e474d-c5d3-4c69-b12f-993ff23910c1-tmp-dir\") pod \"2d6e474d-c5d3-4c69-b12f-993ff23910c1\" (UID: \"2d6e474d-c5d3-4c69-b12f-993ff23910c1\") "
Nov 02 15:46:39.253024 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:39.252917    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d6e474d-c5d3-4c69-b12f-993ff23910c1-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2d6e474d-c5d3-4c69-b12f-993ff23910c1-model-cache\") pod \"2d6e474d-c5d3-4c69-b12f-993ff23910c1\" (UID: \"2d6e474d-c5d3-4c69-b12f-993ff23910c1\") "
Nov 02 15:46:39.253024 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:39.252962    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d6e474d-c5d3-4c69-b12f-993ff23910c1-home" pod "2d6e474d-c5d3-4c69-b12f-993ff23910c1" (UID: "2d6e474d-c5d3-4c69-b12f-993ff23910c1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:46:39.253205 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:39.253184    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d6e474d-c5d3-4c69-b12f-993ff23910c1-tmp-dir" pod "2d6e474d-c5d3-4c69-b12f-993ff23910c1" (UID: "2d6e474d-c5d3-4c69-b12f-993ff23910c1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:46:39.253333 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:39.253277    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d6e474d-c5d3-4c69-b12f-993ff23910c1-model-cache" pod "2d6e474d-c5d3-4c69-b12f-993ff23910c1" (UID: "2d6e474d-c5d3-4c69-b12f-993ff23910c1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:46:39.253729 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:39.253704    5304 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d6e474d-c5d3-4c69-b12f-993ff23910c1-tmp-dir\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:46:39.253858 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:39.253732    5304 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2d6e474d-c5d3-4c69-b12f-993ff23910c1-model-cache\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:46:39.253858 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:39.253745    5304 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2d6e474d-c5d3-4c69-b12f-993ff23910c1-home\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:46:39.255095 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:39.255070    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d6e474d-c5d3-4c69-b12f-993ff23910c1-kube-api-access-2dj2p" pod "2d6e474d-c5d3-4c69-b12f-993ff23910c1" (UID: "2d6e474d-c5d3-4c69-b12f-993ff23910c1"). InnerVolumeSpecName "kube-api-access-2dj2p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:46:39.255095 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:39.255080    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d6e474d-c5d3-4c69-b12f-993ff23910c1-dshm" pod "2d6e474d-c5d3-4c69-b12f-993ff23910c1" (UID: "2d6e474d-c5d3-4c69-b12f-993ff23910c1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:46:39.255231 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:39.255092    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2d6e474d-c5d3-4c69-b12f-993ff23910c1-tls-certs" pod "2d6e474d-c5d3-4c69-b12f-993ff23910c1" (UID: "2d6e474d-c5d3-4c69-b12f-993ff23910c1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:46:39.354810 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:39.354778    5304 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2d6e474d-c5d3-4c69-b12f-993ff23910c1-dshm\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:46:39.354810 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:39.354808    5304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2dj2p\" (UniqueName: \"kubernetes.io/projected/2d6e474d-c5d3-4c69-b12f-993ff23910c1-kube-api-access-2dj2p\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:46:39.355270 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:39.354819    5304 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d6e474d-c5d3-4c69-b12f-993ff23910c1-tls-certs\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:46:39.526274 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:39.526239    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-797488/sub-test-model-797488-kserve-79764959f-k4scd"]
Nov 02 15:46:39.530185 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:39.530154    5304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w2-797488/sub-test-model-797488-kserve-79764959f-k4scd"]
Nov 02 15:46:40.122306 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:40.122272    5304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d6e474d-c5d3-4c69-b12f-993ff23910c1" path="/var/lib/kubelet/pods/2d6e474d-c5d3-4c69-b12f-993ff23910c1/volumes"
Nov 02 15:46:40.190911 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:40.190876    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-24255b-5555cd4dc9-jmck4" event={"ID":"774e4952-c994-4374-aa62-3d67cc777d5a","Type":"ContainerStarted","Data":"16e425e93f9a7e78b486668ef7f7db3b4a2bc1744569f2bbc3ca5e57798348b2"}
Nov 02 15:46:40.191099 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:40.190932    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-24255b-5555cd4dc9-jmck4" event={"ID":"774e4952-c994-4374-aa62-3d67cc777d5a","Type":"ContainerStarted","Data":"1a5b8f14764643709ecd03c9cd785d37a5367c7666d116105dea2d3aa5522a05"}
Nov 02 15:46:40.191099 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:40.190976    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-24255b-5555cd4dc9-jmck4"
Nov 02 15:46:40.234214 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:40.234159    5304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-24255b-5555cd4dc9-jmck4" podStartSLOduration=1.826346977 podStartE2EDuration="2.234144245s" totalImagesPullingTime="407.797268ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:46:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:46:40.23299457 +0000 UTC m=+3000.226818981" watchObservedRunningTime="2026-11-02 15:46:40.234144245 +0000 UTC m=+3000.227968651"
Nov 02 15:46:43.101994 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:43.101954    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-d79991-56bd8b5955-jkfrr"
Nov 02 15:46:43.102433 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:43.102032    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d79991-7765697955-29mq6"
Nov 02 15:46:46.204000 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:46.203969    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-24255b-5555cd4dc9-jmck4"
Nov 02 15:46:46.878540 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:46.878503    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-501738-b94d75b7d-s94zq"]
Nov 02 15:46:46.878836 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:46.878806    5304 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-501738-b94d75b7d-s94zq" podUID="c55a8850-8fb6-4059-851e-47519d695e04" containerName="maas-api" containerID="cri-o://d0fd7613960afb9ab1525da859024bfa6e5e421d172d641faa20c5ba286c4189" gracePeriod=30
Nov 02 15:46:47.251057 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:47.250928    5304 generic.go:356] "Generic (PLEG): container finished" podID="c55a8850-8fb6-4059-851e-47519d695e04" containerID="d0fd7613960afb9ab1525da859024bfa6e5e421d172d641faa20c5ba286c4189" exitCode=0
Nov 02 15:46:47.251057 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:47.250980    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-501738-b94d75b7d-s94zq" event={"ID":"c55a8850-8fb6-4059-851e-47519d695e04","Type":"ContainerDied","Data":"d0fd7613960afb9ab1525da859024bfa6e5e421d172d641faa20c5ba286c4189"}
Nov 02 15:46:47.297724 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:47.297657    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-501738-b94d75b7d-s94zq"
Nov 02 15:46:47.350150 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:47.345580    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c55a8850-8fb6-4059-851e-47519d695e04-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c55a8850-8fb6-4059-851e-47519d695e04-maas-api-tls\") pod \"c55a8850-8fb6-4059-851e-47519d695e04\" (UID: \"c55a8850-8fb6-4059-851e-47519d695e04\") "
Nov 02 15:46:47.350150 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:47.345679    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c55a8850-8fb6-4059-851e-47519d695e04-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c55a8850-8fb6-4059-851e-47519d695e04-openshift-service-ca\") pod \"c55a8850-8fb6-4059-851e-47519d695e04\" (UID: \"c55a8850-8fb6-4059-851e-47519d695e04\") "
Nov 02 15:46:47.350150 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:47.345710    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c55a8850-8fb6-4059-851e-47519d695e04-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c55a8850-8fb6-4059-851e-47519d695e04-metrics-certs\") pod \"c55a8850-8fb6-4059-851e-47519d695e04\" (UID: \"c55a8850-8fb6-4059-851e-47519d695e04\") "
Nov 02 15:46:47.350150 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:47.345869    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c55a8850-8fb6-4059-851e-47519d695e04-kube-api-access-4gqjc\" (UniqueName: \"kubernetes.io/projected/c55a8850-8fb6-4059-851e-47519d695e04-kube-api-access-4gqjc\") pod \"c55a8850-8fb6-4059-851e-47519d695e04\" (UID: \"c55a8850-8fb6-4059-851e-47519d695e04\") "
Nov 02 15:46:47.350150 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:47.347292    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c55a8850-8fb6-4059-851e-47519d695e04-openshift-service-ca" pod "c55a8850-8fb6-4059-851e-47519d695e04" (UID: "c55a8850-8fb6-4059-851e-47519d695e04"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:46:47.383325 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:47.368673    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c55a8850-8fb6-4059-851e-47519d695e04-maas-api-tls" pod "c55a8850-8fb6-4059-851e-47519d695e04" (UID: "c55a8850-8fb6-4059-851e-47519d695e04"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:46:47.383325 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:47.377781    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c55a8850-8fb6-4059-851e-47519d695e04-kube-api-access-4gqjc" pod "c55a8850-8fb6-4059-851e-47519d695e04" (UID: "c55a8850-8fb6-4059-851e-47519d695e04"). InnerVolumeSpecName "kube-api-access-4gqjc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:46:47.383325 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:47.380344    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c55a8850-8fb6-4059-851e-47519d695e04-metrics-certs" pod "c55a8850-8fb6-4059-851e-47519d695e04" (UID: "c55a8850-8fb6-4059-851e-47519d695e04"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:46:47.453747 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:47.453068    5304 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c55a8850-8fb6-4059-851e-47519d695e04-maas-api-tls\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:46:47.453747 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:47.453101    5304 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c55a8850-8fb6-4059-851e-47519d695e04-openshift-service-ca\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:46:47.453747 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:47.453111    5304 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c55a8850-8fb6-4059-851e-47519d695e04-metrics-certs\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:46:47.453747 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:47.453137    5304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4gqjc\" (UniqueName: \"kubernetes.io/projected/c55a8850-8fb6-4059-851e-47519d695e04-kube-api-access-4gqjc\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:46:48.258943 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:48.258909    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-501738-b94d75b7d-s94zq"
Nov 02 15:46:48.259458 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:48.258940    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-501738-b94d75b7d-s94zq" event={"ID":"c55a8850-8fb6-4059-851e-47519d695e04","Type":"ContainerDied","Data":"335cc3e3c0010d97c80ac89f76d433c9d3ec2cd240838736de3b5d30c89b7743"}
Nov 02 15:46:48.259458 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:48.259004    5304 scope.go:122] "RemoveContainer" containerID="d0fd7613960afb9ab1525da859024bfa6e5e421d172d641faa20c5ba286c4189"
Nov 02 15:46:48.289642 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:48.289603    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-501738-767988665-nwzcr" podUID="6eab7ac5-dbb2-4b5d-adac-86e3c425c011" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 15:46:48.289800 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:48.289702    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-501738-864d5f868c-2ssc8" podUID="5ae97c11-e62c-4022-ace2-03a398ff3854" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 15:46:48.312729 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:48.312697    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-501738-b94d75b7d-s94zq"]
Nov 02 15:46:48.325067 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:48.325041    5304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-501738-b94d75b7d-s94zq"]
Nov 02 15:46:50.081000 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:50.080976    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-501738-767988665-nwzcr"
Nov 02 15:46:50.108971 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:50.108946    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-501738-864d5f868c-2ssc8"
Nov 02 15:46:50.120662 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:50.120635    5304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c55a8850-8fb6-4059-851e-47519d695e04" path="/var/lib/kubelet/pods/c55a8850-8fb6-4059-851e-47519d695e04/volumes"
Nov 02 15:46:50.190583 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:50.190549    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6eab7ac5-dbb2-4b5d-adac-86e3c425c011-kube-api-access-pr5p5\" (UniqueName: \"kubernetes.io/projected/6eab7ac5-dbb2-4b5d-adac-86e3c425c011-kube-api-access-pr5p5\") pod \"6eab7ac5-dbb2-4b5d-adac-86e3c425c011\" (UID: \"6eab7ac5-dbb2-4b5d-adac-86e3c425c011\") "
Nov 02 15:46:50.190760 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:50.190662    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ae97c11-e62c-4022-ace2-03a398ff3854-kube-api-access-f7snt\" (UniqueName: \"kubernetes.io/projected/5ae97c11-e62c-4022-ace2-03a398ff3854-kube-api-access-f7snt\") pod \"5ae97c11-e62c-4022-ace2-03a398ff3854\" (UID: \"5ae97c11-e62c-4022-ace2-03a398ff3854\") "
Nov 02 15:46:50.190760 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:50.190703    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6eab7ac5-dbb2-4b5d-adac-86e3c425c011-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6eab7ac5-dbb2-4b5d-adac-86e3c425c011-plugins-config-volume\") pod \"6eab7ac5-dbb2-4b5d-adac-86e3c425c011\" (UID: \"6eab7ac5-dbb2-4b5d-adac-86e3c425c011\") "
Nov 02 15:46:50.190852 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:50.190821    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5ae97c11-e62c-4022-ace2-03a398ff3854-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ae97c11-e62c-4022-ace2-03a398ff3854-plugins-config-volume\") pod \"5ae97c11-e62c-4022-ace2-03a398ff3854\" (UID: \"5ae97c11-e62c-4022-ace2-03a398ff3854\") "
Nov 02 15:46:50.191413 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:50.191378    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ae97c11-e62c-4022-ace2-03a398ff3854-plugins-config-volume" pod "5ae97c11-e62c-4022-ace2-03a398ff3854" (UID: "5ae97c11-e62c-4022-ace2-03a398ff3854"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:46:50.191843 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:50.191809    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6eab7ac5-dbb2-4b5d-adac-86e3c425c011-plugins-config-volume" pod "6eab7ac5-dbb2-4b5d-adac-86e3c425c011" (UID: "6eab7ac5-dbb2-4b5d-adac-86e3c425c011"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:46:50.193329 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:50.193303    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6eab7ac5-dbb2-4b5d-adac-86e3c425c011-kube-api-access-pr5p5" pod "6eab7ac5-dbb2-4b5d-adac-86e3c425c011" (UID: "6eab7ac5-dbb2-4b5d-adac-86e3c425c011"). InnerVolumeSpecName "kube-api-access-pr5p5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:46:50.193420 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:50.193389    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ae97c11-e62c-4022-ace2-03a398ff3854-kube-api-access-f7snt" pod "5ae97c11-e62c-4022-ace2-03a398ff3854" (UID: "5ae97c11-e62c-4022-ace2-03a398ff3854"). InnerVolumeSpecName "kube-api-access-f7snt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:46:50.292775 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:50.292688    5304 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ae97c11-e62c-4022-ace2-03a398ff3854-plugins-config-volume\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:46:50.292775 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:50.292720    5304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pr5p5\" (UniqueName: \"kubernetes.io/projected/6eab7ac5-dbb2-4b5d-adac-86e3c425c011-kube-api-access-pr5p5\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:46:50.292775 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:50.292730    5304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f7snt\" (UniqueName: \"kubernetes.io/projected/5ae97c11-e62c-4022-ace2-03a398ff3854-kube-api-access-f7snt\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:46:50.292775 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:50.292738    5304 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6eab7ac5-dbb2-4b5d-adac-86e3c425c011-plugins-config-volume\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:46:50.307989 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:50.307953    5304 generic.go:356] "Generic (PLEG): container finished" podID="5ae97c11-e62c-4022-ace2-03a398ff3854" containerID="4c4d983a44d0823251a3d5f5eb2c8b42b6389365b1627bf268c2d393ebcac1f7" exitCode=0
Nov 02 15:46:50.308185 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:50.308004    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-501738-864d5f868c-2ssc8"
Nov 02 15:46:50.308185 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:50.308037    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-501738-864d5f868c-2ssc8" event={"ID":"5ae97c11-e62c-4022-ace2-03a398ff3854","Type":"ContainerDied","Data":"4c4d983a44d0823251a3d5f5eb2c8b42b6389365b1627bf268c2d393ebcac1f7"}
Nov 02 15:46:50.308185 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:50.308095    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-501738-864d5f868c-2ssc8" event={"ID":"5ae97c11-e62c-4022-ace2-03a398ff3854","Type":"ContainerDied","Data":"75421e895d5d49ac29e3e0ff39b3c066b7619e0815f3c1407b97e4192fe20788"}
Nov 02 15:46:50.308185 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:50.308148    5304 scope.go:122] "RemoveContainer" containerID="4c4d983a44d0823251a3d5f5eb2c8b42b6389365b1627bf268c2d393ebcac1f7"
Nov 02 15:46:50.309605 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:50.309581    5304 generic.go:356] "Generic (PLEG): container finished" podID="6eab7ac5-dbb2-4b5d-adac-86e3c425c011" containerID="b4997936ebf8111dd7fcd710eee1fdb7909ab32fd4aeb56f3760f614f25f6616" exitCode=0
Nov 02 15:46:50.309729 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:50.309621    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-501738-767988665-nwzcr"
Nov 02 15:46:50.309729 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:50.309627    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-501738-767988665-nwzcr" event={"ID":"6eab7ac5-dbb2-4b5d-adac-86e3c425c011","Type":"ContainerDied","Data":"b4997936ebf8111dd7fcd710eee1fdb7909ab32fd4aeb56f3760f614f25f6616"}
Nov 02 15:46:50.309729 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:50.309670    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-501738-767988665-nwzcr" event={"ID":"6eab7ac5-dbb2-4b5d-adac-86e3c425c011","Type":"ContainerDied","Data":"91e2cceeb6e844436163f10441ebafad7ef1d48d9392a83338923083eecb24d0"}
Nov 02 15:46:50.321298 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:50.320797    5304 scope.go:122] "RemoveContainer" containerID="4c4d983a44d0823251a3d5f5eb2c8b42b6389365b1627bf268c2d393ebcac1f7"
Nov 02 15:46:50.324527 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:46:50.324493    5304 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c4d983a44d0823251a3d5f5eb2c8b42b6389365b1627bf268c2d393ebcac1f7\": container with ID starting with 4c4d983a44d0823251a3d5f5eb2c8b42b6389365b1627bf268c2d393ebcac1f7 not found: ID does not exist" containerID="4c4d983a44d0823251a3d5f5eb2c8b42b6389365b1627bf268c2d393ebcac1f7"
Nov 02 15:46:50.324635 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:50.324537    5304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c4d983a44d0823251a3d5f5eb2c8b42b6389365b1627bf268c2d393ebcac1f7"} err="failed to get container status \"4c4d983a44d0823251a3d5f5eb2c8b42b6389365b1627bf268c2d393ebcac1f7\": rpc error: code = NotFound desc = could not find container \"4c4d983a44d0823251a3d5f5eb2c8b42b6389365b1627bf268c2d393ebcac1f7\": container with ID starting with 4c4d983a44d0823251a3d5f5eb2c8b42b6389365b1627bf268c2d393ebcac1f7 not found: ID does not exist"
Nov 02 15:46:50.324635 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:50.324564    5304 scope.go:122] "RemoveContainer" containerID="b4997936ebf8111dd7fcd710eee1fdb7909ab32fd4aeb56f3760f614f25f6616"
Nov 02 15:46:50.336588 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:50.336568    5304 scope.go:122] "RemoveContainer" containerID="b4997936ebf8111dd7fcd710eee1fdb7909ab32fd4aeb56f3760f614f25f6616"
Nov 02 15:46:50.336897 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:46:50.336873    5304 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b4997936ebf8111dd7fcd710eee1fdb7909ab32fd4aeb56f3760f614f25f6616\": container with ID starting with b4997936ebf8111dd7fcd710eee1fdb7909ab32fd4aeb56f3760f614f25f6616 not found: ID does not exist" containerID="b4997936ebf8111dd7fcd710eee1fdb7909ab32fd4aeb56f3760f614f25f6616"
Nov 02 15:46:50.336953 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:50.336907    5304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b4997936ebf8111dd7fcd710eee1fdb7909ab32fd4aeb56f3760f614f25f6616"} err="failed to get container status \"b4997936ebf8111dd7fcd710eee1fdb7909ab32fd4aeb56f3760f614f25f6616\": rpc error: code = NotFound desc = could not find container \"b4997936ebf8111dd7fcd710eee1fdb7909ab32fd4aeb56f3760f614f25f6616\": container with ID starting with b4997936ebf8111dd7fcd710eee1fdb7909ab32fd4aeb56f3760f614f25f6616 not found: ID does not exist"
Nov 02 15:46:50.361506 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:50.361477    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-501738-864d5f868c-2ssc8"]
Nov 02 15:46:50.365158 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:50.365118    5304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-501738-864d5f868c-2ssc8"]
Nov 02 15:46:50.397798 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:50.397766    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-501738-767988665-nwzcr"]
Nov 02 15:46:50.402222 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:50.402193    5304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-501738-767988665-nwzcr"]
Nov 02 15:46:52.121572 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:52.121533    5304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5ae97c11-e62c-4022-ace2-03a398ff3854" path="/var/lib/kubelet/pods/5ae97c11-e62c-4022-ace2-03a398ff3854/volumes"
Nov 02 15:46:52.122232 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:52.122210    5304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6eab7ac5-dbb2-4b5d-adac-86e3c425c011" path="/var/lib/kubelet/pods/6eab7ac5-dbb2-4b5d-adac-86e3c425c011/volumes"
Nov 02 15:46:54.228044 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:54.228008    5304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-f0039c-7cdbfcf875-tptc9"]
Nov 02 15:46:54.228665 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:54.228649    5304 cpu_manager.go:414] "containerMap: removing container" podUID="2d6e474d-c5d3-4c69-b12f-993ff23910c1" containerName="main"
Nov 02 15:46:54.228718 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:54.228667    5304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d6e474d-c5d3-4c69-b12f-993ff23910c1" containerName="main"
Nov 02 15:46:54.228718 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:54.228688    5304 cpu_manager.go:414] "containerMap: removing container" podUID="6eab7ac5-dbb2-4b5d-adac-86e3c425c011" containerName="payload-processing"
Nov 02 15:46:54.228718 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:54.228694    5304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6eab7ac5-dbb2-4b5d-adac-86e3c425c011" containerName="payload-processing"
Nov 02 15:46:54.228718 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:54.228711    5304 cpu_manager.go:414] "containerMap: removing container" podUID="5ae97c11-e62c-4022-ace2-03a398ff3854" containerName="payload-pre-processing"
Nov 02 15:46:54.228718 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:54.228717    5304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ae97c11-e62c-4022-ace2-03a398ff3854" containerName="payload-pre-processing"
Nov 02 15:46:54.228901 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:54.228732    5304 cpu_manager.go:414] "containerMap: removing container" podUID="c55a8850-8fb6-4059-851e-47519d695e04" containerName="maas-api"
Nov 02 15:46:54.228901 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:54.228737    5304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c55a8850-8fb6-4059-851e-47519d695e04" containerName="maas-api"
Nov 02 15:46:54.228901 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:54.228817    5304 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ae97c11-e62c-4022-ace2-03a398ff3854" containerName="payload-pre-processing"
Nov 02 15:46:54.228901 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:54.228828    5304 memory_manager.go:365] "RemoveStaleState removing state" podUID="c55a8850-8fb6-4059-851e-47519d695e04" containerName="maas-api"
Nov 02 15:46:54.228901 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:54.228838    5304 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d6e474d-c5d3-4c69-b12f-993ff23910c1" containerName="main"
Nov 02 15:46:54.228901 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:54.228846    5304 memory_manager.go:365] "RemoveStaleState removing state" podUID="6eab7ac5-dbb2-4b5d-adac-86e3c425c011" containerName="payload-processing"
Nov 02 15:46:54.233535 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:54.233518    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f0039c-7cdbfcf875-tptc9"
Nov 02 15:46:54.254930 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:54.254892    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-f0039c-7cdbfcf875-tptc9"]
Nov 02 15:46:54.297092 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:54.297055    5304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-f0039c-serving-cert\""
Nov 02 15:46:54.373847 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:54.373812    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/39e13479-f61d-4215-bbb3-03aa8b678618-metrics-certs\") pod \"maas-api-e2e-ipp-a-f0039c-7cdbfcf875-tptc9\" (UID: \"39e13479-f61d-4215-bbb3-03aa8b678618\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f0039c-7cdbfcf875-tptc9"
Nov 02 15:46:54.374031 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:54.373881    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/39e13479-f61d-4215-bbb3-03aa8b678618-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-f0039c-7cdbfcf875-tptc9\" (UID: \"39e13479-f61d-4215-bbb3-03aa8b678618\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f0039c-7cdbfcf875-tptc9"
Nov 02 15:46:54.374031 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:54.374002    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jnq7h\" (UniqueName: \"kubernetes.io/projected/39e13479-f61d-4215-bbb3-03aa8b678618-kube-api-access-jnq7h\") pod \"maas-api-e2e-ipp-a-f0039c-7cdbfcf875-tptc9\" (UID: \"39e13479-f61d-4215-bbb3-03aa8b678618\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f0039c-7cdbfcf875-tptc9"
Nov 02 15:46:54.374112 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:54.374054    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/39e13479-f61d-4215-bbb3-03aa8b678618-maas-api-tls\") pod \"maas-api-e2e-ipp-a-f0039c-7cdbfcf875-tptc9\" (UID: \"39e13479-f61d-4215-bbb3-03aa8b678618\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f0039c-7cdbfcf875-tptc9"
Nov 02 15:46:54.503233 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:54.499348    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jnq7h\" (UniqueName: \"kubernetes.io/projected/39e13479-f61d-4215-bbb3-03aa8b678618-kube-api-access-jnq7h\") pod \"maas-api-e2e-ipp-a-f0039c-7cdbfcf875-tptc9\" (UID: \"39e13479-f61d-4215-bbb3-03aa8b678618\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f0039c-7cdbfcf875-tptc9"
Nov 02 15:46:54.503233 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:54.499433    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/39e13479-f61d-4215-bbb3-03aa8b678618-maas-api-tls\") pod \"maas-api-e2e-ipp-a-f0039c-7cdbfcf875-tptc9\" (UID: \"39e13479-f61d-4215-bbb3-03aa8b678618\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f0039c-7cdbfcf875-tptc9"
Nov 02 15:46:54.503233 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:54.499641    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/39e13479-f61d-4215-bbb3-03aa8b678618-metrics-certs\") pod \"maas-api-e2e-ipp-a-f0039c-7cdbfcf875-tptc9\" (UID: \"39e13479-f61d-4215-bbb3-03aa8b678618\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f0039c-7cdbfcf875-tptc9"
Nov 02 15:46:54.503233 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:54.499719    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/39e13479-f61d-4215-bbb3-03aa8b678618-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-f0039c-7cdbfcf875-tptc9\" (UID: \"39e13479-f61d-4215-bbb3-03aa8b678618\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f0039c-7cdbfcf875-tptc9"
Nov 02 15:46:54.503233 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:54.500809    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/39e13479-f61d-4215-bbb3-03aa8b678618-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-f0039c-7cdbfcf875-tptc9\" (UID: \"39e13479-f61d-4215-bbb3-03aa8b678618\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f0039c-7cdbfcf875-tptc9"
Nov 02 15:46:54.511548 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:54.511510    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/39e13479-f61d-4215-bbb3-03aa8b678618-metrics-certs\") pod \"maas-api-e2e-ipp-a-f0039c-7cdbfcf875-tptc9\" (UID: \"39e13479-f61d-4215-bbb3-03aa8b678618\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f0039c-7cdbfcf875-tptc9"
Nov 02 15:46:54.511800 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:54.511769    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/39e13479-f61d-4215-bbb3-03aa8b678618-maas-api-tls\") pod \"maas-api-e2e-ipp-a-f0039c-7cdbfcf875-tptc9\" (UID: \"39e13479-f61d-4215-bbb3-03aa8b678618\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f0039c-7cdbfcf875-tptc9"
Nov 02 15:46:54.538148 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:54.530343    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jnq7h\" (UniqueName: \"kubernetes.io/projected/39e13479-f61d-4215-bbb3-03aa8b678618-kube-api-access-jnq7h\") pod \"maas-api-e2e-ipp-a-f0039c-7cdbfcf875-tptc9\" (UID: \"39e13479-f61d-4215-bbb3-03aa8b678618\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f0039c-7cdbfcf875-tptc9"
Nov 02 15:46:54.555981 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:54.554690    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f0039c-7cdbfcf875-tptc9"
Nov 02 15:46:54.908974 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:46:54.908930    5304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k6fjk" containerName="extractor"
Nov 02 15:46:54.948724 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:54.948698    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-f0039c-7cdbfcf875-tptc9"]
Nov 02 15:46:54.951367 ip-10-0-135-138 kubenswrapper[5304]: W1102 15:46:54.951343    5304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39e13479_f61d_4215_bbb3_03aa8b678618.slice/crio-847019b1ed96dae9cec53de4c21a908d10f9cd5d33cddf8c0095e40a13cb0727 WatchSource:0}: Error finding container 847019b1ed96dae9cec53de4c21a908d10f9cd5d33cddf8c0095e40a13cb0727: Status 404 returned error can't find the container with id 847019b1ed96dae9cec53de4c21a908d10f9cd5d33cddf8c0095e40a13cb0727
Nov 02 15:46:54.953493 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:54.953472    5304 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 02 15:46:55.350155 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:55.350085    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f0039c-7cdbfcf875-tptc9" event={"ID":"39e13479-f61d-4215-bbb3-03aa8b678618","Type":"ContainerStarted","Data":"847019b1ed96dae9cec53de4c21a908d10f9cd5d33cddf8c0095e40a13cb0727"}
Nov 02 15:46:55.560207 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:55.560047    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-501738-openshift-default-54c4d5d859-bwbmq"]
Nov 02 15:46:55.566468 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:55.566428    5304 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-501738-openshift-default-54c4d5d859-bwbmq" podUID="f7d14663-db08-4ae9-9dae-98f56d51f8a4" containerName="istio-proxy" containerID="cri-o://0a34c6823625ecbc1ed41f24dee0a5cd0cd06159cecd9bba65d71e90fbcc34a8" gracePeriod=30
Nov 02 15:46:56.360633 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:56.360589    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f0039c-7cdbfcf875-tptc9" event={"ID":"39e13479-f61d-4215-bbb3-03aa8b678618","Type":"ContainerStarted","Data":"15725e87a6a3cbe9577f5debe711bf87ca634270346ca83b12faee6047b797f5"}
Nov 02 15:46:56.361108 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:56.360659    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f0039c-7cdbfcf875-tptc9"
Nov 02 15:46:58.024442 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:58.024371    5304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f0039c-7cdbfcf875-tptc9" podStartSLOduration=3.678293123 podStartE2EDuration="4.024350668s" totalImagesPullingTime="346.057545ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:46:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:46:56.405356192 +0000 UTC m=+3016.399180603" watchObservedRunningTime="2026-11-02 15:46:58.024350668 +0000 UTC m=+3018.018175077"
Nov 02 15:46:58.025905 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:58.025877    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-e80c77-dc5cd4d99-plkbt"]
Nov 02 15:46:58.026147 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:58.026113    5304 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e80c77-dc5cd4d99-plkbt" podUID="36430ec8-da3a-4166-b04c-3671a1f3de3f" containerName="payload-pre-processing" containerID="cri-o://632eef1c1f0ceb32dfd7180dc268d4f3f47714f519a1efde73200187082d863c" gracePeriod=30
Nov 02 15:46:58.453149 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:58.449338    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e80c77-dc5cd4d99-plkbt"
Nov 02 15:46:58.525216 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:58.513964    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/36430ec8-da3a-4166-b04c-3671a1f3de3f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/36430ec8-da3a-4166-b04c-3671a1f3de3f-plugins-config-volume\") pod \"36430ec8-da3a-4166-b04c-3671a1f3de3f\" (UID: \"36430ec8-da3a-4166-b04c-3671a1f3de3f\") "
Nov 02 15:46:58.525585 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:58.525560    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/36430ec8-da3a-4166-b04c-3671a1f3de3f-kube-api-access-s7z9g\" (UniqueName: \"kubernetes.io/projected/36430ec8-da3a-4166-b04c-3671a1f3de3f-kube-api-access-s7z9g\") pod \"36430ec8-da3a-4166-b04c-3671a1f3de3f\" (UID: \"36430ec8-da3a-4166-b04c-3671a1f3de3f\") "
Nov 02 15:46:58.527909 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:58.514908    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/36430ec8-da3a-4166-b04c-3671a1f3de3f-plugins-config-volume" pod "36430ec8-da3a-4166-b04c-3671a1f3de3f" (UID: "36430ec8-da3a-4166-b04c-3671a1f3de3f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:46:58.543339 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:58.543065    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36430ec8-da3a-4166-b04c-3671a1f3de3f-kube-api-access-s7z9g" pod "36430ec8-da3a-4166-b04c-3671a1f3de3f" (UID: "36430ec8-da3a-4166-b04c-3671a1f3de3f"). InnerVolumeSpecName "kube-api-access-s7z9g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:46:58.546044 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:58.545995    5304 generic.go:356] "Generic (PLEG): container finished" podID="36430ec8-da3a-4166-b04c-3671a1f3de3f" containerID="632eef1c1f0ceb32dfd7180dc268d4f3f47714f519a1efde73200187082d863c" exitCode=0
Nov 02 15:46:58.546911 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:58.546256    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e80c77-dc5cd4d99-plkbt" event={"ID":"36430ec8-da3a-4166-b04c-3671a1f3de3f","Type":"ContainerDied","Data":"632eef1c1f0ceb32dfd7180dc268d4f3f47714f519a1efde73200187082d863c"}
Nov 02 15:46:58.547175 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:58.547156    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e80c77-dc5cd4d99-plkbt" event={"ID":"36430ec8-da3a-4166-b04c-3671a1f3de3f","Type":"ContainerDied","Data":"a55adb2dd1f93b197953cee12e348f10596fb74b41a2d70a58597a70cf3f3567"}
Nov 02 15:46:58.547330 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:58.547317    5304 scope.go:122] "RemoveContainer" containerID="632eef1c1f0ceb32dfd7180dc268d4f3f47714f519a1efde73200187082d863c"
Nov 02 15:46:58.547563 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:58.546378    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e80c77-dc5cd4d99-plkbt"
Nov 02 15:46:58.576342 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:58.574068    5304 scope.go:122] "RemoveContainer" containerID="632eef1c1f0ceb32dfd7180dc268d4f3f47714f519a1efde73200187082d863c"
Nov 02 15:46:58.577507 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:46:58.577391    5304 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"632eef1c1f0ceb32dfd7180dc268d4f3f47714f519a1efde73200187082d863c\": container with ID starting with 632eef1c1f0ceb32dfd7180dc268d4f3f47714f519a1efde73200187082d863c not found: ID does not exist" containerID="632eef1c1f0ceb32dfd7180dc268d4f3f47714f519a1efde73200187082d863c"
Nov 02 15:46:58.577507 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:58.577431    5304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"632eef1c1f0ceb32dfd7180dc268d4f3f47714f519a1efde73200187082d863c"} err="failed to get container status \"632eef1c1f0ceb32dfd7180dc268d4f3f47714f519a1efde73200187082d863c\": rpc error: code = NotFound desc = could not find container \"632eef1c1f0ceb32dfd7180dc268d4f3f47714f519a1efde73200187082d863c\": container with ID starting with 632eef1c1f0ceb32dfd7180dc268d4f3f47714f519a1efde73200187082d863c not found: ID does not exist"
Nov 02 15:46:58.621407 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:58.621368    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-e80c77-dc5cd4d99-plkbt"]
Nov 02 15:46:58.627589 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:58.627557    5304 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/36430ec8-da3a-4166-b04c-3671a1f3de3f-plugins-config-volume\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:46:58.627589 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:58.627591    5304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s7z9g\" (UniqueName: \"kubernetes.io/projected/36430ec8-da3a-4166-b04c-3671a1f3de3f-kube-api-access-s7z9g\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:46:58.639872 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:58.639831    5304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-e80c77-dc5cd4d99-plkbt"]
Nov 02 15:46:58.760447 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:58.760409    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e80c77-2tvs6"]
Nov 02 15:46:58.763611 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:58.763581    5304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e80c77-2tvs6"]
Nov 02 15:46:59.964870 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:59.964832    5304 patch_prober.go:29] interesting pod/e2e-shared-a-w4-501738-openshift-default-54c4d5d859-bwbmq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 15:46:59.965286 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:46:59.964898    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w4-501738-openshift-default-54c4d5d859-bwbmq" podUID="f7d14663-db08-4ae9-9dae-98f56d51f8a4" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 15:47:00.119933 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:00.119900    5304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="36430ec8-da3a-4166-b04c-3671a1f3de3f" path="/var/lib/kubelet/pods/36430ec8-da3a-4166-b04c-3671a1f3de3f/volumes"
Nov 02 15:47:00.120430 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:00.120415    5304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8bac17da-adf2-45d3-88d2-57c9e56b3db6" path="/var/lib/kubelet/pods/8bac17da-adf2-45d3-88d2-57c9e56b3db6/volumes"
Nov 02 15:47:00.876878 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:00.876846    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-501738-openshift-default-54c4d5d859-bwbmq"
Nov 02 15:47:00.960580 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:00.960544    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f7d14663-db08-4ae9-9dae-98f56d51f8a4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7d14663-db08-4ae9-9dae-98f56d51f8a4-istio-podinfo\") pod \"f7d14663-db08-4ae9-9dae-98f56d51f8a4\" (UID: \"f7d14663-db08-4ae9-9dae-98f56d51f8a4\") "
Nov 02 15:47:00.960886 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:00.960600    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f7d14663-db08-4ae9-9dae-98f56d51f8a4-istio-token\" (UniqueName: \"kubernetes.io/projected/f7d14663-db08-4ae9-9dae-98f56d51f8a4-istio-token\") pod \"f7d14663-db08-4ae9-9dae-98f56d51f8a4\" (UID: \"f7d14663-db08-4ae9-9dae-98f56d51f8a4\") "
Nov 02 15:47:00.960886 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:00.960746    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7d14663-db08-4ae9-9dae-98f56d51f8a4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7d14663-db08-4ae9-9dae-98f56d51f8a4-istio-envoy\") pod \"f7d14663-db08-4ae9-9dae-98f56d51f8a4\" (UID: \"f7d14663-db08-4ae9-9dae-98f56d51f8a4\") "
Nov 02 15:47:00.960886 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:00.960801    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7d14663-db08-4ae9-9dae-98f56d51f8a4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7d14663-db08-4ae9-9dae-98f56d51f8a4-workload-socket\") pod \"f7d14663-db08-4ae9-9dae-98f56d51f8a4\" (UID: \"f7d14663-db08-4ae9-9dae-98f56d51f8a4\") "
Nov 02 15:47:00.960886 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:00.960829    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f7d14663-db08-4ae9-9dae-98f56d51f8a4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7d14663-db08-4ae9-9dae-98f56d51f8a4-istiod-ca-cert\") pod \"f7d14663-db08-4ae9-9dae-98f56d51f8a4\" (UID: \"f7d14663-db08-4ae9-9dae-98f56d51f8a4\") "
Nov 02 15:47:00.961186 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:00.960934    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7d14663-db08-4ae9-9dae-98f56d51f8a4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7d14663-db08-4ae9-9dae-98f56d51f8a4-workload-certs\") pod \"f7d14663-db08-4ae9-9dae-98f56d51f8a4\" (UID: \"f7d14663-db08-4ae9-9dae-98f56d51f8a4\") "
Nov 02 15:47:00.961186 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:00.960964    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7d14663-db08-4ae9-9dae-98f56d51f8a4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7d14663-db08-4ae9-9dae-98f56d51f8a4-istio-data\") pod \"f7d14663-db08-4ae9-9dae-98f56d51f8a4\" (UID: \"f7d14663-db08-4ae9-9dae-98f56d51f8a4\") "
Nov 02 15:47:00.961186 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:00.961026    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f7d14663-db08-4ae9-9dae-98f56d51f8a4-kube-api-access-nggqv\" (UniqueName: \"kubernetes.io/projected/f7d14663-db08-4ae9-9dae-98f56d51f8a4-kube-api-access-nggqv\") pod \"f7d14663-db08-4ae9-9dae-98f56d51f8a4\" (UID: \"f7d14663-db08-4ae9-9dae-98f56d51f8a4\") "
Nov 02 15:47:00.961186 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:00.961070    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7d14663-db08-4ae9-9dae-98f56d51f8a4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7d14663-db08-4ae9-9dae-98f56d51f8a4-credential-socket\") pod \"f7d14663-db08-4ae9-9dae-98f56d51f8a4\" (UID: \"f7d14663-db08-4ae9-9dae-98f56d51f8a4\") "
Nov 02 15:47:00.961186 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:00.961097    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7d14663-db08-4ae9-9dae-98f56d51f8a4-workload-socket" pod "f7d14663-db08-4ae9-9dae-98f56d51f8a4" (UID: "f7d14663-db08-4ae9-9dae-98f56d51f8a4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:47:00.961575 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:00.961546    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7d14663-db08-4ae9-9dae-98f56d51f8a4-istio-data" pod "f7d14663-db08-4ae9-9dae-98f56d51f8a4" (UID: "f7d14663-db08-4ae9-9dae-98f56d51f8a4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:47:00.961812 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:00.961787    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7d14663-db08-4ae9-9dae-98f56d51f8a4-credential-socket" pod "f7d14663-db08-4ae9-9dae-98f56d51f8a4" (UID: "f7d14663-db08-4ae9-9dae-98f56d51f8a4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:47:00.961904 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:00.961879    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7d14663-db08-4ae9-9dae-98f56d51f8a4-istiod-ca-cert" pod "f7d14663-db08-4ae9-9dae-98f56d51f8a4" (UID: "f7d14663-db08-4ae9-9dae-98f56d51f8a4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:47:00.961972 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:00.961942    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7d14663-db08-4ae9-9dae-98f56d51f8a4-workload-certs" pod "f7d14663-db08-4ae9-9dae-98f56d51f8a4" (UID: "f7d14663-db08-4ae9-9dae-98f56d51f8a4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:47:00.962179 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:00.962120    5304 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7d14663-db08-4ae9-9dae-98f56d51f8a4-workload-socket\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:47:00.962311 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:00.962182    5304 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7d14663-db08-4ae9-9dae-98f56d51f8a4-istiod-ca-cert\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:47:00.962311 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:00.962197    5304 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7d14663-db08-4ae9-9dae-98f56d51f8a4-workload-certs\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:47:00.962311 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:00.962209    5304 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7d14663-db08-4ae9-9dae-98f56d51f8a4-istio-data\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:47:00.962311 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:00.962222    5304 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7d14663-db08-4ae9-9dae-98f56d51f8a4-credential-socket\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:47:00.964453 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:00.964431    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f7d14663-db08-4ae9-9dae-98f56d51f8a4-istio-podinfo" pod "f7d14663-db08-4ae9-9dae-98f56d51f8a4" (UID: "f7d14663-db08-4ae9-9dae-98f56d51f8a4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 15:47:00.964453 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:00.964442    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7d14663-db08-4ae9-9dae-98f56d51f8a4-istio-envoy" pod "f7d14663-db08-4ae9-9dae-98f56d51f8a4" (UID: "f7d14663-db08-4ae9-9dae-98f56d51f8a4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:47:00.964587 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:00.964499    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7d14663-db08-4ae9-9dae-98f56d51f8a4-istio-token" pod "f7d14663-db08-4ae9-9dae-98f56d51f8a4" (UID: "f7d14663-db08-4ae9-9dae-98f56d51f8a4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:47:00.965600 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:00.965580    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7d14663-db08-4ae9-9dae-98f56d51f8a4-kube-api-access-nggqv" pod "f7d14663-db08-4ae9-9dae-98f56d51f8a4" (UID: "f7d14663-db08-4ae9-9dae-98f56d51f8a4"). InnerVolumeSpecName "kube-api-access-nggqv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:47:01.063642 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:01.063607    5304 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7d14663-db08-4ae9-9dae-98f56d51f8a4-istio-podinfo\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:47:01.063642 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:01.063637    5304 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7d14663-db08-4ae9-9dae-98f56d51f8a4-istio-token\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:47:01.063642 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:01.063646    5304 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7d14663-db08-4ae9-9dae-98f56d51f8a4-istio-envoy\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:47:01.063899 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:01.063656    5304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nggqv\" (UniqueName: \"kubernetes.io/projected/f7d14663-db08-4ae9-9dae-98f56d51f8a4-kube-api-access-nggqv\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:47:01.580931 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:01.580894    5304 generic.go:356] "Generic (PLEG): container finished" podID="f7d14663-db08-4ae9-9dae-98f56d51f8a4" containerID="0a34c6823625ecbc1ed41f24dee0a5cd0cd06159cecd9bba65d71e90fbcc34a8" exitCode=0
Nov 02 15:47:01.581192 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:01.580949    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-501738-openshift-default-54c4d5d859-bwbmq"
Nov 02 15:47:01.581192 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:01.580971    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-501738-openshift-default-54c4d5d859-bwbmq" event={"ID":"f7d14663-db08-4ae9-9dae-98f56d51f8a4","Type":"ContainerDied","Data":"0a34c6823625ecbc1ed41f24dee0a5cd0cd06159cecd9bba65d71e90fbcc34a8"}
Nov 02 15:47:01.581192 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:01.581022    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-501738-openshift-default-54c4d5d859-bwbmq" event={"ID":"f7d14663-db08-4ae9-9dae-98f56d51f8a4","Type":"ContainerDied","Data":"9a4b4006637ecde4d11d07bb13d66af2bca14171b007f21f8388a6a44efd38c2"}
Nov 02 15:47:01.581192 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:01.581037    5304 scope.go:122] "RemoveContainer" containerID="0a34c6823625ecbc1ed41f24dee0a5cd0cd06159cecd9bba65d71e90fbcc34a8"
Nov 02 15:47:01.594840 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:01.594821    5304 scope.go:122] "RemoveContainer" containerID="26297c35ae35a96a888e684c429bbe4883e98d41daecfe97074a203d088eb6db"
Nov 02 15:47:01.607482 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:01.607465    5304 scope.go:122] "RemoveContainer" containerID="0a34c6823625ecbc1ed41f24dee0a5cd0cd06159cecd9bba65d71e90fbcc34a8"
Nov 02 15:47:01.607779 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:47:01.607741    5304 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a34c6823625ecbc1ed41f24dee0a5cd0cd06159cecd9bba65d71e90fbcc34a8\": container with ID starting with 0a34c6823625ecbc1ed41f24dee0a5cd0cd06159cecd9bba65d71e90fbcc34a8 not found: ID does not exist" containerID="0a34c6823625ecbc1ed41f24dee0a5cd0cd06159cecd9bba65d71e90fbcc34a8"
Nov 02 15:47:01.607916 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:01.607792    5304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a34c6823625ecbc1ed41f24dee0a5cd0cd06159cecd9bba65d71e90fbcc34a8"} err="failed to get container status \"0a34c6823625ecbc1ed41f24dee0a5cd0cd06159cecd9bba65d71e90fbcc34a8\": rpc error: code = NotFound desc = could not find container \"0a34c6823625ecbc1ed41f24dee0a5cd0cd06159cecd9bba65d71e90fbcc34a8\": container with ID starting with 0a34c6823625ecbc1ed41f24dee0a5cd0cd06159cecd9bba65d71e90fbcc34a8 not found: ID does not exist"
Nov 02 15:47:01.607916 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:01.607818    5304 scope.go:122] "RemoveContainer" containerID="26297c35ae35a96a888e684c429bbe4883e98d41daecfe97074a203d088eb6db"
Nov 02 15:47:01.609010 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:47:01.608979    5304 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"26297c35ae35a96a888e684c429bbe4883e98d41daecfe97074a203d088eb6db\": container with ID starting with 26297c35ae35a96a888e684c429bbe4883e98d41daecfe97074a203d088eb6db not found: ID does not exist" containerID="26297c35ae35a96a888e684c429bbe4883e98d41daecfe97074a203d088eb6db"
Nov 02 15:47:01.609158 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:01.609020    5304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"26297c35ae35a96a888e684c429bbe4883e98d41daecfe97074a203d088eb6db"} err="failed to get container status \"26297c35ae35a96a888e684c429bbe4883e98d41daecfe97074a203d088eb6db\": rpc error: code = NotFound desc = could not find container \"26297c35ae35a96a888e684c429bbe4883e98d41daecfe97074a203d088eb6db\": container with ID starting with 26297c35ae35a96a888e684c429bbe4883e98d41daecfe97074a203d088eb6db not found: ID does not exist"
Nov 02 15:47:01.646758 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:01.646726    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-501738-openshift-default-54c4d5d859-bwbmq"]
Nov 02 15:47:01.658666 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:01.658636    5304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-501738-openshift-default-54c4d5d859-bwbmq"]
Nov 02 15:47:02.119140 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:02.119096    5304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f7d14663-db08-4ae9-9dae-98f56d51f8a4" path="/var/lib/kubelet/pods/f7d14663-db08-4ae9-9dae-98f56d51f8a4/volumes"
Nov 02 15:47:02.375452 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:02.375371    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f0039c-7cdbfcf875-tptc9"
Nov 02 15:47:06.378833 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:06.378798    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-smtdr"]
Nov 02 15:47:06.379332 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:06.379000    5304 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-smtdr" podUID="5086f7a8-e10d-4aca-bc70-950a2f0cd585" containerName="istio-proxy" containerID="cri-o://d169223ffff4b48d9ac2bb350d36d9b24a10a9228d6f22abad89d8b26d735c56" gracePeriod=30
Nov 02 15:47:06.411185 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:06.411149    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-2wwx7"]
Nov 02 15:47:06.411438 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:06.411413    5304 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-2wwx7" podUID="e11f9d23-10db-4bad-a457-7c5db3056979" containerName="istio-proxy" containerID="cri-o://b49da12e9c1eaf625d860f22d16960d3837c0f77d80f408d1da9d5f6c30a3835" gracePeriod=30
Nov 02 15:47:06.414070 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:06.414044    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-lvrfc"]
Nov 02 15:47:06.414296 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:06.414267    5304 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-lvrfc" podUID="93e559b7-2a4c-4533-90d5-5f3fd3956bf0" containerName="istio-proxy" containerID="cri-o://6796c86fab92cf70995a4feb5e28881cf73c8de1f4623d22c1dd07a5c4fa9bb4" gracePeriod=30
Nov 02 15:47:06.933195 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:06.933153    5304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-b955c5-xrc9j"]
Nov 02 15:47:06.933820 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:06.933800    5304 cpu_manager.go:414] "containerMap: removing container" podUID="36430ec8-da3a-4166-b04c-3671a1f3de3f" containerName="payload-pre-processing"
Nov 02 15:47:06.933941 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:06.933822    5304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="36430ec8-da3a-4166-b04c-3671a1f3de3f" containerName="payload-pre-processing"
Nov 02 15:47:06.933941 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:06.933865    5304 cpu_manager.go:414] "containerMap: removing container" podUID="f7d14663-db08-4ae9-9dae-98f56d51f8a4" containerName="istio-proxy"
Nov 02 15:47:06.933941 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:06.933873    5304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7d14663-db08-4ae9-9dae-98f56d51f8a4" containerName="istio-proxy"
Nov 02 15:47:06.933941 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:06.933897    5304 cpu_manager.go:414] "containerMap: removing container" podUID="f7d14663-db08-4ae9-9dae-98f56d51f8a4" containerName="istio-proxy"
Nov 02 15:47:06.933941 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:06.933905    5304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7d14663-db08-4ae9-9dae-98f56d51f8a4" containerName="istio-proxy"
Nov 02 15:47:06.934228 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:06.934050    5304 memory_manager.go:365] "RemoveStaleState removing state" podUID="36430ec8-da3a-4166-b04c-3671a1f3de3f" containerName="payload-pre-processing"
Nov 02 15:47:06.934228 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:06.934070    5304 memory_manager.go:365] "RemoveStaleState removing state" podUID="f7d14663-db08-4ae9-9dae-98f56d51f8a4" containerName="istio-proxy"
Nov 02 15:47:06.934228 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:06.934082    5304 memory_manager.go:365] "RemoveStaleState removing state" podUID="f7d14663-db08-4ae9-9dae-98f56d51f8a4" containerName="istio-proxy"
Nov 02 15:47:06.938952 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:06.938930    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-b955c5-xrc9j"
Nov 02 15:47:06.956622 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:06.956590    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-b955c5-xrc9j"]
Nov 02 15:47:06.957274 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:06.957253    5304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 15:47:06.957557 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:06.957543    5304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-m5926\""
Nov 02 15:47:07.066583 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:07.066546    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4116a3df-a6d5-44f2-a54e-e6bb71c3d3ef-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-b955c5-xrc9j\" (UID: \"4116a3df-a6d5-44f2-a54e-e6bb71c3d3ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-b955c5-xrc9j"
Nov 02 15:47:07.066773 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:07.066740    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4116a3df-a6d5-44f2-a54e-e6bb71c3d3ef-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-b955c5-xrc9j\" (UID: \"4116a3df-a6d5-44f2-a54e-e6bb71c3d3ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-b955c5-xrc9j"
Nov 02 15:47:07.066893 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:07.066874    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l25qq\" (UniqueName: \"kubernetes.io/projected/4116a3df-a6d5-44f2-a54e-e6bb71c3d3ef-kube-api-access-l25qq\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-b955c5-xrc9j\" (UID: \"4116a3df-a6d5-44f2-a54e-e6bb71c3d3ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-b955c5-xrc9j"
Nov 02 15:47:07.168722 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:07.168679    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4116a3df-a6d5-44f2-a54e-e6bb71c3d3ef-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-b955c5-xrc9j\" (UID: \"4116a3df-a6d5-44f2-a54e-e6bb71c3d3ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-b955c5-xrc9j"
Nov 02 15:47:07.168950 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:07.168860    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4116a3df-a6d5-44f2-a54e-e6bb71c3d3ef-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-b955c5-xrc9j\" (UID: \"4116a3df-a6d5-44f2-a54e-e6bb71c3d3ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-b955c5-xrc9j"
Nov 02 15:47:07.169032 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:07.168973    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l25qq\" (UniqueName: \"kubernetes.io/projected/4116a3df-a6d5-44f2-a54e-e6bb71c3d3ef-kube-api-access-l25qq\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-b955c5-xrc9j\" (UID: \"4116a3df-a6d5-44f2-a54e-e6bb71c3d3ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-b955c5-xrc9j"
Nov 02 15:47:07.169625 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:07.169595    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4116a3df-a6d5-44f2-a54e-e6bb71c3d3ef-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-b955c5-xrc9j\" (UID: \"4116a3df-a6d5-44f2-a54e-e6bb71c3d3ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-b955c5-xrc9j"
Nov 02 15:47:07.171276 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:07.171255    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4116a3df-a6d5-44f2-a54e-e6bb71c3d3ef-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-b955c5-xrc9j\" (UID: \"4116a3df-a6d5-44f2-a54e-e6bb71c3d3ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-b955c5-xrc9j"
Nov 02 15:47:07.192499 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:07.192431    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l25qq\" (UniqueName: \"kubernetes.io/projected/4116a3df-a6d5-44f2-a54e-e6bb71c3d3ef-kube-api-access-l25qq\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-b955c5-xrc9j\" (UID: \"4116a3df-a6d5-44f2-a54e-e6bb71c3d3ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-b955c5-xrc9j"
Nov 02 15:47:07.256381 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:07.256349    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-b955c5-xrc9j"
Nov 02 15:47:07.624456 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:07.624386    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-b955c5-xrc9j"]
Nov 02 15:47:08.639210 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:08.639176    5304 generic.go:356] "Generic (PLEG): container finished" podID="4116a3df-a6d5-44f2-a54e-e6bb71c3d3ef" containerID="1f5839bf02cbda85b66db58ada186123f878967d3b3ed2c8a98aa959be4211aa" exitCode=0
Nov 02 15:47:08.639593 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:08.639258    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-b955c5-xrc9j" event={"ID":"4116a3df-a6d5-44f2-a54e-e6bb71c3d3ef","Type":"ContainerDied","Data":"1f5839bf02cbda85b66db58ada186123f878967d3b3ed2c8a98aa959be4211aa"}
Nov 02 15:47:08.639593 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:08.639312    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-b955c5-xrc9j" event={"ID":"4116a3df-a6d5-44f2-a54e-e6bb71c3d3ef","Type":"ContainerStarted","Data":"6306f034cd4e99c0fdfa8d25e23b9810bf20eb21fcc3c78442a171acd5168383"}
Nov 02 15:47:09.813335 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:09.813305    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-b955c5-xrc9j"
Nov 02 15:47:09.914278 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:09.914198    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4116a3df-a6d5-44f2-a54e-e6bb71c3d3ef-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4116a3df-a6d5-44f2-a54e-e6bb71c3d3ef-maas-api-xks-ca\") pod \"4116a3df-a6d5-44f2-a54e-e6bb71c3d3ef\" (UID: \"4116a3df-a6d5-44f2-a54e-e6bb71c3d3ef\") "
Nov 02 15:47:09.914449 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:09.914273    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4116a3df-a6d5-44f2-a54e-e6bb71c3d3ef-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4116a3df-a6d5-44f2-a54e-e6bb71c3d3ef-maas-api-openshift-ca\") pod \"4116a3df-a6d5-44f2-a54e-e6bb71c3d3ef\" (UID: \"4116a3df-a6d5-44f2-a54e-e6bb71c3d3ef\") "
Nov 02 15:47:09.914449 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:09.914329    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4116a3df-a6d5-44f2-a54e-e6bb71c3d3ef-kube-api-access-l25qq\" (UniqueName: \"kubernetes.io/projected/4116a3df-a6d5-44f2-a54e-e6bb71c3d3ef-kube-api-access-l25qq\") pod \"4116a3df-a6d5-44f2-a54e-e6bb71c3d3ef\" (UID: \"4116a3df-a6d5-44f2-a54e-e6bb71c3d3ef\") "
Nov 02 15:47:09.914735 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:09.914686    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4116a3df-a6d5-44f2-a54e-e6bb71c3d3ef-maas-api-openshift-ca" pod "4116a3df-a6d5-44f2-a54e-e6bb71c3d3ef" (UID: "4116a3df-a6d5-44f2-a54e-e6bb71c3d3ef"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:47:09.914952 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:09.914929    5304 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4116a3df-a6d5-44f2-a54e-e6bb71c3d3ef-maas-api-openshift-ca\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:47:09.916959 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:09.916824    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4116a3df-a6d5-44f2-a54e-e6bb71c3d3ef-kube-api-access-l25qq" pod "4116a3df-a6d5-44f2-a54e-e6bb71c3d3ef" (UID: "4116a3df-a6d5-44f2-a54e-e6bb71c3d3ef"). InnerVolumeSpecName "kube-api-access-l25qq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:47:09.916959 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:09.916942    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4116a3df-a6d5-44f2-a54e-e6bb71c3d3ef-maas-api-xks-ca" pod "4116a3df-a6d5-44f2-a54e-e6bb71c3d3ef" (UID: "4116a3df-a6d5-44f2-a54e-e6bb71c3d3ef"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:47:10.016677 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:10.016642    5304 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4116a3df-a6d5-44f2-a54e-e6bb71c3d3ef-maas-api-xks-ca\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:47:10.016677 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:10.016671    5304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l25qq\" (UniqueName: \"kubernetes.io/projected/4116a3df-a6d5-44f2-a54e-e6bb71c3d3ef-kube-api-access-l25qq\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:47:10.653400 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:10.653367    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-b955c5-xrc9j"
Nov 02 15:47:10.653400 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:10.653396    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-b955c5-xrc9j" event={"ID":"4116a3df-a6d5-44f2-a54e-e6bb71c3d3ef","Type":"ContainerDied","Data":"6306f034cd4e99c0fdfa8d25e23b9810bf20eb21fcc3c78442a171acd5168383"}
Nov 02 15:47:10.653676 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:10.653443    5304 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6306f034cd4e99c0fdfa8d25e23b9810bf20eb21fcc3c78442a171acd5168383"
Nov 02 15:47:11.662404 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:11.662372    5304 generic.go:356] "Generic (PLEG): container finished" podID="5086f7a8-e10d-4aca-bc70-950a2f0cd585" containerID="d169223ffff4b48d9ac2bb350d36d9b24a10a9228d6f22abad89d8b26d735c56" exitCode=0
Nov 02 15:47:11.662861 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:11.662446    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-smtdr" event={"ID":"5086f7a8-e10d-4aca-bc70-950a2f0cd585","Type":"ContainerDied","Data":"d169223ffff4b48d9ac2bb350d36d9b24a10a9228d6f22abad89d8b26d735c56"}
Nov 02 15:47:11.665780 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:11.665753    5304 generic.go:356] "Generic (PLEG): container finished" podID="e11f9d23-10db-4bad-a457-7c5db3056979" containerID="b49da12e9c1eaf625d860f22d16960d3837c0f77d80f408d1da9d5f6c30a3835" exitCode=0
Nov 02 15:47:11.665982 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:11.665837    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-2wwx7" event={"ID":"e11f9d23-10db-4bad-a457-7c5db3056979","Type":"ContainerDied","Data":"b49da12e9c1eaf625d860f22d16960d3837c0f77d80f408d1da9d5f6c30a3835"}
Nov 02 15:47:11.665982 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:11.665911    5304 scope.go:122] "RemoveContainer" containerID="82afe6e52444f7b0e63d73c694b765f1e4a10e6f6969c22dddf3cbbce5aedf44"
Nov 02 15:47:11.670411 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:11.670381    5304 generic.go:356] "Generic (PLEG): container finished" podID="93e559b7-2a4c-4533-90d5-5f3fd3956bf0" containerID="6796c86fab92cf70995a4feb5e28881cf73c8de1f4623d22c1dd07a5c4fa9bb4" exitCode=0
Nov 02 15:47:11.670623 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:11.670489    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-lvrfc" event={"ID":"93e559b7-2a4c-4533-90d5-5f3fd3956bf0","Type":"ContainerDied","Data":"6796c86fab92cf70995a4feb5e28881cf73c8de1f4623d22c1dd07a5c4fa9bb4"}
Nov 02 15:47:11.725968 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:11.725940    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-smtdr"
Nov 02 15:47:11.766745 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:11.766714    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-lvrfc"
Nov 02 15:47:11.771795 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:11.771770    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-2wwx7"
Nov 02 15:47:11.845082 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:11.845049    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e11f9d23-10db-4bad-a457-7c5db3056979-istio-token\" (UniqueName: \"kubernetes.io/projected/e11f9d23-10db-4bad-a457-7c5db3056979-istio-token\") pod \"e11f9d23-10db-4bad-a457-7c5db3056979\" (UID: \"e11f9d23-10db-4bad-a457-7c5db3056979\") "
Nov 02 15:47:11.845325 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:11.845146    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e11f9d23-10db-4bad-a457-7c5db3056979-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e11f9d23-10db-4bad-a457-7c5db3056979-workload-certs\") pod \"e11f9d23-10db-4bad-a457-7c5db3056979\" (UID: \"e11f9d23-10db-4bad-a457-7c5db3056979\") "
Nov 02 15:47:11.845325 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:11.845184    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93e559b7-2a4c-4533-90d5-5f3fd3956bf0-istio-token\" (UniqueName: \"kubernetes.io/projected/93e559b7-2a4c-4533-90d5-5f3fd3956bf0-istio-token\") pod \"93e559b7-2a4c-4533-90d5-5f3fd3956bf0\" (UID: \"93e559b7-2a4c-4533-90d5-5f3fd3956bf0\") "
Nov 02 15:47:11.845325 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:11.845224    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93e559b7-2a4c-4533-90d5-5f3fd3956bf0-kube-api-access-sv2vr\" (UniqueName: \"kubernetes.io/projected/93e559b7-2a4c-4533-90d5-5f3fd3956bf0-kube-api-access-sv2vr\") pod \"93e559b7-2a4c-4533-90d5-5f3fd3956bf0\" (UID: \"93e559b7-2a4c-4533-90d5-5f3fd3956bf0\") "
Nov 02 15:47:11.845325 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:11.845268    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/93e559b7-2a4c-4533-90d5-5f3fd3956bf0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93e559b7-2a4c-4533-90d5-5f3fd3956bf0-istiod-ca-cert\") pod \"93e559b7-2a4c-4533-90d5-5f3fd3956bf0\" (UID: \"93e559b7-2a4c-4533-90d5-5f3fd3956bf0\") "
Nov 02 15:47:11.845325 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:11.845318    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93e559b7-2a4c-4533-90d5-5f3fd3956bf0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93e559b7-2a4c-4533-90d5-5f3fd3956bf0-workload-certs\") pod \"93e559b7-2a4c-4533-90d5-5f3fd3956bf0\" (UID: \"93e559b7-2a4c-4533-90d5-5f3fd3956bf0\") "
Nov 02 15:47:11.845539 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:11.845354    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5086f7a8-e10d-4aca-bc70-950a2f0cd585-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5086f7a8-e10d-4aca-bc70-950a2f0cd585-credential-socket\") pod \"5086f7a8-e10d-4aca-bc70-950a2f0cd585\" (UID: \"5086f7a8-e10d-4aca-bc70-950a2f0cd585\") "
Nov 02 15:47:11.845539 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:11.845423    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5086f7a8-e10d-4aca-bc70-950a2f0cd585-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5086f7a8-e10d-4aca-bc70-950a2f0cd585-workload-certs\") pod \"5086f7a8-e10d-4aca-bc70-950a2f0cd585\" (UID: \"5086f7a8-e10d-4aca-bc70-950a2f0cd585\") "
Nov 02 15:47:11.845539 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:11.845462    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5086f7a8-e10d-4aca-bc70-950a2f0cd585-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5086f7a8-e10d-4aca-bc70-950a2f0cd585-istio-podinfo\") pod \"5086f7a8-e10d-4aca-bc70-950a2f0cd585\" (UID: \"5086f7a8-e10d-4aca-bc70-950a2f0cd585\") "
Nov 02 15:47:11.845539 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:11.845491    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e11f9d23-10db-4bad-a457-7c5db3056979-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e11f9d23-10db-4bad-a457-7c5db3056979-istio-podinfo\") pod \"e11f9d23-10db-4bad-a457-7c5db3056979\" (UID: \"e11f9d23-10db-4bad-a457-7c5db3056979\") "
Nov 02 15:47:11.845539 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:11.845530    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93e559b7-2a4c-4533-90d5-5f3fd3956bf0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93e559b7-2a4c-4533-90d5-5f3fd3956bf0-istio-envoy\") pod \"93e559b7-2a4c-4533-90d5-5f3fd3956bf0\" (UID: \"93e559b7-2a4c-4533-90d5-5f3fd3956bf0\") "
Nov 02 15:47:11.845783 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:11.845559    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5086f7a8-e10d-4aca-bc70-950a2f0cd585-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5086f7a8-e10d-4aca-bc70-950a2f0cd585-istio-envoy\") pod \"5086f7a8-e10d-4aca-bc70-950a2f0cd585\" (UID: \"5086f7a8-e10d-4aca-bc70-950a2f0cd585\") "
Nov 02 15:47:11.845783 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:11.845588    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/93e559b7-2a4c-4533-90d5-5f3fd3956bf0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93e559b7-2a4c-4533-90d5-5f3fd3956bf0-istio-podinfo\") pod \"93e559b7-2a4c-4533-90d5-5f3fd3956bf0\" (UID: \"93e559b7-2a4c-4533-90d5-5f3fd3956bf0\") "
Nov 02 15:47:11.845783 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:11.845625    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5086f7a8-e10d-4aca-bc70-950a2f0cd585-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5086f7a8-e10d-4aca-bc70-950a2f0cd585-istiod-ca-cert\") pod \"5086f7a8-e10d-4aca-bc70-950a2f0cd585\" (UID: \"5086f7a8-e10d-4aca-bc70-950a2f0cd585\") "
Nov 02 15:47:11.846802 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:11.846767    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5086f7a8-e10d-4aca-bc70-950a2f0cd585-istiod-ca-cert" pod "5086f7a8-e10d-4aca-bc70-950a2f0cd585" (UID: "5086f7a8-e10d-4aca-bc70-950a2f0cd585"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:47:11.847219 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:11.847191    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93e559b7-2a4c-4533-90d5-5f3fd3956bf0-istiod-ca-cert" pod "93e559b7-2a4c-4533-90d5-5f3fd3956bf0" (UID: "93e559b7-2a4c-4533-90d5-5f3fd3956bf0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:47:11.847451 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:11.847428    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e11f9d23-10db-4bad-a457-7c5db3056979-workload-certs" pod "e11f9d23-10db-4bad-a457-7c5db3056979" (UID: "e11f9d23-10db-4bad-a457-7c5db3056979"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:47:11.847829 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:11.847617    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5086f7a8-e10d-4aca-bc70-950a2f0cd585-workload-certs" pod "5086f7a8-e10d-4aca-bc70-950a2f0cd585" (UID: "5086f7a8-e10d-4aca-bc70-950a2f0cd585"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:47:11.847963 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:11.847942    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93e559b7-2a4c-4533-90d5-5f3fd3956bf0-workload-certs" pod "93e559b7-2a4c-4533-90d5-5f3fd3956bf0" (UID: "93e559b7-2a4c-4533-90d5-5f3fd3956bf0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:47:11.848730 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:11.848709    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5086f7a8-e10d-4aca-bc70-950a2f0cd585-credential-socket" pod "5086f7a8-e10d-4aca-bc70-950a2f0cd585" (UID: "5086f7a8-e10d-4aca-bc70-950a2f0cd585"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:47:11.850523 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:11.850489    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e11f9d23-10db-4bad-a457-7c5db3056979-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e11f9d23-10db-4bad-a457-7c5db3056979-istio-envoy\") pod \"e11f9d23-10db-4bad-a457-7c5db3056979\" (UID: \"e11f9d23-10db-4bad-a457-7c5db3056979\") "
Nov 02 15:47:11.850631 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:11.850570    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5086f7a8-e10d-4aca-bc70-950a2f0cd585-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5086f7a8-e10d-4aca-bc70-950a2f0cd585-istio-data\") pod \"5086f7a8-e10d-4aca-bc70-950a2f0cd585\" (UID: \"5086f7a8-e10d-4aca-bc70-950a2f0cd585\") "
Nov 02 15:47:11.850742 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:11.850722    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93e559b7-2a4c-4533-90d5-5f3fd3956bf0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93e559b7-2a4c-4533-90d5-5f3fd3956bf0-istio-data\") pod \"93e559b7-2a4c-4533-90d5-5f3fd3956bf0\" (UID: \"93e559b7-2a4c-4533-90d5-5f3fd3956bf0\") "
Nov 02 15:47:11.850839 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:11.850782    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93e559b7-2a4c-4533-90d5-5f3fd3956bf0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93e559b7-2a4c-4533-90d5-5f3fd3956bf0-workload-socket\") pod \"93e559b7-2a4c-4533-90d5-5f3fd3956bf0\" (UID: \"93e559b7-2a4c-4533-90d5-5f3fd3956bf0\") "
Nov 02 15:47:11.850897 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:11.850851    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93e559b7-2a4c-4533-90d5-5f3fd3956bf0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93e559b7-2a4c-4533-90d5-5f3fd3956bf0-credential-socket\") pod \"93e559b7-2a4c-4533-90d5-5f3fd3956bf0\" (UID: \"93e559b7-2a4c-4533-90d5-5f3fd3956bf0\") "
Nov 02 15:47:11.850950 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:11.850928    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e11f9d23-10db-4bad-a457-7c5db3056979-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e11f9d23-10db-4bad-a457-7c5db3056979-credential-socket\") pod \"e11f9d23-10db-4bad-a457-7c5db3056979\" (UID: \"e11f9d23-10db-4bad-a457-7c5db3056979\") "
Nov 02 15:47:11.851007 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:11.850962    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e11f9d23-10db-4bad-a457-7c5db3056979-kube-api-access-x4p6m\" (UniqueName: \"kubernetes.io/projected/e11f9d23-10db-4bad-a457-7c5db3056979-kube-api-access-x4p6m\") pod \"e11f9d23-10db-4bad-a457-7c5db3056979\" (UID: \"e11f9d23-10db-4bad-a457-7c5db3056979\") "
Nov 02 15:47:11.851061 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:11.851017    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5086f7a8-e10d-4aca-bc70-950a2f0cd585-istio-token\" (UniqueName: \"kubernetes.io/projected/5086f7a8-e10d-4aca-bc70-950a2f0cd585-istio-token\") pod \"5086f7a8-e10d-4aca-bc70-950a2f0cd585\" (UID: \"5086f7a8-e10d-4aca-bc70-950a2f0cd585\") "
Nov 02 15:47:11.851116 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:11.851079    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e11f9d23-10db-4bad-a457-7c5db3056979-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e11f9d23-10db-4bad-a457-7c5db3056979-istio-data\") pod \"e11f9d23-10db-4bad-a457-7c5db3056979\" (UID: \"e11f9d23-10db-4bad-a457-7c5db3056979\") "
Nov 02 15:47:11.852358 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:11.851702    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93e559b7-2a4c-4533-90d5-5f3fd3956bf0-credential-socket" pod "93e559b7-2a4c-4533-90d5-5f3fd3956bf0" (UID: "93e559b7-2a4c-4533-90d5-5f3fd3956bf0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:47:11.852358 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:11.851976    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5086f7a8-e10d-4aca-bc70-950a2f0cd585-istio-data" pod "5086f7a8-e10d-4aca-bc70-950a2f0cd585" (UID: "5086f7a8-e10d-4aca-bc70-950a2f0cd585"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:47:11.852358 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:11.852188    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93e559b7-2a4c-4533-90d5-5f3fd3956bf0-istio-data" pod "93e559b7-2a4c-4533-90d5-5f3fd3956bf0" (UID: "93e559b7-2a4c-4533-90d5-5f3fd3956bf0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:47:11.852941 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:11.852660    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93e559b7-2a4c-4533-90d5-5f3fd3956bf0-workload-socket" pod "93e559b7-2a4c-4533-90d5-5f3fd3956bf0" (UID: "93e559b7-2a4c-4533-90d5-5f3fd3956bf0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:47:11.853505 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:11.853460    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93e559b7-2a4c-4533-90d5-5f3fd3956bf0-istio-token" pod "93e559b7-2a4c-4533-90d5-5f3fd3956bf0" (UID: "93e559b7-2a4c-4533-90d5-5f3fd3956bf0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:47:11.854572 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:11.853684    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e11f9d23-10db-4bad-a457-7c5db3056979-credential-socket" pod "e11f9d23-10db-4bad-a457-7c5db3056979" (UID: "e11f9d23-10db-4bad-a457-7c5db3056979"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:47:11.854572 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:11.854045    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e11f9d23-10db-4bad-a457-7c5db3056979-istio-data" pod "e11f9d23-10db-4bad-a457-7c5db3056979" (UID: "e11f9d23-10db-4bad-a457-7c5db3056979"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:47:11.854572 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:11.854371    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e11f9d23-10db-4bad-a457-7c5db3056979-istio-envoy" pod "e11f9d23-10db-4bad-a457-7c5db3056979" (UID: "e11f9d23-10db-4bad-a457-7c5db3056979"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:47:11.854903 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:11.854868    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e11f9d23-10db-4bad-a457-7c5db3056979-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e11f9d23-10db-4bad-a457-7c5db3056979-istiod-ca-cert\") pod \"e11f9d23-10db-4bad-a457-7c5db3056979\" (UID: \"e11f9d23-10db-4bad-a457-7c5db3056979\") "
Nov 02 15:47:11.854980 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:11.854943    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e11f9d23-10db-4bad-a457-7c5db3056979-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e11f9d23-10db-4bad-a457-7c5db3056979-workload-socket\") pod \"e11f9d23-10db-4bad-a457-7c5db3056979\" (UID: \"e11f9d23-10db-4bad-a457-7c5db3056979\") "
Nov 02 15:47:11.854980 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:11.854976    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5086f7a8-e10d-4aca-bc70-950a2f0cd585-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5086f7a8-e10d-4aca-bc70-950a2f0cd585-workload-socket\") pod \"5086f7a8-e10d-4aca-bc70-950a2f0cd585\" (UID: \"5086f7a8-e10d-4aca-bc70-950a2f0cd585\") "
Nov 02 15:47:11.855107 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:11.855047    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5086f7a8-e10d-4aca-bc70-950a2f0cd585-kube-api-access-vrvm8\" (UniqueName: \"kubernetes.io/projected/5086f7a8-e10d-4aca-bc70-950a2f0cd585-kube-api-access-vrvm8\") pod \"5086f7a8-e10d-4aca-bc70-950a2f0cd585\" (UID: \"5086f7a8-e10d-4aca-bc70-950a2f0cd585\") "
Nov 02 15:47:11.855364 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:11.855339    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e11f9d23-10db-4bad-a457-7c5db3056979-istiod-ca-cert" pod "e11f9d23-10db-4bad-a457-7c5db3056979" (UID: "e11f9d23-10db-4bad-a457-7c5db3056979"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:47:11.855780 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:11.855756    5304 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93e559b7-2a4c-4533-90d5-5f3fd3956bf0-istio-data\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:47:11.855872 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:11.855787    5304 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93e559b7-2a4c-4533-90d5-5f3fd3956bf0-workload-socket\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:47:11.855872 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:11.855801    5304 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93e559b7-2a4c-4533-90d5-5f3fd3956bf0-credential-socket\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:47:11.855872 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:11.855817    5304 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e11f9d23-10db-4bad-a457-7c5db3056979-credential-socket\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:47:11.855872 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:11.855831    5304 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e11f9d23-10db-4bad-a457-7c5db3056979-istio-data\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:47:11.855872 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:11.855844    5304 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e11f9d23-10db-4bad-a457-7c5db3056979-istiod-ca-cert\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:47:11.855872 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:11.855856    5304 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e11f9d23-10db-4bad-a457-7c5db3056979-workload-certs\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:47:11.855872 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:11.855869    5304 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/93e559b7-2a4c-4533-90d5-5f3fd3956bf0-istio-token\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:47:11.856272 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:11.855881    5304 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93e559b7-2a4c-4533-90d5-5f3fd3956bf0-istiod-ca-cert\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:47:11.856272 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:11.855893    5304 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93e559b7-2a4c-4533-90d5-5f3fd3956bf0-workload-certs\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:47:11.856272 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:11.855906    5304 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5086f7a8-e10d-4aca-bc70-950a2f0cd585-credential-socket\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:47:11.856272 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:11.855920    5304 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5086f7a8-e10d-4aca-bc70-950a2f0cd585-workload-certs\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:47:11.856272 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:11.855934    5304 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5086f7a8-e10d-4aca-bc70-950a2f0cd585-istiod-ca-cert\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:47:11.856272 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:11.855945    5304 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e11f9d23-10db-4bad-a457-7c5db3056979-istio-envoy\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:47:11.856272 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:11.855959    5304 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5086f7a8-e10d-4aca-bc70-950a2f0cd585-istio-data\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:47:11.856272 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:11.856181    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e11f9d23-10db-4bad-a457-7c5db3056979-workload-socket" pod "e11f9d23-10db-4bad-a457-7c5db3056979" (UID: "e11f9d23-10db-4bad-a457-7c5db3056979"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:47:11.856272 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:11.856280    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5086f7a8-e10d-4aca-bc70-950a2f0cd585-workload-socket" pod "5086f7a8-e10d-4aca-bc70-950a2f0cd585" (UID: "5086f7a8-e10d-4aca-bc70-950a2f0cd585"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:47:11.857498 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:11.857473    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e11f9d23-10db-4bad-a457-7c5db3056979-istio-token" pod "e11f9d23-10db-4bad-a457-7c5db3056979" (UID: "e11f9d23-10db-4bad-a457-7c5db3056979"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:47:11.859820 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:11.859784    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e11f9d23-10db-4bad-a457-7c5db3056979-kube-api-access-x4p6m" pod "e11f9d23-10db-4bad-a457-7c5db3056979" (UID: "e11f9d23-10db-4bad-a457-7c5db3056979"). InnerVolumeSpecName "kube-api-access-x4p6m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:47:11.860943 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:11.860920    5304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-28da58-openshift-default-76446c9f79-rg6rm"]
Nov 02 15:47:11.861464 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:11.861439    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5086f7a8-e10d-4aca-bc70-950a2f0cd585-kube-api-access-vrvm8" pod "5086f7a8-e10d-4aca-bc70-950a2f0cd585" (UID: "5086f7a8-e10d-4aca-bc70-950a2f0cd585"). InnerVolumeSpecName "kube-api-access-vrvm8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:47:11.861698 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:11.861672    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5086f7a8-e10d-4aca-bc70-950a2f0cd585-istio-token" pod "5086f7a8-e10d-4aca-bc70-950a2f0cd585" (UID: "5086f7a8-e10d-4aca-bc70-950a2f0cd585"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:47:11.861813 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:11.861721    5304 cpu_manager.go:414] "containerMap: removing container" podUID="5086f7a8-e10d-4aca-bc70-950a2f0cd585" containerName="istio-proxy"
Nov 02 15:47:11.861813 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:11.861736    5304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5086f7a8-e10d-4aca-bc70-950a2f0cd585" containerName="istio-proxy"
Nov 02 15:47:11.861813 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:11.861747    5304 cpu_manager.go:414] "containerMap: removing container" podUID="e11f9d23-10db-4bad-a457-7c5db3056979" containerName="istio-proxy"
Nov 02 15:47:11.861813 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:11.861753    5304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e11f9d23-10db-4bad-a457-7c5db3056979" containerName="istio-proxy"
Nov 02 15:47:11.861813 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:11.861785    5304 cpu_manager.go:414] "containerMap: removing container" podUID="e11f9d23-10db-4bad-a457-7c5db3056979" containerName="istio-proxy"
Nov 02 15:47:11.861813 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:11.861792    5304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e11f9d23-10db-4bad-a457-7c5db3056979" containerName="istio-proxy"
Nov 02 15:47:11.861813 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:11.861809    5304 cpu_manager.go:414] "containerMap: removing container" podUID="4116a3df-a6d5-44f2-a54e-e6bb71c3d3ef" containerName="revoke-keys"
Nov 02 15:47:11.861813 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:11.861813    5304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4116a3df-a6d5-44f2-a54e-e6bb71c3d3ef" containerName="revoke-keys"
Nov 02 15:47:11.862282 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:11.861825    5304 cpu_manager.go:414] "containerMap: removing container" podUID="93e559b7-2a4c-4533-90d5-5f3fd3956bf0" containerName="istio-proxy"
Nov 02 15:47:11.862282 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:11.861830    5304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93e559b7-2a4c-4533-90d5-5f3fd3956bf0" containerName="istio-proxy"
Nov 02 15:47:11.862282 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:11.861921    5304 memory_manager.go:365] "RemoveStaleState removing state" podUID="e11f9d23-10db-4bad-a457-7c5db3056979" containerName="istio-proxy"
Nov 02 15:47:11.862282 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:11.861933    5304 memory_manager.go:365] "RemoveStaleState removing state" podUID="4116a3df-a6d5-44f2-a54e-e6bb71c3d3ef" containerName="revoke-keys"
Nov 02 15:47:11.862282 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:11.861942    5304 memory_manager.go:365] "RemoveStaleState removing state" podUID="93e559b7-2a4c-4533-90d5-5f3fd3956bf0" containerName="istio-proxy"
Nov 02 15:47:11.862282 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:11.861951    5304 memory_manager.go:365] "RemoveStaleState removing state" podUID="5086f7a8-e10d-4aca-bc70-950a2f0cd585" containerName="istio-proxy"
Nov 02 15:47:11.865458 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:11.865430    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5086f7a8-e10d-4aca-bc70-950a2f0cd585-istio-envoy" pod "5086f7a8-e10d-4aca-bc70-950a2f0cd585" (UID: "5086f7a8-e10d-4aca-bc70-950a2f0cd585"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:47:11.865458 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:11.865446    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e11f9d23-10db-4bad-a457-7c5db3056979-istio-podinfo" pod "e11f9d23-10db-4bad-a457-7c5db3056979" (UID: "e11f9d23-10db-4bad-a457-7c5db3056979"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 15:47:11.865593 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:11.865459    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93e559b7-2a4c-4533-90d5-5f3fd3956bf0-istio-envoy" pod "93e559b7-2a4c-4533-90d5-5f3fd3956bf0" (UID: "93e559b7-2a4c-4533-90d5-5f3fd3956bf0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:47:11.865593 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:11.865444    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5086f7a8-e10d-4aca-bc70-950a2f0cd585-istio-podinfo" pod "5086f7a8-e10d-4aca-bc70-950a2f0cd585" (UID: "5086f7a8-e10d-4aca-bc70-950a2f0cd585"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 15:47:11.865593 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:11.865512    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93e559b7-2a4c-4533-90d5-5f3fd3956bf0-kube-api-access-sv2vr" pod "93e559b7-2a4c-4533-90d5-5f3fd3956bf0" (UID: "93e559b7-2a4c-4533-90d5-5f3fd3956bf0"). InnerVolumeSpecName "kube-api-access-sv2vr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:47:11.866620 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:11.866598    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-28da58-openshift-default-76446c9f79-rg6rm"
Nov 02 15:47:11.866707 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:11.866677    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/93e559b7-2a4c-4533-90d5-5f3fd3956bf0-istio-podinfo" pod "93e559b7-2a4c-4533-90d5-5f3fd3956bf0" (UID: "93e559b7-2a4c-4533-90d5-5f3fd3956bf0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 15:47:11.883428 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:11.883400    5304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-28da58-openshift-default-dockercfg-lg4q9\""
Nov 02 15:47:11.886518 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:11.886484    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-28da58-openshift-default-76446c9f79-rg6rm"]
Nov 02 15:47:11.972846 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:11.971272    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e80dea97-6fba-4ad0-b574-da148b1979db-istio-podinfo\") pod \"e2e-shared-b-w3-28da58-openshift-default-76446c9f79-rg6rm\" (UID: \"e80dea97-6fba-4ad0-b574-da148b1979db\") " pod="openshift-ingress/e2e-shared-b-w3-28da58-openshift-default-76446c9f79-rg6rm"
Nov 02 15:47:11.978246 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:11.978206    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e80dea97-6fba-4ad0-b574-da148b1979db-istio-envoy\") pod \"e2e-shared-b-w3-28da58-openshift-default-76446c9f79-rg6rm\" (UID: \"e80dea97-6fba-4ad0-b574-da148b1979db\") " pod="openshift-ingress/e2e-shared-b-w3-28da58-openshift-default-76446c9f79-rg6rm"
Nov 02 15:47:11.978508 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:11.978489    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e80dea97-6fba-4ad0-b574-da148b1979db-istiod-ca-cert\") pod \"e2e-shared-b-w3-28da58-openshift-default-76446c9f79-rg6rm\" (UID: \"e80dea97-6fba-4ad0-b574-da148b1979db\") " pod="openshift-ingress/e2e-shared-b-w3-28da58-openshift-default-76446c9f79-rg6rm"
Nov 02 15:47:11.978665 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:11.978649    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e80dea97-6fba-4ad0-b574-da148b1979db-workload-certs\") pod \"e2e-shared-b-w3-28da58-openshift-default-76446c9f79-rg6rm\" (UID: \"e80dea97-6fba-4ad0-b574-da148b1979db\") " pod="openshift-ingress/e2e-shared-b-w3-28da58-openshift-default-76446c9f79-rg6rm"
Nov 02 15:47:11.978876 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:11.978861    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e80dea97-6fba-4ad0-b574-da148b1979db-istio-token\") pod \"e2e-shared-b-w3-28da58-openshift-default-76446c9f79-rg6rm\" (UID: \"e80dea97-6fba-4ad0-b574-da148b1979db\") " pod="openshift-ingress/e2e-shared-b-w3-28da58-openshift-default-76446c9f79-rg6rm"
Nov 02 15:47:11.979044 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:11.979029    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e80dea97-6fba-4ad0-b574-da148b1979db-workload-socket\") pod \"e2e-shared-b-w3-28da58-openshift-default-76446c9f79-rg6rm\" (UID: \"e80dea97-6fba-4ad0-b574-da148b1979db\") " pod="openshift-ingress/e2e-shared-b-w3-28da58-openshift-default-76446c9f79-rg6rm"
Nov 02 15:47:11.979321 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:11.979303    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e80dea97-6fba-4ad0-b574-da148b1979db-istio-data\") pod \"e2e-shared-b-w3-28da58-openshift-default-76446c9f79-rg6rm\" (UID: \"e80dea97-6fba-4ad0-b574-da148b1979db\") " pod="openshift-ingress/e2e-shared-b-w3-28da58-openshift-default-76446c9f79-rg6rm"
Nov 02 15:47:11.979489 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:11.979475    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e80dea97-6fba-4ad0-b574-da148b1979db-credential-socket\") pod \"e2e-shared-b-w3-28da58-openshift-default-76446c9f79-rg6rm\" (UID: \"e80dea97-6fba-4ad0-b574-da148b1979db\") " pod="openshift-ingress/e2e-shared-b-w3-28da58-openshift-default-76446c9f79-rg6rm"
Nov 02 15:47:11.979729 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:11.979694    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wzcdr\" (UniqueName: \"kubernetes.io/projected/e80dea97-6fba-4ad0-b574-da148b1979db-kube-api-access-wzcdr\") pod \"e2e-shared-b-w3-28da58-openshift-default-76446c9f79-rg6rm\" (UID: \"e80dea97-6fba-4ad0-b574-da148b1979db\") " pod="openshift-ingress/e2e-shared-b-w3-28da58-openshift-default-76446c9f79-rg6rm"
Nov 02 15:47:11.980065 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:11.980042    5304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x4p6m\" (UniqueName: \"kubernetes.io/projected/e11f9d23-10db-4bad-a457-7c5db3056979-kube-api-access-x4p6m\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:47:11.980204 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:11.980181    5304 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5086f7a8-e10d-4aca-bc70-950a2f0cd585-istio-token\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:47:11.980349 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:11.980334    5304 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e11f9d23-10db-4bad-a457-7c5db3056979-workload-socket\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:47:11.980518 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:11.980494    5304 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5086f7a8-e10d-4aca-bc70-950a2f0cd585-workload-socket\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:47:11.980669 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:11.980655    5304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vrvm8\" (UniqueName: \"kubernetes.io/projected/5086f7a8-e10d-4aca-bc70-950a2f0cd585-kube-api-access-vrvm8\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:47:11.980778 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:11.980766    5304 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e11f9d23-10db-4bad-a457-7c5db3056979-istio-token\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:47:11.980879 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:11.980869    5304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sv2vr\" (UniqueName: \"kubernetes.io/projected/93e559b7-2a4c-4533-90d5-5f3fd3956bf0-kube-api-access-sv2vr\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:47:11.980973 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:11.980961    5304 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5086f7a8-e10d-4aca-bc70-950a2f0cd585-istio-podinfo\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:47:11.981058 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:11.981048    5304 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e11f9d23-10db-4bad-a457-7c5db3056979-istio-podinfo\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:47:11.981159 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:11.981146    5304 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93e559b7-2a4c-4533-90d5-5f3fd3956bf0-istio-envoy\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:47:11.981285 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:11.981269    5304 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5086f7a8-e10d-4aca-bc70-950a2f0cd585-istio-envoy\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:47:11.981391 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:11.981378    5304 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93e559b7-2a4c-4533-90d5-5f3fd3956bf0-istio-podinfo\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:47:12.082761 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:12.082724    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e80dea97-6fba-4ad0-b574-da148b1979db-istio-data\") pod \"e2e-shared-b-w3-28da58-openshift-default-76446c9f79-rg6rm\" (UID: \"e80dea97-6fba-4ad0-b574-da148b1979db\") " pod="openshift-ingress/e2e-shared-b-w3-28da58-openshift-default-76446c9f79-rg6rm"
Nov 02 15:47:12.082761 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:12.082767    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e80dea97-6fba-4ad0-b574-da148b1979db-credential-socket\") pod \"e2e-shared-b-w3-28da58-openshift-default-76446c9f79-rg6rm\" (UID: \"e80dea97-6fba-4ad0-b574-da148b1979db\") " pod="openshift-ingress/e2e-shared-b-w3-28da58-openshift-default-76446c9f79-rg6rm"
Nov 02 15:47:12.083037 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:12.082897    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wzcdr\" (UniqueName: \"kubernetes.io/projected/e80dea97-6fba-4ad0-b574-da148b1979db-kube-api-access-wzcdr\") pod \"e2e-shared-b-w3-28da58-openshift-default-76446c9f79-rg6rm\" (UID: \"e80dea97-6fba-4ad0-b574-da148b1979db\") " pod="openshift-ingress/e2e-shared-b-w3-28da58-openshift-default-76446c9f79-rg6rm"
Nov 02 15:47:12.083037 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:12.082992    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e80dea97-6fba-4ad0-b574-da148b1979db-istio-podinfo\") pod \"e2e-shared-b-w3-28da58-openshift-default-76446c9f79-rg6rm\" (UID: \"e80dea97-6fba-4ad0-b574-da148b1979db\") " pod="openshift-ingress/e2e-shared-b-w3-28da58-openshift-default-76446c9f79-rg6rm"
Nov 02 15:47:12.083213 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:12.083057    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e80dea97-6fba-4ad0-b574-da148b1979db-istio-envoy\") pod \"e2e-shared-b-w3-28da58-openshift-default-76446c9f79-rg6rm\" (UID: \"e80dea97-6fba-4ad0-b574-da148b1979db\") " pod="openshift-ingress/e2e-shared-b-w3-28da58-openshift-default-76446c9f79-rg6rm"
Nov 02 15:47:12.083213 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:12.083109    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e80dea97-6fba-4ad0-b574-da148b1979db-istiod-ca-cert\") pod \"e2e-shared-b-w3-28da58-openshift-default-76446c9f79-rg6rm\" (UID: \"e80dea97-6fba-4ad0-b574-da148b1979db\") " pod="openshift-ingress/e2e-shared-b-w3-28da58-openshift-default-76446c9f79-rg6rm"
Nov 02 15:47:12.083213 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:12.083145    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e80dea97-6fba-4ad0-b574-da148b1979db-credential-socket\") pod \"e2e-shared-b-w3-28da58-openshift-default-76446c9f79-rg6rm\" (UID: \"e80dea97-6fba-4ad0-b574-da148b1979db\") " pod="openshift-ingress/e2e-shared-b-w3-28da58-openshift-default-76446c9f79-rg6rm"
Nov 02 15:47:12.083213 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:12.083174    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e80dea97-6fba-4ad0-b574-da148b1979db-workload-certs\") pod \"e2e-shared-b-w3-28da58-openshift-default-76446c9f79-rg6rm\" (UID: \"e80dea97-6fba-4ad0-b574-da148b1979db\") " pod="openshift-ingress/e2e-shared-b-w3-28da58-openshift-default-76446c9f79-rg6rm"
Nov 02 15:47:12.083444 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:12.083247    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e80dea97-6fba-4ad0-b574-da148b1979db-istio-token\") pod \"e2e-shared-b-w3-28da58-openshift-default-76446c9f79-rg6rm\" (UID: \"e80dea97-6fba-4ad0-b574-da148b1979db\") " pod="openshift-ingress/e2e-shared-b-w3-28da58-openshift-default-76446c9f79-rg6rm"
Nov 02 15:47:12.083444 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:12.083247    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e80dea97-6fba-4ad0-b574-da148b1979db-istio-data\") pod \"e2e-shared-b-w3-28da58-openshift-default-76446c9f79-rg6rm\" (UID: \"e80dea97-6fba-4ad0-b574-da148b1979db\") " pod="openshift-ingress/e2e-shared-b-w3-28da58-openshift-default-76446c9f79-rg6rm"
Nov 02 15:47:12.083444 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:12.083314    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e80dea97-6fba-4ad0-b574-da148b1979db-workload-socket\") pod \"e2e-shared-b-w3-28da58-openshift-default-76446c9f79-rg6rm\" (UID: \"e80dea97-6fba-4ad0-b574-da148b1979db\") " pod="openshift-ingress/e2e-shared-b-w3-28da58-openshift-default-76446c9f79-rg6rm"
Nov 02 15:47:12.083872 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:12.083506    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e80dea97-6fba-4ad0-b574-da148b1979db-workload-certs\") pod \"e2e-shared-b-w3-28da58-openshift-default-76446c9f79-rg6rm\" (UID: \"e80dea97-6fba-4ad0-b574-da148b1979db\") " pod="openshift-ingress/e2e-shared-b-w3-28da58-openshift-default-76446c9f79-rg6rm"
Nov 02 15:47:12.083872 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:12.083724    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e80dea97-6fba-4ad0-b574-da148b1979db-workload-socket\") pod \"e2e-shared-b-w3-28da58-openshift-default-76446c9f79-rg6rm\" (UID: \"e80dea97-6fba-4ad0-b574-da148b1979db\") " pod="openshift-ingress/e2e-shared-b-w3-28da58-openshift-default-76446c9f79-rg6rm"
Nov 02 15:47:12.083872 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:12.083784    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e80dea97-6fba-4ad0-b574-da148b1979db-istiod-ca-cert\") pod \"e2e-shared-b-w3-28da58-openshift-default-76446c9f79-rg6rm\" (UID: \"e80dea97-6fba-4ad0-b574-da148b1979db\") " pod="openshift-ingress/e2e-shared-b-w3-28da58-openshift-default-76446c9f79-rg6rm"
Nov 02 15:47:12.085449 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:12.085429    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e80dea97-6fba-4ad0-b574-da148b1979db-istio-envoy\") pod \"e2e-shared-b-w3-28da58-openshift-default-76446c9f79-rg6rm\" (UID: \"e80dea97-6fba-4ad0-b574-da148b1979db\") " pod="openshift-ingress/e2e-shared-b-w3-28da58-openshift-default-76446c9f79-rg6rm"
Nov 02 15:47:12.085553 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:12.085536    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e80dea97-6fba-4ad0-b574-da148b1979db-istio-podinfo\") pod \"e2e-shared-b-w3-28da58-openshift-default-76446c9f79-rg6rm\" (UID: \"e80dea97-6fba-4ad0-b574-da148b1979db\") " pod="openshift-ingress/e2e-shared-b-w3-28da58-openshift-default-76446c9f79-rg6rm"
Nov 02 15:47:12.106686 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:12.106653    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e80dea97-6fba-4ad0-b574-da148b1979db-istio-token\") pod \"e2e-shared-b-w3-28da58-openshift-default-76446c9f79-rg6rm\" (UID: \"e80dea97-6fba-4ad0-b574-da148b1979db\") " pod="openshift-ingress/e2e-shared-b-w3-28da58-openshift-default-76446c9f79-rg6rm"
Nov 02 15:47:12.106862 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:12.106771    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wzcdr\" (UniqueName: \"kubernetes.io/projected/e80dea97-6fba-4ad0-b574-da148b1979db-kube-api-access-wzcdr\") pod \"e2e-shared-b-w3-28da58-openshift-default-76446c9f79-rg6rm\" (UID: \"e80dea97-6fba-4ad0-b574-da148b1979db\") " pod="openshift-ingress/e2e-shared-b-w3-28da58-openshift-default-76446c9f79-rg6rm"
Nov 02 15:47:12.182108 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:12.182074    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-28da58-openshift-default-76446c9f79-rg6rm"
Nov 02 15:47:12.509045 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:12.509011    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-28da58-openshift-default-76446c9f79-rg6rm"]
Nov 02 15:47:12.511226 ip-10-0-135-138 kubenswrapper[5304]: W1102 15:47:12.511197    5304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode80dea97_6fba_4ad0_b574_da148b1979db.slice/crio-8e19dc84487f1a25cd147f8dabc05fabcc2fba0d4d8c8d732a1cb6ee607f24f0 WatchSource:0}: Error finding container 8e19dc84487f1a25cd147f8dabc05fabcc2fba0d4d8c8d732a1cb6ee607f24f0: Status 404 returned error can't find the container with id 8e19dc84487f1a25cd147f8dabc05fabcc2fba0d4d8c8d732a1cb6ee607f24f0
Nov 02 15:47:12.514408 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:12.514370    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:47:12.514548 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:12.514462    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:47:12.514548 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:12.514512    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:47:12.680590 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:12.680562    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-lvrfc"
Nov 02 15:47:12.680590 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:12.680579    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-lvrfc" event={"ID":"93e559b7-2a4c-4533-90d5-5f3fd3956bf0","Type":"ContainerDied","Data":"624678ae56f61e393423e502c4620825666ba829d18033e1df0baaf2b64d3778"}
Nov 02 15:47:12.681160 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:12.680654    5304 scope.go:122] "RemoveContainer" containerID="6796c86fab92cf70995a4feb5e28881cf73c8de1f4623d22c1dd07a5c4fa9bb4"
Nov 02 15:47:12.682752 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:12.682724    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-28da58-openshift-default-76446c9f79-rg6rm" event={"ID":"e80dea97-6fba-4ad0-b574-da148b1979db","Type":"ContainerStarted","Data":"af24a6c1d1d4fd1dbe13f28dd9e59a5861478f779204f9941d54d12f7ec52d9c"}
Nov 02 15:47:12.682970 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:12.682780    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-28da58-openshift-default-76446c9f79-rg6rm" event={"ID":"e80dea97-6fba-4ad0-b574-da148b1979db","Type":"ContainerStarted","Data":"8e19dc84487f1a25cd147f8dabc05fabcc2fba0d4d8c8d732a1cb6ee607f24f0"}
Nov 02 15:47:12.685016 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:12.684963    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-smtdr" event={"ID":"5086f7a8-e10d-4aca-bc70-950a2f0cd585","Type":"ContainerDied","Data":"a3e7ea72059348bb25b850e69731bc2f278ecda4a7024812ec9749740a4cb7ce"}
Nov 02 15:47:12.685193 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:12.685172    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-smtdr"
Nov 02 15:47:12.686617 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:12.686591    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-2wwx7" event={"ID":"e11f9d23-10db-4bad-a457-7c5db3056979","Type":"ContainerDied","Data":"db6e887cf31577dc937784515fa0fd0f8d8be227246059d5a9e25b7e41acc35a"}
Nov 02 15:47:12.686943 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:12.686627    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-2wwx7"
Nov 02 15:47:12.696378 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:12.696359    5304 scope.go:122] "RemoveContainer" containerID="d169223ffff4b48d9ac2bb350d36d9b24a10a9228d6f22abad89d8b26d735c56"
Nov 02 15:47:12.707649 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:12.707624    5304 scope.go:122] "RemoveContainer" containerID="b49da12e9c1eaf625d860f22d16960d3837c0f77d80f408d1da9d5f6c30a3835"
Nov 02 15:47:12.737595 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:12.737562    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-smtdr"]
Nov 02 15:47:12.740862 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:12.740838    5304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-smtdr"]
Nov 02 15:47:12.776726 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:12.776624    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-lvrfc"]
Nov 02 15:47:12.782734 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:12.782704    5304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-lvrfc"]
Nov 02 15:47:12.825877 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:12.825814    5304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-28da58-openshift-default-76446c9f79-rg6rm" podStartSLOduration=1.825797881 podStartE2EDuration="1.825797881s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:47:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:47:12.821165887 +0000 UTC m=+3032.814990299" watchObservedRunningTime="2026-11-02 15:47:12.825797881 +0000 UTC m=+3032.819622289"
Nov 02 15:47:12.898512 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:12.898469    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-2wwx7"]
Nov 02 15:47:12.915172 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:12.914973    5304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-2wwx7"]
Nov 02 15:47:13.182723 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:13.182688    5304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-28da58-openshift-default-76446c9f79-rg6rm"
Nov 02 15:47:13.184545 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:13.184517    5304 patch_prober.go:29] interesting pod/e2e-shared-b-w3-28da58-openshift-default-76446c9f79-rg6rm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.206:15021/healthz/ready\": dial tcp 10.133.0.206:15021: connect: connection refused" start-of-body=
Nov 02 15:47:13.184676 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:13.184569    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-28da58-openshift-default-76446c9f79-rg6rm" podUID="e80dea97-6fba-4ad0-b574-da148b1979db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.206:15021/healthz/ready\": dial tcp 10.133.0.206:15021: connect: connection refused"
Nov 02 15:47:14.119727 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:14.119687    5304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5086f7a8-e10d-4aca-bc70-950a2f0cd585" path="/var/lib/kubelet/pods/5086f7a8-e10d-4aca-bc70-950a2f0cd585/volumes"
Nov 02 15:47:14.120410 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:14.120387    5304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="93e559b7-2a4c-4533-90d5-5f3fd3956bf0" path="/var/lib/kubelet/pods/93e559b7-2a4c-4533-90d5-5f3fd3956bf0/volumes"
Nov 02 15:47:14.120935 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:14.120913    5304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e11f9d23-10db-4bad-a457-7c5db3056979" path="/var/lib/kubelet/pods/e11f9d23-10db-4bad-a457-7c5db3056979/volumes"
Nov 02 15:47:14.183093 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:14.183063    5304 patch_prober.go:29] interesting pod/e2e-shared-b-w3-28da58-openshift-default-76446c9f79-rg6rm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.206:15021/healthz/ready\": dial tcp 10.133.0.206:15021: connect: connection refused" start-of-body=
Nov 02 15:47:14.183266 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:14.183113    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-28da58-openshift-default-76446c9f79-rg6rm" podUID="e80dea97-6fba-4ad0-b574-da148b1979db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.206:15021/healthz/ready\": dial tcp 10.133.0.206:15021: connect: connection refused"
Nov 02 15:47:15.183456 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:15.183419    5304 patch_prober.go:29] interesting pod/e2e-shared-b-w3-28da58-openshift-default-76446c9f79-rg6rm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.206:15021/healthz/ready\": dial tcp 10.133.0.206:15021: connect: connection refused" start-of-body=
Nov 02 15:47:15.183858 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:15.183474    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-28da58-openshift-default-76446c9f79-rg6rm" podUID="e80dea97-6fba-4ad0-b574-da148b1979db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.206:15021/healthz/ready\": dial tcp 10.133.0.206:15021: connect: connection refused"
Nov 02 15:47:16.183273 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:16.183229    5304 patch_prober.go:29] interesting pod/e2e-shared-b-w3-28da58-openshift-default-76446c9f79-rg6rm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.206:15021/healthz/ready\": dial tcp 10.133.0.206:15021: connect: connection refused" start-of-body=
Nov 02 15:47:16.183437 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:16.183290    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-28da58-openshift-default-76446c9f79-rg6rm" podUID="e80dea97-6fba-4ad0-b574-da148b1979db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.206:15021/healthz/ready\": dial tcp 10.133.0.206:15021: connect: connection refused"
Nov 02 15:47:17.186136 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:17.186084    5304 patch_prober.go:29] interesting pod/e2e-shared-b-w3-28da58-openshift-default-76446c9f79-rg6rm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.206:15021/healthz/ready\": dial tcp 10.133.0.206:15021: connect: connection refused" start-of-body=
Nov 02 15:47:17.186730 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:17.186698    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-28da58-openshift-default-76446c9f79-rg6rm" podUID="e80dea97-6fba-4ad0-b574-da148b1979db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.206:15021/healthz/ready\": dial tcp 10.133.0.206:15021: connect: connection refused"
Nov 02 15:47:17.643912 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:17.643868    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-b955c5-578fdd955-qxkzl"]
Nov 02 15:47:18.183241 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:18.183205    5304 patch_prober.go:29] interesting pod/e2e-shared-b-w3-28da58-openshift-default-76446c9f79-rg6rm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.206:15021/healthz/ready\": dial tcp 10.133.0.206:15021: connect: connection refused" start-of-body=
Nov 02 15:47:18.183452 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:18.183281    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-28da58-openshift-default-76446c9f79-rg6rm" podUID="e80dea97-6fba-4ad0-b574-da148b1979db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.206:15021/healthz/ready\": dial tcp 10.133.0.206:15021: connect: connection refused"
Nov 02 15:47:19.182601 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:19.182564    5304 patch_prober.go:29] interesting pod/e2e-shared-b-w3-28da58-openshift-default-76446c9f79-rg6rm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.206:15021/healthz/ready\": dial tcp 10.133.0.206:15021: connect: connection refused" start-of-body=
Nov 02 15:47:19.183174 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:19.182616    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-28da58-openshift-default-76446c9f79-rg6rm" podUID="e80dea97-6fba-4ad0-b574-da148b1979db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.206:15021/healthz/ready\": dial tcp 10.133.0.206:15021: connect: connection refused"
Nov 02 15:47:20.187268 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:20.187239    5304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-28da58-openshift-default-76446c9f79-rg6rm"
Nov 02 15:47:20.187720 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:20.187698    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-28da58-openshift-default-76446c9f79-rg6rm"
Nov 02 15:47:20.188429 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:20.188408    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-28da58-openshift-default-76446c9f79-rg6rm"
Nov 02 15:47:22.645732 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:22.645691    5304 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-b955c5-578fdd955-qxkzl" podUID="52064ae5-d62f-42f6-a50e-5030d6382aa3" containerName="payload-pre-processing" containerID="cri-o://80666ffdeeb5ff716fcebe7423c23bcbdf7346f148a5e923df725361c0c89214" gracePeriod=25
Nov 02 15:47:22.881032 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:22.880993    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-7ecfe1/distinct-w4-7ecfe1-kserve-6c847c647d-d2jvv"]
Nov 02 15:47:24.798507 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:47:24.798471    5304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k6fjk" containerName="extractor"
Nov 02 15:47:28.731079 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:28.731033    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-b955c5-578fdd955-qxkzl" podUID="52064ae5-d62f-42f6-a50e-5030d6382aa3" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 15:47:32.444357 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:32.444317    5304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-a55a55-59db66d795-crnzc"]
Nov 02 15:47:32.446033 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:32.445998    5304 memory_manager.go:365] "RemoveStaleState removing state" podUID="e11f9d23-10db-4bad-a457-7c5db3056979" containerName="istio-proxy"
Nov 02 15:47:32.469183 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:32.451901    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a55a55-59db66d795-crnzc"
Nov 02 15:47:32.484400 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:32.484368    5304 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-a55a55\""
Nov 02 15:47:32.484630 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:32.484522    5304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-a55a55-dockercfg-mhkwj\""
Nov 02 15:47:32.499156 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:32.497616    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ns8sq\" (UniqueName: \"kubernetes.io/projected/d77e8cf8-6cdc-4458-8c0c-564144f06220-kube-api-access-ns8sq\") pod \"payload-pre-processing-e2e-ipp-a-a55a55-59db66d795-crnzc\" (UID: \"d77e8cf8-6cdc-4458-8c0c-564144f06220\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a55a55-59db66d795-crnzc"
Nov 02 15:47:32.499156 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:32.497875    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d77e8cf8-6cdc-4458-8c0c-564144f06220-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-a55a55-59db66d795-crnzc\" (UID: \"d77e8cf8-6cdc-4458-8c0c-564144f06220\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a55a55-59db66d795-crnzc"
Nov 02 15:47:32.608956 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:32.608920    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-a55a55-59db66d795-crnzc"]
Nov 02 15:47:32.716764 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:32.716679    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d77e8cf8-6cdc-4458-8c0c-564144f06220-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-a55a55-59db66d795-crnzc\" (UID: \"d77e8cf8-6cdc-4458-8c0c-564144f06220\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a55a55-59db66d795-crnzc"
Nov 02 15:47:32.717930 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:32.717898    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d77e8cf8-6cdc-4458-8c0c-564144f06220-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-a55a55-59db66d795-crnzc\" (UID: \"d77e8cf8-6cdc-4458-8c0c-564144f06220\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a55a55-59db66d795-crnzc"
Nov 02 15:47:32.718141 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:32.718102    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ns8sq\" (UniqueName: \"kubernetes.io/projected/d77e8cf8-6cdc-4458-8c0c-564144f06220-kube-api-access-ns8sq\") pod \"payload-pre-processing-e2e-ipp-a-a55a55-59db66d795-crnzc\" (UID: \"d77e8cf8-6cdc-4458-8c0c-564144f06220\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a55a55-59db66d795-crnzc"
Nov 02 15:47:32.727587 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:32.727551    5304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-a55a55-59b6968f7d-c496x"]
Nov 02 15:47:32.797377 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:32.797340    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-a55a55-59b6968f7d-c496x"]
Nov 02 15:47:32.797571 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:32.797489    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-a55a55-59b6968f7d-c496x"
Nov 02 15:47:32.807166 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:32.806921    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ns8sq\" (UniqueName: \"kubernetes.io/projected/d77e8cf8-6cdc-4458-8c0c-564144f06220-kube-api-access-ns8sq\") pod \"payload-pre-processing-e2e-ipp-a-a55a55-59db66d795-crnzc\" (UID: \"d77e8cf8-6cdc-4458-8c0c-564144f06220\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a55a55-59db66d795-crnzc"
Nov 02 15:47:32.924077 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:32.924042    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9sbq5\" (UniqueName: \"kubernetes.io/projected/eb8e624a-b173-4fc3-b5b5-4183ebf2eda7-kube-api-access-9sbq5\") pod \"payload-processing-e2e-ipp-a-a55a55-59b6968f7d-c496x\" (UID: \"eb8e624a-b173-4fc3-b5b5-4183ebf2eda7\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-a55a55-59b6968f7d-c496x"
Nov 02 15:47:32.924394 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:32.924378    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb8e624a-b173-4fc3-b5b5-4183ebf2eda7-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-a55a55-59b6968f7d-c496x\" (UID: \"eb8e624a-b173-4fc3-b5b5-4183ebf2eda7\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-a55a55-59b6968f7d-c496x"
Nov 02 15:47:33.025774 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:33.025681    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9sbq5\" (UniqueName: \"kubernetes.io/projected/eb8e624a-b173-4fc3-b5b5-4183ebf2eda7-kube-api-access-9sbq5\") pod \"payload-processing-e2e-ipp-a-a55a55-59b6968f7d-c496x\" (UID: \"eb8e624a-b173-4fc3-b5b5-4183ebf2eda7\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-a55a55-59b6968f7d-c496x"
Nov 02 15:47:33.025966 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:33.025810    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb8e624a-b173-4fc3-b5b5-4183ebf2eda7-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-a55a55-59b6968f7d-c496x\" (UID: \"eb8e624a-b173-4fc3-b5b5-4183ebf2eda7\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-a55a55-59b6968f7d-c496x"
Nov 02 15:47:33.026744 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:33.026716    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb8e624a-b173-4fc3-b5b5-4183ebf2eda7-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-a55a55-59b6968f7d-c496x\" (UID: \"eb8e624a-b173-4fc3-b5b5-4183ebf2eda7\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-a55a55-59b6968f7d-c496x"
Nov 02 15:47:33.083543 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:33.082733    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9sbq5\" (UniqueName: \"kubernetes.io/projected/eb8e624a-b173-4fc3-b5b5-4183ebf2eda7-kube-api-access-9sbq5\") pod \"payload-processing-e2e-ipp-a-a55a55-59b6968f7d-c496x\" (UID: \"eb8e624a-b173-4fc3-b5b5-4183ebf2eda7\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-a55a55-59b6968f7d-c496x"
Nov 02 15:47:33.083543 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:33.083216    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a55a55-59db66d795-crnzc"
Nov 02 15:47:33.127078 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:33.127044    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-a55a55-59b6968f7d-c496x"
Nov 02 15:47:33.677154 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:33.654069    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-b955c5-xrc9j"]
Nov 02 15:47:33.677154 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:33.654176    5304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-b955c5-xrc9j"]
Nov 02 15:47:33.826256 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:33.826076    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-a55a55-59db66d795-crnzc"]
Nov 02 15:47:33.986241 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:33.983621    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a55a55-59db66d795-crnzc" event={"ID":"d77e8cf8-6cdc-4458-8c0c-564144f06220","Type":"ContainerStarted","Data":"fb9d61a2520bbd33c35133df75aac14bb5ee36a76373c459bed53c91699287e0"}
Nov 02 15:47:34.122333 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:34.121439    5304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4116a3df-a6d5-44f2-a54e-e6bb71c3d3ef" path="/var/lib/kubelet/pods/4116a3df-a6d5-44f2-a54e-e6bb71c3d3ef/volumes"
Nov 02 15:47:34.189412 ip-10-0-135-138 kubenswrapper[5304]: W1102 15:47:34.188339    5304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb8e624a_b173_4fc3_b5b5_4183ebf2eda7.slice/crio-29efbd66a8785f9eb305b0d086705dd6897d3c64696d1112c6638d437af4fc52 WatchSource:0}: Error finding container 29efbd66a8785f9eb305b0d086705dd6897d3c64696d1112c6638d437af4fc52: Status 404 returned error can't find the container with id 29efbd66a8785f9eb305b0d086705dd6897d3c64696d1112c6638d437af4fc52
Nov 02 15:47:34.189806 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:34.189784    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-a55a55-59b6968f7d-c496x"]
Nov 02 15:47:34.999989 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:34.999950    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a55a55-59db66d795-crnzc" event={"ID":"d77e8cf8-6cdc-4458-8c0c-564144f06220","Type":"ContainerStarted","Data":"bcd04148bb8837a6d6fb432a8fcc29856dabbddb303cf6ae68f421f1e351de43"}
Nov 02 15:47:35.000489 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:35.000237    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a55a55-59db66d795-crnzc"
Nov 02 15:47:35.002831 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:35.002332    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-a55a55-59b6968f7d-c496x" event={"ID":"eb8e624a-b173-4fc3-b5b5-4183ebf2eda7","Type":"ContainerStarted","Data":"faba378e91ad9c15eb6272580b5d3f1169f5438d9d07d54300bb8c43aecc74a7"}
Nov 02 15:47:35.002831 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:35.002382    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-a55a55-59b6968f7d-c496x" event={"ID":"eb8e624a-b173-4fc3-b5b5-4183ebf2eda7","Type":"ContainerStarted","Data":"29efbd66a8785f9eb305b0d086705dd6897d3c64696d1112c6638d437af4fc52"}
Nov 02 15:47:35.002831 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:35.002407    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-a55a55-59b6968f7d-c496x"
Nov 02 15:47:35.082188 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:35.066741    5304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a55a55-59db66d795-crnzc" podStartSLOduration=3.066719087 podStartE2EDuration="3.066719087s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:47:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:47:35.048788412 +0000 UTC m=+3055.042612838" watchObservedRunningTime="2026-11-02 15:47:35.066719087 +0000 UTC m=+3055.060543497"
Nov 02 15:47:35.105373 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:35.105315    5304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-a55a55-59b6968f7d-c496x" podStartSLOduration=3.105290368 podStartE2EDuration="3.105290368s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:47:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:47:35.103590503 +0000 UTC m=+3055.097414927" watchObservedRunningTime="2026-11-02 15:47:35.105290368 +0000 UTC m=+3055.099114776"
Nov 02 15:47:37.902500 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:37.902457    5304 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-7ecfe1/distinct-w4-7ecfe1-kserve-6c847c647d-d2jvv" podUID="f1f1ca5e-cca7-4c54-939f-06f095defcdd" containerName="main" containerID="cri-o://6c2db364224fb63629fd94c848db97a5b531496d14e8d28e482fe1cc74c4089b" gracePeriod=45
Nov 02 15:47:38.037174 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:38.037093    5304 generic.go:356] "Generic (PLEG): container finished" podID="f1f1ca5e-cca7-4c54-939f-06f095defcdd" containerID="6c2db364224fb63629fd94c848db97a5b531496d14e8d28e482fe1cc74c4089b" exitCode=0
Nov 02 15:47:38.037174 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:38.037161    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-7ecfe1/distinct-w4-7ecfe1-kserve-6c847c647d-d2jvv" event={"ID":"f1f1ca5e-cca7-4c54-939f-06f095defcdd","Type":"ContainerDied","Data":"6c2db364224fb63629fd94c848db97a5b531496d14e8d28e482fe1cc74c4089b"}
Nov 02 15:47:38.268360 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:38.268328    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-7ecfe1/distinct-w4-7ecfe1-kserve-6c847c647d-d2jvv"
Nov 02 15:47:38.388371 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:38.388332    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1f1ca5e-cca7-4c54-939f-06f095defcdd-kube-api-access-jrdpj\" (UniqueName: \"kubernetes.io/projected/f1f1ca5e-cca7-4c54-939f-06f095defcdd-kube-api-access-jrdpj\") pod \"f1f1ca5e-cca7-4c54-939f-06f095defcdd\" (UID: \"f1f1ca5e-cca7-4c54-939f-06f095defcdd\") "
Nov 02 15:47:38.388650 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:38.388387    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1f1ca5e-cca7-4c54-939f-06f095defcdd-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f1f1ca5e-cca7-4c54-939f-06f095defcdd-model-cache\") pod \"f1f1ca5e-cca7-4c54-939f-06f095defcdd\" (UID: \"f1f1ca5e-cca7-4c54-939f-06f095defcdd\") "
Nov 02 15:47:38.388650 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:38.388472    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1f1ca5e-cca7-4c54-939f-06f095defcdd-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f1f1ca5e-cca7-4c54-939f-06f095defcdd-dshm\") pod \"f1f1ca5e-cca7-4c54-939f-06f095defcdd\" (UID: \"f1f1ca5e-cca7-4c54-939f-06f095defcdd\") "
Nov 02 15:47:38.388650 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:38.388506    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1f1ca5e-cca7-4c54-939f-06f095defcdd-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f1f1ca5e-cca7-4c54-939f-06f095defcdd-tmp-dir\") pod \"f1f1ca5e-cca7-4c54-939f-06f095defcdd\" (UID: \"f1f1ca5e-cca7-4c54-939f-06f095defcdd\") "
Nov 02 15:47:38.388650 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:38.388562    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1f1ca5e-cca7-4c54-939f-06f095defcdd-home\" (UniqueName: \"kubernetes.io/empty-dir/f1f1ca5e-cca7-4c54-939f-06f095defcdd-home\") pod \"f1f1ca5e-cca7-4c54-939f-06f095defcdd\" (UID: \"f1f1ca5e-cca7-4c54-939f-06f095defcdd\") "
Nov 02 15:47:38.388650 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:38.388647    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f1f1ca5e-cca7-4c54-939f-06f095defcdd-tls-certs\" (UniqueName: \"kubernetes.io/secret/f1f1ca5e-cca7-4c54-939f-06f095defcdd-tls-certs\") pod \"f1f1ca5e-cca7-4c54-939f-06f095defcdd\" (UID: \"f1f1ca5e-cca7-4c54-939f-06f095defcdd\") "
Nov 02 15:47:38.388944 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:38.388735    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1f1ca5e-cca7-4c54-939f-06f095defcdd-model-cache" pod "f1f1ca5e-cca7-4c54-939f-06f095defcdd" (UID: "f1f1ca5e-cca7-4c54-939f-06f095defcdd"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:47:38.388944 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:38.388840    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1f1ca5e-cca7-4c54-939f-06f095defcdd-tmp-dir" pod "f1f1ca5e-cca7-4c54-939f-06f095defcdd" (UID: "f1f1ca5e-cca7-4c54-939f-06f095defcdd"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:47:38.388944 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:38.388890    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1f1ca5e-cca7-4c54-939f-06f095defcdd-home" pod "f1f1ca5e-cca7-4c54-939f-06f095defcdd" (UID: "f1f1ca5e-cca7-4c54-939f-06f095defcdd"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:47:38.389329 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:38.389279    5304 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f1f1ca5e-cca7-4c54-939f-06f095defcdd-model-cache\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:47:38.389329 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:38.389307    5304 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f1f1ca5e-cca7-4c54-939f-06f095defcdd-tmp-dir\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:47:38.389329 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:38.389319    5304 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f1f1ca5e-cca7-4c54-939f-06f095defcdd-home\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:47:38.391036 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:38.391004    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f1f1ca5e-cca7-4c54-939f-06f095defcdd-tls-certs" pod "f1f1ca5e-cca7-4c54-939f-06f095defcdd" (UID: "f1f1ca5e-cca7-4c54-939f-06f095defcdd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:47:38.391244 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:38.391223    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1f1ca5e-cca7-4c54-939f-06f095defcdd-kube-api-access-jrdpj" pod "f1f1ca5e-cca7-4c54-939f-06f095defcdd" (UID: "f1f1ca5e-cca7-4c54-939f-06f095defcdd"). InnerVolumeSpecName "kube-api-access-jrdpj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:47:38.391326 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:38.391256    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1f1ca5e-cca7-4c54-939f-06f095defcdd-dshm" pod "f1f1ca5e-cca7-4c54-939f-06f095defcdd" (UID: "f1f1ca5e-cca7-4c54-939f-06f095defcdd"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:47:38.496210 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:38.495667    5304 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f1f1ca5e-cca7-4c54-939f-06f095defcdd-tls-certs\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:47:38.496210 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:38.495708    5304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jrdpj\" (UniqueName: \"kubernetes.io/projected/f1f1ca5e-cca7-4c54-939f-06f095defcdd-kube-api-access-jrdpj\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:47:38.496210 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:38.495726    5304 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f1f1ca5e-cca7-4c54-939f-06f095defcdd-dshm\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:47:38.769350 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:38.751474    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-b955c5-578fdd955-qxkzl" podUID="52064ae5-d62f-42f6-a50e-5030d6382aa3" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 15:47:39.046049 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:39.045956    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-7ecfe1/distinct-w4-7ecfe1-kserve-6c847c647d-d2jvv"
Nov 02 15:47:39.046049 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:39.045978    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-7ecfe1/distinct-w4-7ecfe1-kserve-6c847c647d-d2jvv" event={"ID":"f1f1ca5e-cca7-4c54-939f-06f095defcdd","Type":"ContainerDied","Data":"65bcb65d729e2e3cf76abff90ed7d7b84bd240495edaed7f8ddfad0539e0d27f"}
Nov 02 15:47:39.046049 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:39.046035    5304 scope.go:122] "RemoveContainer" containerID="6c2db364224fb63629fd94c848db97a5b531496d14e8d28e482fe1cc74c4089b"
Nov 02 15:47:39.103782 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:39.103741    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-7ecfe1/distinct-w4-7ecfe1-kserve-6c847c647d-d2jvv"]
Nov 02 15:47:39.118296 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:39.118258    5304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-7ecfe1/distinct-w4-7ecfe1-kserve-6c847c647d-d2jvv"]
Nov 02 15:47:39.524349 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:39.524299    5304 patch_prober.go:29] interesting pod/e2e-ipp-b-24255b-openshift-default-6687484f49-992gq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.195:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 15:47:39.524587 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:39.524373    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-24255b-openshift-default-6687484f49-992gq" podUID="c19acd3a-5dbf-4da3-b88f-e36284b22243" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.195:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:47:39.682975 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:39.682939    5304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-e40940-openshift-default-6fb44b465f-75t4b"]
Nov 02 15:47:39.683658 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:39.683632    5304 cpu_manager.go:414] "containerMap: removing container" podUID="f1f1ca5e-cca7-4c54-939f-06f095defcdd" containerName="main"
Nov 02 15:47:39.683658 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:39.683653    5304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1f1ca5e-cca7-4c54-939f-06f095defcdd" containerName="main"
Nov 02 15:47:39.683830 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:39.683802    5304 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1f1ca5e-cca7-4c54-939f-06f095defcdd" containerName="main"
Nov 02 15:47:39.768230 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:39.765982    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-e40940-openshift-default-6fb44b465f-75t4b"]
Nov 02 15:47:39.768230 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:39.766103    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-e40940-openshift-default-6fb44b465f-75t4b"
Nov 02 15:47:39.782345 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:39.782316    5304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-e40940-openshift-default-dockercfg-48t4g\""
Nov 02 15:47:39.924487 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:39.924435    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7ab8a29-4410-4fde-8975-49c13b3ff1fb-istio-envoy\") pod \"e2e-ipp-a-e40940-openshift-default-6fb44b465f-75t4b\" (UID: \"d7ab8a29-4410-4fde-8975-49c13b3ff1fb\") " pod="openshift-ingress/e2e-ipp-a-e40940-openshift-default-6fb44b465f-75t4b"
Nov 02 15:47:39.924698 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:39.924519    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7ab8a29-4410-4fde-8975-49c13b3ff1fb-workload-socket\") pod \"e2e-ipp-a-e40940-openshift-default-6fb44b465f-75t4b\" (UID: \"d7ab8a29-4410-4fde-8975-49c13b3ff1fb\") " pod="openshift-ingress/e2e-ipp-a-e40940-openshift-default-6fb44b465f-75t4b"
Nov 02 15:47:39.924698 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:39.924593    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qgb8c\" (UniqueName: \"kubernetes.io/projected/d7ab8a29-4410-4fde-8975-49c13b3ff1fb-kube-api-access-qgb8c\") pod \"e2e-ipp-a-e40940-openshift-default-6fb44b465f-75t4b\" (UID: \"d7ab8a29-4410-4fde-8975-49c13b3ff1fb\") " pod="openshift-ingress/e2e-ipp-a-e40940-openshift-default-6fb44b465f-75t4b"
Nov 02 15:47:39.924698 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:39.924628    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7ab8a29-4410-4fde-8975-49c13b3ff1fb-istio-token\") pod \"e2e-ipp-a-e40940-openshift-default-6fb44b465f-75t4b\" (UID: \"d7ab8a29-4410-4fde-8975-49c13b3ff1fb\") " pod="openshift-ingress/e2e-ipp-a-e40940-openshift-default-6fb44b465f-75t4b"
Nov 02 15:47:39.924698 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:39.924662    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7ab8a29-4410-4fde-8975-49c13b3ff1fb-istiod-ca-cert\") pod \"e2e-ipp-a-e40940-openshift-default-6fb44b465f-75t4b\" (UID: \"d7ab8a29-4410-4fde-8975-49c13b3ff1fb\") " pod="openshift-ingress/e2e-ipp-a-e40940-openshift-default-6fb44b465f-75t4b"
Nov 02 15:47:39.924936 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:39.924736    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7ab8a29-4410-4fde-8975-49c13b3ff1fb-credential-socket\") pod \"e2e-ipp-a-e40940-openshift-default-6fb44b465f-75t4b\" (UID: \"d7ab8a29-4410-4fde-8975-49c13b3ff1fb\") " pod="openshift-ingress/e2e-ipp-a-e40940-openshift-default-6fb44b465f-75t4b"
Nov 02 15:47:39.924936 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:39.924758    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7ab8a29-4410-4fde-8975-49c13b3ff1fb-workload-certs\") pod \"e2e-ipp-a-e40940-openshift-default-6fb44b465f-75t4b\" (UID: \"d7ab8a29-4410-4fde-8975-49c13b3ff1fb\") " pod="openshift-ingress/e2e-ipp-a-e40940-openshift-default-6fb44b465f-75t4b"
Nov 02 15:47:39.924936 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:39.924822    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7ab8a29-4410-4fde-8975-49c13b3ff1fb-istio-podinfo\") pod \"e2e-ipp-a-e40940-openshift-default-6fb44b465f-75t4b\" (UID: \"d7ab8a29-4410-4fde-8975-49c13b3ff1fb\") " pod="openshift-ingress/e2e-ipp-a-e40940-openshift-default-6fb44b465f-75t4b"
Nov 02 15:47:39.924936 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:39.924887    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7ab8a29-4410-4fde-8975-49c13b3ff1fb-istio-data\") pod \"e2e-ipp-a-e40940-openshift-default-6fb44b465f-75t4b\" (UID: \"d7ab8a29-4410-4fde-8975-49c13b3ff1fb\") " pod="openshift-ingress/e2e-ipp-a-e40940-openshift-default-6fb44b465f-75t4b"
Nov 02 15:47:40.025749 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:40.025710    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7ab8a29-4410-4fde-8975-49c13b3ff1fb-istio-envoy\") pod \"e2e-ipp-a-e40940-openshift-default-6fb44b465f-75t4b\" (UID: \"d7ab8a29-4410-4fde-8975-49c13b3ff1fb\") " pod="openshift-ingress/e2e-ipp-a-e40940-openshift-default-6fb44b465f-75t4b"
Nov 02 15:47:40.025966 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:40.025773    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7ab8a29-4410-4fde-8975-49c13b3ff1fb-workload-socket\") pod \"e2e-ipp-a-e40940-openshift-default-6fb44b465f-75t4b\" (UID: \"d7ab8a29-4410-4fde-8975-49c13b3ff1fb\") " pod="openshift-ingress/e2e-ipp-a-e40940-openshift-default-6fb44b465f-75t4b"
Nov 02 15:47:40.025966 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:40.025843    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qgb8c\" (UniqueName: \"kubernetes.io/projected/d7ab8a29-4410-4fde-8975-49c13b3ff1fb-kube-api-access-qgb8c\") pod \"e2e-ipp-a-e40940-openshift-default-6fb44b465f-75t4b\" (UID: \"d7ab8a29-4410-4fde-8975-49c13b3ff1fb\") " pod="openshift-ingress/e2e-ipp-a-e40940-openshift-default-6fb44b465f-75t4b"
Nov 02 15:47:40.025966 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:40.025869    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7ab8a29-4410-4fde-8975-49c13b3ff1fb-istio-token\") pod \"e2e-ipp-a-e40940-openshift-default-6fb44b465f-75t4b\" (UID: \"d7ab8a29-4410-4fde-8975-49c13b3ff1fb\") " pod="openshift-ingress/e2e-ipp-a-e40940-openshift-default-6fb44b465f-75t4b"
Nov 02 15:47:40.025966 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:40.025893    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7ab8a29-4410-4fde-8975-49c13b3ff1fb-istiod-ca-cert\") pod \"e2e-ipp-a-e40940-openshift-default-6fb44b465f-75t4b\" (UID: \"d7ab8a29-4410-4fde-8975-49c13b3ff1fb\") " pod="openshift-ingress/e2e-ipp-a-e40940-openshift-default-6fb44b465f-75t4b"
Nov 02 15:47:40.025966 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:40.025959    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7ab8a29-4410-4fde-8975-49c13b3ff1fb-credential-socket\") pod \"e2e-ipp-a-e40940-openshift-default-6fb44b465f-75t4b\" (UID: \"d7ab8a29-4410-4fde-8975-49c13b3ff1fb\") " pod="openshift-ingress/e2e-ipp-a-e40940-openshift-default-6fb44b465f-75t4b"
Nov 02 15:47:40.026293 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:40.025986    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7ab8a29-4410-4fde-8975-49c13b3ff1fb-workload-certs\") pod \"e2e-ipp-a-e40940-openshift-default-6fb44b465f-75t4b\" (UID: \"d7ab8a29-4410-4fde-8975-49c13b3ff1fb\") " pod="openshift-ingress/e2e-ipp-a-e40940-openshift-default-6fb44b465f-75t4b"
Nov 02 15:47:40.026293 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:40.026042    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7ab8a29-4410-4fde-8975-49c13b3ff1fb-istio-podinfo\") pod \"e2e-ipp-a-e40940-openshift-default-6fb44b465f-75t4b\" (UID: \"d7ab8a29-4410-4fde-8975-49c13b3ff1fb\") " pod="openshift-ingress/e2e-ipp-a-e40940-openshift-default-6fb44b465f-75t4b"
Nov 02 15:47:40.026293 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:40.026107    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7ab8a29-4410-4fde-8975-49c13b3ff1fb-istio-data\") pod \"e2e-ipp-a-e40940-openshift-default-6fb44b465f-75t4b\" (UID: \"d7ab8a29-4410-4fde-8975-49c13b3ff1fb\") " pod="openshift-ingress/e2e-ipp-a-e40940-openshift-default-6fb44b465f-75t4b"
Nov 02 15:47:40.026293 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:40.026268    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7ab8a29-4410-4fde-8975-49c13b3ff1fb-workload-socket\") pod \"e2e-ipp-a-e40940-openshift-default-6fb44b465f-75t4b\" (UID: \"d7ab8a29-4410-4fde-8975-49c13b3ff1fb\") " pod="openshift-ingress/e2e-ipp-a-e40940-openshift-default-6fb44b465f-75t4b"
Nov 02 15:47:40.026516 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:40.026410    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7ab8a29-4410-4fde-8975-49c13b3ff1fb-credential-socket\") pod \"e2e-ipp-a-e40940-openshift-default-6fb44b465f-75t4b\" (UID: \"d7ab8a29-4410-4fde-8975-49c13b3ff1fb\") " pod="openshift-ingress/e2e-ipp-a-e40940-openshift-default-6fb44b465f-75t4b"
Nov 02 15:47:40.026584 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:40.026530    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7ab8a29-4410-4fde-8975-49c13b3ff1fb-workload-certs\") pod \"e2e-ipp-a-e40940-openshift-default-6fb44b465f-75t4b\" (UID: \"d7ab8a29-4410-4fde-8975-49c13b3ff1fb\") " pod="openshift-ingress/e2e-ipp-a-e40940-openshift-default-6fb44b465f-75t4b"
Nov 02 15:47:40.026584 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:40.026557    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7ab8a29-4410-4fde-8975-49c13b3ff1fb-istio-data\") pod \"e2e-ipp-a-e40940-openshift-default-6fb44b465f-75t4b\" (UID: \"d7ab8a29-4410-4fde-8975-49c13b3ff1fb\") " pod="openshift-ingress/e2e-ipp-a-e40940-openshift-default-6fb44b465f-75t4b"
Nov 02 15:47:40.027087 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:40.027066    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7ab8a29-4410-4fde-8975-49c13b3ff1fb-istiod-ca-cert\") pod \"e2e-ipp-a-e40940-openshift-default-6fb44b465f-75t4b\" (UID: \"d7ab8a29-4410-4fde-8975-49c13b3ff1fb\") " pod="openshift-ingress/e2e-ipp-a-e40940-openshift-default-6fb44b465f-75t4b"
Nov 02 15:47:40.028855 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:40.028833    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7ab8a29-4410-4fde-8975-49c13b3ff1fb-istio-envoy\") pod \"e2e-ipp-a-e40940-openshift-default-6fb44b465f-75t4b\" (UID: \"d7ab8a29-4410-4fde-8975-49c13b3ff1fb\") " pod="openshift-ingress/e2e-ipp-a-e40940-openshift-default-6fb44b465f-75t4b"
Nov 02 15:47:40.028968 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:40.028893    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7ab8a29-4410-4fde-8975-49c13b3ff1fb-istio-podinfo\") pod \"e2e-ipp-a-e40940-openshift-default-6fb44b465f-75t4b\" (UID: \"d7ab8a29-4410-4fde-8975-49c13b3ff1fb\") " pod="openshift-ingress/e2e-ipp-a-e40940-openshift-default-6fb44b465f-75t4b"
Nov 02 15:47:40.050366 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:40.050291    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7ab8a29-4410-4fde-8975-49c13b3ff1fb-istio-token\") pod \"e2e-ipp-a-e40940-openshift-default-6fb44b465f-75t4b\" (UID: \"d7ab8a29-4410-4fde-8975-49c13b3ff1fb\") " pod="openshift-ingress/e2e-ipp-a-e40940-openshift-default-6fb44b465f-75t4b"
Nov 02 15:47:40.050724 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:40.050669    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qgb8c\" (UniqueName: \"kubernetes.io/projected/d7ab8a29-4410-4fde-8975-49c13b3ff1fb-kube-api-access-qgb8c\") pod \"e2e-ipp-a-e40940-openshift-default-6fb44b465f-75t4b\" (UID: \"d7ab8a29-4410-4fde-8975-49c13b3ff1fb\") " pod="openshift-ingress/e2e-ipp-a-e40940-openshift-default-6fb44b465f-75t4b"
Nov 02 15:47:40.121831 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:40.121792    5304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f1f1ca5e-cca7-4c54-939f-06f095defcdd" path="/var/lib/kubelet/pods/f1f1ca5e-cca7-4c54-939f-06f095defcdd/volumes"
Nov 02 15:47:40.160343 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:40.160302    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-e40940-openshift-default-6fb44b465f-75t4b"
Nov 02 15:47:40.589606 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:40.589582    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-e40940-openshift-default-6fb44b465f-75t4b"]
Nov 02 15:47:40.592771 ip-10-0-135-138 kubenswrapper[5304]: W1102 15:47:40.592740    5304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7ab8a29_4410_4fde_8975_49c13b3ff1fb.slice/crio-589f75da3f367475110a0d32be806ffd801606686d5381afab1592bdfe09c28d WatchSource:0}: Error finding container 589f75da3f367475110a0d32be806ffd801606686d5381afab1592bdfe09c28d: Status 404 returned error can't find the container with id 589f75da3f367475110a0d32be806ffd801606686d5381afab1592bdfe09c28d
Nov 02 15:47:40.595875 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:40.595833    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:47:40.595989 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:40.595922    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:47:40.595989 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:40.595969    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:47:40.952358 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:40.952268    5304 patch_prober.go:29] interesting pod/e2e-ipp-a-f0039c-openshift-default-5f656fc965-rjf2w container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.196:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 15:47:40.952358 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:40.952341    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-f0039c-openshift-default-5f656fc965-rjf2w" podUID="9178b9f8-6fda-40a9-8850-fa0097f2d39a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.196:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:47:41.069397 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:41.069352    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-e40940-openshift-default-6fb44b465f-75t4b" event={"ID":"d7ab8a29-4410-4fde-8975-49c13b3ff1fb","Type":"ContainerStarted","Data":"c4fc082d0ebbceccdf88601b58447c83a6bc5dd2fe8ba878d8d75c78d9d8729f"}
Nov 02 15:47:41.069932 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:41.069419    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-e40940-openshift-default-6fb44b465f-75t4b" event={"ID":"d7ab8a29-4410-4fde-8975-49c13b3ff1fb","Type":"ContainerStarted","Data":"589f75da3f367475110a0d32be806ffd801606686d5381afab1592bdfe09c28d"}
Nov 02 15:47:41.118241 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:41.118184    5304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-e40940-openshift-default-6fb44b465f-75t4b" podStartSLOduration=2.118169557 podStartE2EDuration="2.118169557s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:47:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:47:41.114275815 +0000 UTC m=+3061.108100224" watchObservedRunningTime="2026-11-02 15:47:41.118169557 +0000 UTC m=+3061.111993966"
Nov 02 15:47:41.160981 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:41.160943    5304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-e40940-openshift-default-6fb44b465f-75t4b"
Nov 02 15:47:41.162342 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:41.162309    5304 patch_prober.go:29] interesting pod/e2e-ipp-a-e40940-openshift-default-6fb44b465f-75t4b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused" start-of-body=
Nov 02 15:47:41.162508 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:41.162362    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-e40940-openshift-default-6fb44b465f-75t4b" podUID="d7ab8a29-4410-4fde-8975-49c13b3ff1fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused"
Nov 02 15:47:41.737055 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:41.737016    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-b955c5-openshift-default-9fd99b497-tpwvk"]
Nov 02 15:47:41.737375 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:41.737304    5304 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w6-b955c5-openshift-default-9fd99b497-tpwvk" podUID="783e5ed2-e421-4d98-87e2-6b57a5662b66" containerName="istio-proxy" containerID="cri-o://bbb27e8a8c3a112958382317700b3ddaaa5ed7a1515d2dc3af164ca7c82eba22" gracePeriod=30
Nov 02 15:47:42.165419 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:42.165386    5304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-e40940-openshift-default-6fb44b465f-75t4b"
Nov 02 15:47:42.980641 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:42.980611    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-b955c5-578fdd955-qxkzl"
Nov 02 15:47:43.085259 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:43.085224    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52064ae5-d62f-42f6-a50e-5030d6382aa3-kube-api-access-6752f\" (UniqueName: \"kubernetes.io/projected/52064ae5-d62f-42f6-a50e-5030d6382aa3-kube-api-access-6752f\") pod \"52064ae5-d62f-42f6-a50e-5030d6382aa3\" (UID: \"52064ae5-d62f-42f6-a50e-5030d6382aa3\") "
Nov 02 15:47:43.085497 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:43.085318    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/52064ae5-d62f-42f6-a50e-5030d6382aa3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52064ae5-d62f-42f6-a50e-5030d6382aa3-plugins-config-volume\") pod \"52064ae5-d62f-42f6-a50e-5030d6382aa3\" (UID: \"52064ae5-d62f-42f6-a50e-5030d6382aa3\") "
Nov 02 15:47:43.085717 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:43.085695    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52064ae5-d62f-42f6-a50e-5030d6382aa3-plugins-config-volume" pod "52064ae5-d62f-42f6-a50e-5030d6382aa3" (UID: "52064ae5-d62f-42f6-a50e-5030d6382aa3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:47:43.085975 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:43.085954    5304 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52064ae5-d62f-42f6-a50e-5030d6382aa3-plugins-config-volume\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:47:43.087300 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:43.087274    5304 generic.go:356] "Generic (PLEG): container finished" podID="52064ae5-d62f-42f6-a50e-5030d6382aa3" containerID="80666ffdeeb5ff716fcebe7423c23bcbdf7346f148a5e923df725361c0c89214" exitCode=0
Nov 02 15:47:43.087431 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:43.087324    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-b955c5-578fdd955-qxkzl"
Nov 02 15:47:43.087431 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:43.087359    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-b955c5-578fdd955-qxkzl" event={"ID":"52064ae5-d62f-42f6-a50e-5030d6382aa3","Type":"ContainerDied","Data":"80666ffdeeb5ff716fcebe7423c23bcbdf7346f148a5e923df725361c0c89214"}
Nov 02 15:47:43.087657 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:43.087436    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-b955c5-578fdd955-qxkzl" event={"ID":"52064ae5-d62f-42f6-a50e-5030d6382aa3","Type":"ContainerDied","Data":"6fe52feaff20589dc9aae37856f47506176bc0e944c1ac704c4dc2b8f7b56bdd"}
Nov 02 15:47:43.087657 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:43.087473    5304 scope.go:122] "RemoveContainer" containerID="80666ffdeeb5ff716fcebe7423c23bcbdf7346f148a5e923df725361c0c89214"
Nov 02 15:47:43.088013 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:43.087898    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52064ae5-d62f-42f6-a50e-5030d6382aa3-kube-api-access-6752f" pod "52064ae5-d62f-42f6-a50e-5030d6382aa3" (UID: "52064ae5-d62f-42f6-a50e-5030d6382aa3"). InnerVolumeSpecName "kube-api-access-6752f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:47:43.089488 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:43.089463    5304 generic.go:356] "Generic (PLEG): container finished" podID="9178b9f8-6fda-40a9-8850-fa0097f2d39a" containerID="7f93fabea8b9718b8713ce2ac008f397f62178433855a93d33fdd815291278ee" exitCode=0
Nov 02 15:47:43.090236 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:43.090209    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f0039c-openshift-default-5f656fc965-rjf2w" event={"ID":"9178b9f8-6fda-40a9-8850-fa0097f2d39a","Type":"ContainerDied","Data":"7f93fabea8b9718b8713ce2ac008f397f62178433855a93d33fdd815291278ee"}
Nov 02 15:47:43.090344 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:43.090275    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-e40940-openshift-default-6fb44b465f-75t4b"
Nov 02 15:47:43.090665 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:43.090643    5304 scope.go:122] "RemoveContainer" containerID="7f93fabea8b9718b8713ce2ac008f397f62178433855a93d33fdd815291278ee"
Nov 02 15:47:43.091657 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:43.091642    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-e40940-openshift-default-6fb44b465f-75t4b"
Nov 02 15:47:43.092568 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:43.092534    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:47:43.092677 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:43.092634    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:47:43.092745 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:43.092687    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:47:43.112569 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:43.112540    5304 scope.go:122] "RemoveContainer" containerID="80666ffdeeb5ff716fcebe7423c23bcbdf7346f148a5e923df725361c0c89214"
Nov 02 15:47:43.112908 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:47:43.112888    5304 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"80666ffdeeb5ff716fcebe7423c23bcbdf7346f148a5e923df725361c0c89214\": container with ID starting with 80666ffdeeb5ff716fcebe7423c23bcbdf7346f148a5e923df725361c0c89214 not found: ID does not exist" containerID="80666ffdeeb5ff716fcebe7423c23bcbdf7346f148a5e923df725361c0c89214"
Nov 02 15:47:43.112963 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:43.112919    5304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"80666ffdeeb5ff716fcebe7423c23bcbdf7346f148a5e923df725361c0c89214"} err="failed to get container status \"80666ffdeeb5ff716fcebe7423c23bcbdf7346f148a5e923df725361c0c89214\": rpc error: code = NotFound desc = could not find container \"80666ffdeeb5ff716fcebe7423c23bcbdf7346f148a5e923df725361c0c89214\": container with ID starting with 80666ffdeeb5ff716fcebe7423c23bcbdf7346f148a5e923df725361c0c89214 not found: ID does not exist"
Nov 02 15:47:43.204375 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:43.201486    5304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6752f\" (UniqueName: \"kubernetes.io/projected/52064ae5-d62f-42f6-a50e-5030d6382aa3-kube-api-access-6752f\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:47:43.439643 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:43.439608    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-b955c5-578fdd955-qxkzl"]
Nov 02 15:47:43.446233 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:43.446197    5304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-b955c5-578fdd955-qxkzl"]
Nov 02 15:47:43.466702 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:43.466668    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f0039c-b78956c9-vm85j"]
Nov 02 15:47:43.505764 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:43.505727    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f0039c-6c5bf99c7f-62jkr"]
Nov 02 15:47:44.099453 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:44.099406    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f0039c-openshift-default-5f656fc965-rjf2w" event={"ID":"9178b9f8-6fda-40a9-8850-fa0097f2d39a","Type":"ContainerStarted","Data":"75004946a6197bf76c248dc08a3e0e00c4632a344e8913e3d4bb7dddfa8ec931"}
Nov 02 15:47:44.121956 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:44.121905    5304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="52064ae5-d62f-42f6-a50e-5030d6382aa3" path="/var/lib/kubelet/pods/52064ae5-d62f-42f6-a50e-5030d6382aa3/volumes"
Nov 02 15:47:44.373213 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:44.372924    5304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-f0039c-openshift-default-5f656fc965-rjf2w"
Nov 02 15:47:44.373213 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:44.373070    5304 patch_prober.go:29] interesting pod/e2e-ipp-a-f0039c-openshift-default-5f656fc965-rjf2w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused" start-of-body=
Nov 02 15:47:44.373213 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:44.373114    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f0039c-openshift-default-5f656fc965-rjf2w" podUID="9178b9f8-6fda-40a9-8850-fa0097f2d39a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused"
Nov 02 15:47:44.857616 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:44.857576    5304 patch_prober.go:29] interesting pod/e2e-shared-a-w6-b955c5-openshift-default-9fd99b497-tpwvk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 15:47:44.857802 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:44.857655    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w6-b955c5-openshift-default-9fd99b497-tpwvk" podUID="783e5ed2-e421-4d98-87e2-6b57a5662b66" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 15:47:45.107216 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:45.107180    5304 generic.go:356] "Generic (PLEG): container finished" podID="c19acd3a-5dbf-4da3-b88f-e36284b22243" containerID="2a334a4d6be2f27cefd44840ccc5c33c95658bbdb551ba895da7970f96112d02" exitCode=0
Nov 02 15:47:45.107368 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:45.107253    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-24255b-openshift-default-6687484f49-992gq" event={"ID":"c19acd3a-5dbf-4da3-b88f-e36284b22243","Type":"ContainerDied","Data":"2a334a4d6be2f27cefd44840ccc5c33c95658bbdb551ba895da7970f96112d02"}
Nov 02 15:47:45.108068 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:45.108014    5304 scope.go:122] "RemoveContainer" containerID="2a334a4d6be2f27cefd44840ccc5c33c95658bbdb551ba895da7970f96112d02"
Nov 02 15:47:45.109616 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:45.109587    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:47:45.109693 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:45.109655    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:47:45.109693 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:45.109686    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:47:45.359637 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:45.359547    5304 patch_prober.go:29] interesting pod/e2e-ipp-a-f0039c-openshift-default-5f656fc965-rjf2w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused" start-of-body=
Nov 02 15:47:45.359850 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:45.359648    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f0039c-openshift-default-5f656fc965-rjf2w" podUID="9178b9f8-6fda-40a9-8850-fa0097f2d39a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused"
Nov 02 15:47:46.014289 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:46.014258    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-a55a55-59b6968f7d-c496x"
Nov 02 15:47:46.014757 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:46.014647    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a55a55-59db66d795-crnzc"
Nov 02 15:47:46.228705 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:46.228637    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-24255b-openshift-default-6687484f49-992gq" event={"ID":"c19acd3a-5dbf-4da3-b88f-e36284b22243","Type":"ContainerStarted","Data":"4128ca1e0694d81d32c0ef9ddbec96c80ea57847126a2ebb01bc306d74548a25"}
Nov 02 15:47:46.370153 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:46.369582    5304 patch_prober.go:29] interesting pod/e2e-ipp-a-f0039c-openshift-default-5f656fc965-rjf2w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused" start-of-body=
Nov 02 15:47:46.370480 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:46.370445    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f0039c-openshift-default-5f656fc965-rjf2w" podUID="9178b9f8-6fda-40a9-8850-fa0097f2d39a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused"
Nov 02 15:47:46.722317 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:46.722244    5304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-24255b-openshift-default-6687484f49-992gq"
Nov 02 15:47:46.722599 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:46.722574    5304 patch_prober.go:29] interesting pod/e2e-ipp-b-24255b-openshift-default-6687484f49-992gq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused" start-of-body=
Nov 02 15:47:46.722676 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:46.722633    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-24255b-openshift-default-6687484f49-992gq" podUID="c19acd3a-5dbf-4da3-b88f-e36284b22243" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused"
Nov 02 15:47:47.204470 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:47.204355    5304 generic.go:356] "Generic (PLEG): container finished" podID="783e5ed2-e421-4d98-87e2-6b57a5662b66" containerID="bbb27e8a8c3a112958382317700b3ddaaa5ed7a1515d2dc3af164ca7c82eba22" exitCode=0
Nov 02 15:47:47.205410 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:47.205383    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-b955c5-openshift-default-9fd99b497-tpwvk" event={"ID":"783e5ed2-e421-4d98-87e2-6b57a5662b66","Type":"ContainerDied","Data":"bbb27e8a8c3a112958382317700b3ddaaa5ed7a1515d2dc3af164ca7c82eba22"}
Nov 02 15:47:47.205517 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:47.205447    5304 scope.go:122] "RemoveContainer" containerID="b76a218e8ab6e917ec5fcb6945b04f34c456fa3c5c5a1ed0b17b71a32a20b8c8"
Nov 02 15:47:47.227818 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:47.227797    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-b955c5-openshift-default-9fd99b497-tpwvk"
Nov 02 15:47:47.320777 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:47.320062    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/783e5ed2-e421-4d98-87e2-6b57a5662b66-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/783e5ed2-e421-4d98-87e2-6b57a5662b66-istio-data\") pod \"783e5ed2-e421-4d98-87e2-6b57a5662b66\" (UID: \"783e5ed2-e421-4d98-87e2-6b57a5662b66\") "
Nov 02 15:47:47.320777 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:47.320144    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/783e5ed2-e421-4d98-87e2-6b57a5662b66-istio-token\" (UniqueName: \"kubernetes.io/projected/783e5ed2-e421-4d98-87e2-6b57a5662b66-istio-token\") pod \"783e5ed2-e421-4d98-87e2-6b57a5662b66\" (UID: \"783e5ed2-e421-4d98-87e2-6b57a5662b66\") "
Nov 02 15:47:47.320777 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:47.320213    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/783e5ed2-e421-4d98-87e2-6b57a5662b66-kube-api-access-5q6wz\" (UniqueName: \"kubernetes.io/projected/783e5ed2-e421-4d98-87e2-6b57a5662b66-kube-api-access-5q6wz\") pod \"783e5ed2-e421-4d98-87e2-6b57a5662b66\" (UID: \"783e5ed2-e421-4d98-87e2-6b57a5662b66\") "
Nov 02 15:47:47.320777 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:47.320264    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/783e5ed2-e421-4d98-87e2-6b57a5662b66-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/783e5ed2-e421-4d98-87e2-6b57a5662b66-workload-socket\") pod \"783e5ed2-e421-4d98-87e2-6b57a5662b66\" (UID: \"783e5ed2-e421-4d98-87e2-6b57a5662b66\") "
Nov 02 15:47:47.320777 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:47.320293    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/783e5ed2-e421-4d98-87e2-6b57a5662b66-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/783e5ed2-e421-4d98-87e2-6b57a5662b66-istio-envoy\") pod \"783e5ed2-e421-4d98-87e2-6b57a5662b66\" (UID: \"783e5ed2-e421-4d98-87e2-6b57a5662b66\") "
Nov 02 15:47:47.320777 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:47.320378    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/783e5ed2-e421-4d98-87e2-6b57a5662b66-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/783e5ed2-e421-4d98-87e2-6b57a5662b66-istio-podinfo\") pod \"783e5ed2-e421-4d98-87e2-6b57a5662b66\" (UID: \"783e5ed2-e421-4d98-87e2-6b57a5662b66\") "
Nov 02 15:47:47.320777 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:47.320446    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/783e5ed2-e421-4d98-87e2-6b57a5662b66-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/783e5ed2-e421-4d98-87e2-6b57a5662b66-workload-certs\") pod \"783e5ed2-e421-4d98-87e2-6b57a5662b66\" (UID: \"783e5ed2-e421-4d98-87e2-6b57a5662b66\") "
Nov 02 15:47:47.320777 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:47.320529    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/783e5ed2-e421-4d98-87e2-6b57a5662b66-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/783e5ed2-e421-4d98-87e2-6b57a5662b66-credential-socket\") pod \"783e5ed2-e421-4d98-87e2-6b57a5662b66\" (UID: \"783e5ed2-e421-4d98-87e2-6b57a5662b66\") "
Nov 02 15:47:47.320777 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:47.320571    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/783e5ed2-e421-4d98-87e2-6b57a5662b66-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/783e5ed2-e421-4d98-87e2-6b57a5662b66-istiod-ca-cert\") pod \"783e5ed2-e421-4d98-87e2-6b57a5662b66\" (UID: \"783e5ed2-e421-4d98-87e2-6b57a5662b66\") "
Nov 02 15:47:47.321487 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:47.321214    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/783e5ed2-e421-4d98-87e2-6b57a5662b66-istio-data" pod "783e5ed2-e421-4d98-87e2-6b57a5662b66" (UID: "783e5ed2-e421-4d98-87e2-6b57a5662b66"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:47:47.321487 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:47.321393    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/783e5ed2-e421-4d98-87e2-6b57a5662b66-workload-certs" pod "783e5ed2-e421-4d98-87e2-6b57a5662b66" (UID: "783e5ed2-e421-4d98-87e2-6b57a5662b66"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:47:47.321722 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:47.321699    5304 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/783e5ed2-e421-4d98-87e2-6b57a5662b66-istio-data\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:47:47.321798 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:47.321727    5304 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/783e5ed2-e421-4d98-87e2-6b57a5662b66-workload-certs\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:47:47.321798 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:47.321780    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/783e5ed2-e421-4d98-87e2-6b57a5662b66-istiod-ca-cert" pod "783e5ed2-e421-4d98-87e2-6b57a5662b66" (UID: "783e5ed2-e421-4d98-87e2-6b57a5662b66"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:47:47.321902 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:47.321852    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/783e5ed2-e421-4d98-87e2-6b57a5662b66-workload-socket" pod "783e5ed2-e421-4d98-87e2-6b57a5662b66" (UID: "783e5ed2-e421-4d98-87e2-6b57a5662b66"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:47:47.321972 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:47.321956    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/783e5ed2-e421-4d98-87e2-6b57a5662b66-credential-socket" pod "783e5ed2-e421-4d98-87e2-6b57a5662b66" (UID: "783e5ed2-e421-4d98-87e2-6b57a5662b66"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:47:47.323806 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:47.323765    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/783e5ed2-e421-4d98-87e2-6b57a5662b66-istio-token" pod "783e5ed2-e421-4d98-87e2-6b57a5662b66" (UID: "783e5ed2-e421-4d98-87e2-6b57a5662b66"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:47:47.325458 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:47.325428    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/783e5ed2-e421-4d98-87e2-6b57a5662b66-istio-envoy" pod "783e5ed2-e421-4d98-87e2-6b57a5662b66" (UID: "783e5ed2-e421-4d98-87e2-6b57a5662b66"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:47:47.325818 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:47.325789    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/783e5ed2-e421-4d98-87e2-6b57a5662b66-istio-podinfo" pod "783e5ed2-e421-4d98-87e2-6b57a5662b66" (UID: "783e5ed2-e421-4d98-87e2-6b57a5662b66"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 15:47:47.326175 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:47.326149    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/783e5ed2-e421-4d98-87e2-6b57a5662b66-kube-api-access-5q6wz" pod "783e5ed2-e421-4d98-87e2-6b57a5662b66" (UID: "783e5ed2-e421-4d98-87e2-6b57a5662b66"). InnerVolumeSpecName "kube-api-access-5q6wz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:47:47.383152 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:47.376908    5304 patch_prober.go:29] interesting pod/e2e-ipp-a-f0039c-openshift-default-5f656fc965-rjf2w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused" start-of-body=
Nov 02 15:47:47.383152 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:47.376983    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f0039c-openshift-default-5f656fc965-rjf2w" podUID="9178b9f8-6fda-40a9-8850-fa0097f2d39a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused"
Nov 02 15:47:47.423114 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:47.423023    5304 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/783e5ed2-e421-4d98-87e2-6b57a5662b66-istio-podinfo\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:47:47.423114 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:47.423049    5304 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/783e5ed2-e421-4d98-87e2-6b57a5662b66-credential-socket\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:47:47.423114 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:47.423059    5304 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/783e5ed2-e421-4d98-87e2-6b57a5662b66-istiod-ca-cert\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:47:47.423114 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:47.423068    5304 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/783e5ed2-e421-4d98-87e2-6b57a5662b66-istio-token\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:47:47.423114 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:47.423076    5304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5q6wz\" (UniqueName: \"kubernetes.io/projected/783e5ed2-e421-4d98-87e2-6b57a5662b66-kube-api-access-5q6wz\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:47:47.423114 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:47.423084    5304 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/783e5ed2-e421-4d98-87e2-6b57a5662b66-workload-socket\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:47:47.423114 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:47.423093    5304 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/783e5ed2-e421-4d98-87e2-6b57a5662b66-istio-envoy\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:47:47.722178 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:47.722114    5304 patch_prober.go:29] interesting pod/e2e-ipp-b-24255b-openshift-default-6687484f49-992gq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused" start-of-body=
Nov 02 15:47:47.722410 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:47.722216    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-24255b-openshift-default-6687484f49-992gq" podUID="c19acd3a-5dbf-4da3-b88f-e36284b22243" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused"
Nov 02 15:47:48.211712 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:48.211673    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-b955c5-openshift-default-9fd99b497-tpwvk" event={"ID":"783e5ed2-e421-4d98-87e2-6b57a5662b66","Type":"ContainerDied","Data":"74b9156014112823728effc572cb4aa2048b4f72e8412f3b76f76e88e4e46648"}
Nov 02 15:47:48.212189 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:48.211744    5304 scope.go:122] "RemoveContainer" containerID="bbb27e8a8c3a112958382317700b3ddaaa5ed7a1515d2dc3af164ca7c82eba22"
Nov 02 15:47:48.212189 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:48.211863    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-b955c5-openshift-default-9fd99b497-tpwvk"
Nov 02 15:47:48.322543 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:48.322510    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-b955c5-openshift-default-9fd99b497-tpwvk"]
Nov 02 15:47:48.325634 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:48.325606    5304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-b955c5-openshift-default-9fd99b497-tpwvk"]
Nov 02 15:47:48.361151 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:48.358566    5304 patch_prober.go:29] interesting pod/e2e-ipp-a-f0039c-openshift-default-5f656fc965-rjf2w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused" start-of-body=
Nov 02 15:47:48.361151 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:48.358629    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f0039c-openshift-default-5f656fc965-rjf2w" podUID="9178b9f8-6fda-40a9-8850-fa0097f2d39a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused"
Nov 02 15:47:48.474213 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:48.473190    5304 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-f0039c-b78956c9-vm85j" podUID="97dde426-8f56-4bdd-a2f8-cb78c00e27a6" containerName="payload-processing" containerID="cri-o://a24d3aa92683081a4e12edb7e7c80257264e7d3f80f1c1548b9dee328972327c" gracePeriod=25
Nov 02 15:47:48.506677 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:48.506628    5304 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f0039c-6c5bf99c7f-62jkr" podUID="6d2ec092-d84c-4335-99b9-6b14712f57ed" containerName="payload-pre-processing" containerID="cri-o://702a6227dd8fed3d4853cbba6c8d431c3de634078f8847c887db5c3778399290" gracePeriod=25
Nov 02 15:47:48.721920 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:48.721837    5304 patch_prober.go:29] interesting pod/e2e-ipp-b-24255b-openshift-default-6687484f49-992gq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused" start-of-body=
Nov 02 15:47:48.722093 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:48.721915    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-24255b-openshift-default-6687484f49-992gq" podUID="c19acd3a-5dbf-4da3-b88f-e36284b22243" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused"
Nov 02 15:47:48.928753 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:48.928724    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-f0039c-b78956c9-vm85j"
Nov 02 15:47:48.994155 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:48.993177    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/97dde426-8f56-4bdd-a2f8-cb78c00e27a6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97dde426-8f56-4bdd-a2f8-cb78c00e27a6-plugins-config-volume\") pod \"97dde426-8f56-4bdd-a2f8-cb78c00e27a6\" (UID: \"97dde426-8f56-4bdd-a2f8-cb78c00e27a6\") "
Nov 02 15:47:48.994155 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:48.993381    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97dde426-8f56-4bdd-a2f8-cb78c00e27a6-kube-api-access-w8mdb\" (UniqueName: \"kubernetes.io/projected/97dde426-8f56-4bdd-a2f8-cb78c00e27a6-kube-api-access-w8mdb\") pod \"97dde426-8f56-4bdd-a2f8-cb78c00e27a6\" (UID: \"97dde426-8f56-4bdd-a2f8-cb78c00e27a6\") "
Nov 02 15:47:48.994155 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:48.993614    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97dde426-8f56-4bdd-a2f8-cb78c00e27a6-plugins-config-volume" pod "97dde426-8f56-4bdd-a2f8-cb78c00e27a6" (UID: "97dde426-8f56-4bdd-a2f8-cb78c00e27a6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:47:48.994155 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:48.993846    5304 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97dde426-8f56-4bdd-a2f8-cb78c00e27a6-plugins-config-volume\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:47:48.999891 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:48.999862    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97dde426-8f56-4bdd-a2f8-cb78c00e27a6-kube-api-access-w8mdb" pod "97dde426-8f56-4bdd-a2f8-cb78c00e27a6" (UID: "97dde426-8f56-4bdd-a2f8-cb78c00e27a6"). InnerVolumeSpecName "kube-api-access-w8mdb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:47:49.088049 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:49.088021    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f0039c-6c5bf99c7f-62jkr"
Nov 02 15:47:49.094597 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:49.094567    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6d2ec092-d84c-4335-99b9-6b14712f57ed-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d2ec092-d84c-4335-99b9-6b14712f57ed-plugins-config-volume\") pod \"6d2ec092-d84c-4335-99b9-6b14712f57ed\" (UID: \"6d2ec092-d84c-4335-99b9-6b14712f57ed\") "
Nov 02 15:47:49.094751 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:49.094616    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d2ec092-d84c-4335-99b9-6b14712f57ed-kube-api-access-rtgv6\" (UniqueName: \"kubernetes.io/projected/6d2ec092-d84c-4335-99b9-6b14712f57ed-kube-api-access-rtgv6\") pod \"6d2ec092-d84c-4335-99b9-6b14712f57ed\" (UID: \"6d2ec092-d84c-4335-99b9-6b14712f57ed\") "
Nov 02 15:47:49.094958 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:49.094932    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d2ec092-d84c-4335-99b9-6b14712f57ed-plugins-config-volume" pod "6d2ec092-d84c-4335-99b9-6b14712f57ed" (UID: "6d2ec092-d84c-4335-99b9-6b14712f57ed"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:47:49.095257 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:49.095237    5304 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d2ec092-d84c-4335-99b9-6b14712f57ed-plugins-config-volume\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:47:49.095350 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:49.095261    5304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w8mdb\" (UniqueName: \"kubernetes.io/projected/97dde426-8f56-4bdd-a2f8-cb78c00e27a6-kube-api-access-w8mdb\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:47:49.096790 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:49.096768    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d2ec092-d84c-4335-99b9-6b14712f57ed-kube-api-access-rtgv6" pod "6d2ec092-d84c-4335-99b9-6b14712f57ed" (UID: "6d2ec092-d84c-4335-99b9-6b14712f57ed"). InnerVolumeSpecName "kube-api-access-rtgv6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:47:49.196984 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:49.196892    5304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rtgv6\" (UniqueName: \"kubernetes.io/projected/6d2ec092-d84c-4335-99b9-6b14712f57ed-kube-api-access-rtgv6\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:47:49.222139 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:49.221872    5304 generic.go:356] "Generic (PLEG): container finished" podID="6d2ec092-d84c-4335-99b9-6b14712f57ed" containerID="702a6227dd8fed3d4853cbba6c8d431c3de634078f8847c887db5c3778399290" exitCode=0
Nov 02 15:47:49.222139 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:49.222054    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f0039c-6c5bf99c7f-62jkr"
Nov 02 15:47:49.223736 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:49.222683    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f0039c-6c5bf99c7f-62jkr" event={"ID":"6d2ec092-d84c-4335-99b9-6b14712f57ed","Type":"ContainerDied","Data":"702a6227dd8fed3d4853cbba6c8d431c3de634078f8847c887db5c3778399290"}
Nov 02 15:47:49.223736 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:49.222750    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f0039c-6c5bf99c7f-62jkr" event={"ID":"6d2ec092-d84c-4335-99b9-6b14712f57ed","Type":"ContainerDied","Data":"3338303cf3fdbf856359b2e81a0b5dae9e4857c1c917f9e9669e1e34672def1e"}
Nov 02 15:47:49.223736 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:49.222783    5304 scope.go:122] "RemoveContainer" containerID="702a6227dd8fed3d4853cbba6c8d431c3de634078f8847c887db5c3778399290"
Nov 02 15:47:49.225200 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:49.225102    5304 generic.go:356] "Generic (PLEG): container finished" podID="97dde426-8f56-4bdd-a2f8-cb78c00e27a6" containerID="a24d3aa92683081a4e12edb7e7c80257264e7d3f80f1c1548b9dee328972327c" exitCode=0
Nov 02 15:47:49.225200 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:49.225167    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-f0039c-b78956c9-vm85j"
Nov 02 15:47:49.225318 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:49.225193    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-f0039c-b78956c9-vm85j" event={"ID":"97dde426-8f56-4bdd-a2f8-cb78c00e27a6","Type":"ContainerDied","Data":"a24d3aa92683081a4e12edb7e7c80257264e7d3f80f1c1548b9dee328972327c"}
Nov 02 15:47:49.225318 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:49.225248    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-f0039c-b78956c9-vm85j" event={"ID":"97dde426-8f56-4bdd-a2f8-cb78c00e27a6","Type":"ContainerDied","Data":"f2f15c11268dec6b6dd0394a9c56b8ae89820f9efa2b82d082ab2ff517c25602"}
Nov 02 15:47:49.285499 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:49.285478    5304 scope.go:122] "RemoveContainer" containerID="702a6227dd8fed3d4853cbba6c8d431c3de634078f8847c887db5c3778399290"
Nov 02 15:47:49.286496 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:47:49.286469    5304 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"702a6227dd8fed3d4853cbba6c8d431c3de634078f8847c887db5c3778399290\": container with ID starting with 702a6227dd8fed3d4853cbba6c8d431c3de634078f8847c887db5c3778399290 not found: ID does not exist" containerID="702a6227dd8fed3d4853cbba6c8d431c3de634078f8847c887db5c3778399290"
Nov 02 15:47:49.286808 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:49.286508    5304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"702a6227dd8fed3d4853cbba6c8d431c3de634078f8847c887db5c3778399290"} err="failed to get container status \"702a6227dd8fed3d4853cbba6c8d431c3de634078f8847c887db5c3778399290\": rpc error: code = NotFound desc = could not find container \"702a6227dd8fed3d4853cbba6c8d431c3de634078f8847c887db5c3778399290\": container with ID starting with 702a6227dd8fed3d4853cbba6c8d431c3de634078f8847c887db5c3778399290 not found: ID does not exist"
Nov 02 15:47:49.286808 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:49.286532    5304 scope.go:122] "RemoveContainer" containerID="a24d3aa92683081a4e12edb7e7c80257264e7d3f80f1c1548b9dee328972327c"
Nov 02 15:47:49.296299 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:49.296278    5304 scope.go:122] "RemoveContainer" containerID="a24d3aa92683081a4e12edb7e7c80257264e7d3f80f1c1548b9dee328972327c"
Nov 02 15:47:49.296580 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:47:49.296558    5304 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a24d3aa92683081a4e12edb7e7c80257264e7d3f80f1c1548b9dee328972327c\": container with ID starting with a24d3aa92683081a4e12edb7e7c80257264e7d3f80f1c1548b9dee328972327c not found: ID does not exist" containerID="a24d3aa92683081a4e12edb7e7c80257264e7d3f80f1c1548b9dee328972327c"
Nov 02 15:47:49.296630 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:49.296607    5304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a24d3aa92683081a4e12edb7e7c80257264e7d3f80f1c1548b9dee328972327c"} err="failed to get container status \"a24d3aa92683081a4e12edb7e7c80257264e7d3f80f1c1548b9dee328972327c\": rpc error: code = NotFound desc = could not find container \"a24d3aa92683081a4e12edb7e7c80257264e7d3f80f1c1548b9dee328972327c\": container with ID starting with a24d3aa92683081a4e12edb7e7c80257264e7d3f80f1c1548b9dee328972327c not found: ID does not exist"
Nov 02 15:47:49.382178 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:49.368522    5304 patch_prober.go:29] interesting pod/e2e-ipp-a-f0039c-openshift-default-5f656fc965-rjf2w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused" start-of-body=
Nov 02 15:47:49.382178 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:49.368575    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f0039c-openshift-default-5f656fc965-rjf2w" podUID="9178b9f8-6fda-40a9-8850-fa0097f2d39a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused"
Nov 02 15:47:49.383279 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:49.383234    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f0039c-6c5bf99c7f-62jkr"]
Nov 02 15:47:49.386924 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:49.386902    5304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f0039c-6c5bf99c7f-62jkr"]
Nov 02 15:47:49.393700 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:49.393673    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f0039c-b78956c9-vm85j"]
Nov 02 15:47:49.396207 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:49.396184    5304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f0039c-b78956c9-vm85j"]
Nov 02 15:47:49.722323 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:49.722284    5304 patch_prober.go:29] interesting pod/e2e-ipp-b-24255b-openshift-default-6687484f49-992gq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused" start-of-body=
Nov 02 15:47:49.722506 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:49.722350    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-24255b-openshift-default-6687484f49-992gq" podUID="c19acd3a-5dbf-4da3-b88f-e36284b22243" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused"
Nov 02 15:47:50.169149 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:50.165802    5304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6d2ec092-d84c-4335-99b9-6b14712f57ed" path="/var/lib/kubelet/pods/6d2ec092-d84c-4335-99b9-6b14712f57ed/volumes"
Nov 02 15:47:50.169149 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:50.166510    5304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="783e5ed2-e421-4d98-87e2-6b57a5662b66" path="/var/lib/kubelet/pods/783e5ed2-e421-4d98-87e2-6b57a5662b66/volumes"
Nov 02 15:47:50.169149 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:50.167080    5304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="97dde426-8f56-4bdd-a2f8-cb78c00e27a6" path="/var/lib/kubelet/pods/97dde426-8f56-4bdd-a2f8-cb78c00e27a6/volumes"
Nov 02 15:47:50.363344 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:50.363306    5304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-f0039c-openshift-default-5f656fc965-rjf2w"
Nov 02 15:47:50.363859 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:50.363723    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-f0039c-openshift-default-5f656fc965-rjf2w"
Nov 02 15:47:50.364394 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:50.364370    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-f0039c-openshift-default-5f656fc965-rjf2w"
Nov 02 15:47:50.722060 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:50.722020    5304 patch_prober.go:29] interesting pod/e2e-ipp-b-24255b-openshift-default-6687484f49-992gq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused" start-of-body=
Nov 02 15:47:50.722287 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:50.722082    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-24255b-openshift-default-6687484f49-992gq" podUID="c19acd3a-5dbf-4da3-b88f-e36284b22243" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused"
Nov 02 15:47:51.035112 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:51.034958    5304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-5721b1-openshift-default-56c5cbb5cc-987lj"]
Nov 02 15:47:51.036176 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:51.036149    5304 cpu_manager.go:414] "containerMap: removing container" podUID="783e5ed2-e421-4d98-87e2-6b57a5662b66" containerName="istio-proxy"
Nov 02 15:47:51.036660 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:51.036638    5304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="783e5ed2-e421-4d98-87e2-6b57a5662b66" containerName="istio-proxy"
Nov 02 15:47:51.036817 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:51.036802    5304 cpu_manager.go:414] "containerMap: removing container" podUID="97dde426-8f56-4bdd-a2f8-cb78c00e27a6" containerName="payload-processing"
Nov 02 15:47:51.036904 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:51.036892    5304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97dde426-8f56-4bdd-a2f8-cb78c00e27a6" containerName="payload-processing"
Nov 02 15:47:51.037035 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:51.037021    5304 cpu_manager.go:414] "containerMap: removing container" podUID="6d2ec092-d84c-4335-99b9-6b14712f57ed" containerName="payload-pre-processing"
Nov 02 15:47:51.037160 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:51.037146    5304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d2ec092-d84c-4335-99b9-6b14712f57ed" containerName="payload-pre-processing"
Nov 02 15:47:51.037316 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:51.037303    5304 cpu_manager.go:414] "containerMap: removing container" podUID="52064ae5-d62f-42f6-a50e-5030d6382aa3" containerName="payload-pre-processing"
Nov 02 15:47:51.037410 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:51.037396    5304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52064ae5-d62f-42f6-a50e-5030d6382aa3" containerName="payload-pre-processing"
Nov 02 15:47:51.037655 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:51.037638    5304 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d2ec092-d84c-4335-99b9-6b14712f57ed" containerName="payload-pre-processing"
Nov 02 15:47:51.037764 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:51.037751    5304 memory_manager.go:365] "RemoveStaleState removing state" podUID="783e5ed2-e421-4d98-87e2-6b57a5662b66" containerName="istio-proxy"
Nov 02 15:47:51.037857 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:51.037846    5304 memory_manager.go:365] "RemoveStaleState removing state" podUID="52064ae5-d62f-42f6-a50e-5030d6382aa3" containerName="payload-pre-processing"
Nov 02 15:47:51.037965 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:51.037954    5304 memory_manager.go:365] "RemoveStaleState removing state" podUID="97dde426-8f56-4bdd-a2f8-cb78c00e27a6" containerName="payload-processing"
Nov 02 15:47:51.075395 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:51.074016    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-5721b1-openshift-default-56c5cbb5cc-987lj"]
Nov 02 15:47:51.075395 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:51.074166    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-5721b1-openshift-default-56c5cbb5cc-987lj"
Nov 02 15:47:51.093725 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:51.090602    5304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-5721b1-openshift-default-dockercfg-v94mw\""
Nov 02 15:47:51.120992 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:51.120953    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1285186b-0b54-48b3-84c6-b640966bb312-istio-podinfo\") pod \"e2e-ipp-b-5721b1-openshift-default-56c5cbb5cc-987lj\" (UID: \"1285186b-0b54-48b3-84c6-b640966bb312\") " pod="openshift-ingress/e2e-ipp-b-5721b1-openshift-default-56c5cbb5cc-987lj"
Nov 02 15:47:51.121192 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:51.121011    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b2fbt\" (UniqueName: \"kubernetes.io/projected/1285186b-0b54-48b3-84c6-b640966bb312-kube-api-access-b2fbt\") pod \"e2e-ipp-b-5721b1-openshift-default-56c5cbb5cc-987lj\" (UID: \"1285186b-0b54-48b3-84c6-b640966bb312\") " pod="openshift-ingress/e2e-ipp-b-5721b1-openshift-default-56c5cbb5cc-987lj"
Nov 02 15:47:51.121192 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:51.121103    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1285186b-0b54-48b3-84c6-b640966bb312-workload-socket\") pod \"e2e-ipp-b-5721b1-openshift-default-56c5cbb5cc-987lj\" (UID: \"1285186b-0b54-48b3-84c6-b640966bb312\") " pod="openshift-ingress/e2e-ipp-b-5721b1-openshift-default-56c5cbb5cc-987lj"
Nov 02 15:47:51.121289 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:51.121222    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1285186b-0b54-48b3-84c6-b640966bb312-istiod-ca-cert\") pod \"e2e-ipp-b-5721b1-openshift-default-56c5cbb5cc-987lj\" (UID: \"1285186b-0b54-48b3-84c6-b640966bb312\") " pod="openshift-ingress/e2e-ipp-b-5721b1-openshift-default-56c5cbb5cc-987lj"
Nov 02 15:47:51.121289 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:51.121264    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1285186b-0b54-48b3-84c6-b640966bb312-istio-envoy\") pod \"e2e-ipp-b-5721b1-openshift-default-56c5cbb5cc-987lj\" (UID: \"1285186b-0b54-48b3-84c6-b640966bb312\") " pod="openshift-ingress/e2e-ipp-b-5721b1-openshift-default-56c5cbb5cc-987lj"
Nov 02 15:47:51.121425 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:51.121402    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1285186b-0b54-48b3-84c6-b640966bb312-istio-data\") pod \"e2e-ipp-b-5721b1-openshift-default-56c5cbb5cc-987lj\" (UID: \"1285186b-0b54-48b3-84c6-b640966bb312\") " pod="openshift-ingress/e2e-ipp-b-5721b1-openshift-default-56c5cbb5cc-987lj"
Nov 02 15:47:51.121486 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:51.121458    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1285186b-0b54-48b3-84c6-b640966bb312-credential-socket\") pod \"e2e-ipp-b-5721b1-openshift-default-56c5cbb5cc-987lj\" (UID: \"1285186b-0b54-48b3-84c6-b640966bb312\") " pod="openshift-ingress/e2e-ipp-b-5721b1-openshift-default-56c5cbb5cc-987lj"
Nov 02 15:47:51.121538 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:51.121517    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1285186b-0b54-48b3-84c6-b640966bb312-istio-token\") pod \"e2e-ipp-b-5721b1-openshift-default-56c5cbb5cc-987lj\" (UID: \"1285186b-0b54-48b3-84c6-b640966bb312\") " pod="openshift-ingress/e2e-ipp-b-5721b1-openshift-default-56c5cbb5cc-987lj"
Nov 02 15:47:51.121596 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:51.121559    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1285186b-0b54-48b3-84c6-b640966bb312-workload-certs\") pod \"e2e-ipp-b-5721b1-openshift-default-56c5cbb5cc-987lj\" (UID: \"1285186b-0b54-48b3-84c6-b640966bb312\") " pod="openshift-ingress/e2e-ipp-b-5721b1-openshift-default-56c5cbb5cc-987lj"
Nov 02 15:47:51.234162 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:51.225361    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1285186b-0b54-48b3-84c6-b640966bb312-istio-podinfo\") pod \"e2e-ipp-b-5721b1-openshift-default-56c5cbb5cc-987lj\" (UID: \"1285186b-0b54-48b3-84c6-b640966bb312\") " pod="openshift-ingress/e2e-ipp-b-5721b1-openshift-default-56c5cbb5cc-987lj"
Nov 02 15:47:51.234162 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:51.225643    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b2fbt\" (UniqueName: \"kubernetes.io/projected/1285186b-0b54-48b3-84c6-b640966bb312-kube-api-access-b2fbt\") pod \"e2e-ipp-b-5721b1-openshift-default-56c5cbb5cc-987lj\" (UID: \"1285186b-0b54-48b3-84c6-b640966bb312\") " pod="openshift-ingress/e2e-ipp-b-5721b1-openshift-default-56c5cbb5cc-987lj"
Nov 02 15:47:51.234162 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:51.225785    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1285186b-0b54-48b3-84c6-b640966bb312-workload-socket\") pod \"e2e-ipp-b-5721b1-openshift-default-56c5cbb5cc-987lj\" (UID: \"1285186b-0b54-48b3-84c6-b640966bb312\") " pod="openshift-ingress/e2e-ipp-b-5721b1-openshift-default-56c5cbb5cc-987lj"
Nov 02 15:47:51.234162 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:51.226121    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1285186b-0b54-48b3-84c6-b640966bb312-istiod-ca-cert\") pod \"e2e-ipp-b-5721b1-openshift-default-56c5cbb5cc-987lj\" (UID: \"1285186b-0b54-48b3-84c6-b640966bb312\") " pod="openshift-ingress/e2e-ipp-b-5721b1-openshift-default-56c5cbb5cc-987lj"
Nov 02 15:47:51.234162 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:51.226236    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1285186b-0b54-48b3-84c6-b640966bb312-istio-envoy\") pod \"e2e-ipp-b-5721b1-openshift-default-56c5cbb5cc-987lj\" (UID: \"1285186b-0b54-48b3-84c6-b640966bb312\") " pod="openshift-ingress/e2e-ipp-b-5721b1-openshift-default-56c5cbb5cc-987lj"
Nov 02 15:47:51.234162 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:51.226567    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1285186b-0b54-48b3-84c6-b640966bb312-istio-data\") pod \"e2e-ipp-b-5721b1-openshift-default-56c5cbb5cc-987lj\" (UID: \"1285186b-0b54-48b3-84c6-b640966bb312\") " pod="openshift-ingress/e2e-ipp-b-5721b1-openshift-default-56c5cbb5cc-987lj"
Nov 02 15:47:51.234162 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:51.226616    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1285186b-0b54-48b3-84c6-b640966bb312-credential-socket\") pod \"e2e-ipp-b-5721b1-openshift-default-56c5cbb5cc-987lj\" (UID: \"1285186b-0b54-48b3-84c6-b640966bb312\") " pod="openshift-ingress/e2e-ipp-b-5721b1-openshift-default-56c5cbb5cc-987lj"
Nov 02 15:47:51.234162 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:51.226668    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1285186b-0b54-48b3-84c6-b640966bb312-istio-token\") pod \"e2e-ipp-b-5721b1-openshift-default-56c5cbb5cc-987lj\" (UID: \"1285186b-0b54-48b3-84c6-b640966bb312\") " pod="openshift-ingress/e2e-ipp-b-5721b1-openshift-default-56c5cbb5cc-987lj"
Nov 02 15:47:51.234162 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:51.226707    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1285186b-0b54-48b3-84c6-b640966bb312-workload-certs\") pod \"e2e-ipp-b-5721b1-openshift-default-56c5cbb5cc-987lj\" (UID: \"1285186b-0b54-48b3-84c6-b640966bb312\") " pod="openshift-ingress/e2e-ipp-b-5721b1-openshift-default-56c5cbb5cc-987lj"
Nov 02 15:47:51.234162 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:51.227617    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1285186b-0b54-48b3-84c6-b640966bb312-workload-socket\") pod \"e2e-ipp-b-5721b1-openshift-default-56c5cbb5cc-987lj\" (UID: \"1285186b-0b54-48b3-84c6-b640966bb312\") " pod="openshift-ingress/e2e-ipp-b-5721b1-openshift-default-56c5cbb5cc-987lj"
Nov 02 15:47:51.247188 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:51.243496    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1285186b-0b54-48b3-84c6-b640966bb312-credential-socket\") pod \"e2e-ipp-b-5721b1-openshift-default-56c5cbb5cc-987lj\" (UID: \"1285186b-0b54-48b3-84c6-b640966bb312\") " pod="openshift-ingress/e2e-ipp-b-5721b1-openshift-default-56c5cbb5cc-987lj"
Nov 02 15:47:51.247188 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:51.243771    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1285186b-0b54-48b3-84c6-b640966bb312-istio-data\") pod \"e2e-ipp-b-5721b1-openshift-default-56c5cbb5cc-987lj\" (UID: \"1285186b-0b54-48b3-84c6-b640966bb312\") " pod="openshift-ingress/e2e-ipp-b-5721b1-openshift-default-56c5cbb5cc-987lj"
Nov 02 15:47:51.247188 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:51.244034    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1285186b-0b54-48b3-84c6-b640966bb312-workload-certs\") pod \"e2e-ipp-b-5721b1-openshift-default-56c5cbb5cc-987lj\" (UID: \"1285186b-0b54-48b3-84c6-b640966bb312\") " pod="openshift-ingress/e2e-ipp-b-5721b1-openshift-default-56c5cbb5cc-987lj"
Nov 02 15:47:51.270151 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:51.257182    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1285186b-0b54-48b3-84c6-b640966bb312-istiod-ca-cert\") pod \"e2e-ipp-b-5721b1-openshift-default-56c5cbb5cc-987lj\" (UID: \"1285186b-0b54-48b3-84c6-b640966bb312\") " pod="openshift-ingress/e2e-ipp-b-5721b1-openshift-default-56c5cbb5cc-987lj"
Nov 02 15:47:51.292199 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:51.271863    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1285186b-0b54-48b3-84c6-b640966bb312-istio-podinfo\") pod \"e2e-ipp-b-5721b1-openshift-default-56c5cbb5cc-987lj\" (UID: \"1285186b-0b54-48b3-84c6-b640966bb312\") " pod="openshift-ingress/e2e-ipp-b-5721b1-openshift-default-56c5cbb5cc-987lj"
Nov 02 15:47:51.292199 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:51.274451    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1285186b-0b54-48b3-84c6-b640966bb312-istio-envoy\") pod \"e2e-ipp-b-5721b1-openshift-default-56c5cbb5cc-987lj\" (UID: \"1285186b-0b54-48b3-84c6-b640966bb312\") " pod="openshift-ingress/e2e-ipp-b-5721b1-openshift-default-56c5cbb5cc-987lj"
Nov 02 15:47:51.292199 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:51.286217    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b2fbt\" (UniqueName: \"kubernetes.io/projected/1285186b-0b54-48b3-84c6-b640966bb312-kube-api-access-b2fbt\") pod \"e2e-ipp-b-5721b1-openshift-default-56c5cbb5cc-987lj\" (UID: \"1285186b-0b54-48b3-84c6-b640966bb312\") " pod="openshift-ingress/e2e-ipp-b-5721b1-openshift-default-56c5cbb5cc-987lj"
Nov 02 15:47:51.313151 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:51.312009    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1285186b-0b54-48b3-84c6-b640966bb312-istio-token\") pod \"e2e-ipp-b-5721b1-openshift-default-56c5cbb5cc-987lj\" (UID: \"1285186b-0b54-48b3-84c6-b640966bb312\") " pod="openshift-ingress/e2e-ipp-b-5721b1-openshift-default-56c5cbb5cc-987lj"
Nov 02 15:47:51.403199 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:51.394172    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-5721b1-openshift-default-56c5cbb5cc-987lj"
Nov 02 15:47:51.722576 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:51.722540    5304 patch_prober.go:29] interesting pod/e2e-ipp-b-24255b-openshift-default-6687484f49-992gq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused" start-of-body=
Nov 02 15:47:51.722760 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:51.722598    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-24255b-openshift-default-6687484f49-992gq" podUID="c19acd3a-5dbf-4da3-b88f-e36284b22243" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused"
Nov 02 15:47:51.774189 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:51.774014    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-5721b1-openshift-default-56c5cbb5cc-987lj"]
Nov 02 15:47:51.777153 ip-10-0-135-138 kubenswrapper[5304]: W1102 15:47:51.777110    5304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1285186b_0b54_48b3_84c6_b640966bb312.slice/crio-9b4e1b02012bf83802c44c13f4572ee55eb7cb0e0bdcaf043c9f10b5e0af75d3 WatchSource:0}: Error finding container 9b4e1b02012bf83802c44c13f4572ee55eb7cb0e0bdcaf043c9f10b5e0af75d3: Status 404 returned error can't find the container with id 9b4e1b02012bf83802c44c13f4572ee55eb7cb0e0bdcaf043c9f10b5e0af75d3
Nov 02 15:47:51.779954 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:51.779919    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:47:51.780045 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:51.779989    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:47:51.780045 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:51.780021    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:47:52.307874 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:52.307826    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-5721b1-openshift-default-56c5cbb5cc-987lj" event={"ID":"1285186b-0b54-48b3-84c6-b640966bb312","Type":"ContainerStarted","Data":"4eb2e692475edc363f1e913a7bc8179f0ccb216421f28c4a087c0e4a9bcd6285"}
Nov 02 15:47:52.308091 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:52.307885    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-5721b1-openshift-default-56c5cbb5cc-987lj" event={"ID":"1285186b-0b54-48b3-84c6-b640966bb312","Type":"ContainerStarted","Data":"9b4e1b02012bf83802c44c13f4572ee55eb7cb0e0bdcaf043c9f10b5e0af75d3"}
Nov 02 15:47:52.355695 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:52.355638    5304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-5721b1-openshift-default-56c5cbb5cc-987lj" podStartSLOduration=1.355623122 podStartE2EDuration="1.355623122s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:47:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:47:52.35227173 +0000 UTC m=+3072.346096139" watchObservedRunningTime="2026-11-02 15:47:52.355623122 +0000 UTC m=+3072.349447608"
Nov 02 15:47:52.395213 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:52.395184    5304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-5721b1-openshift-default-56c5cbb5cc-987lj"
Nov 02 15:47:52.401336 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:52.401308    5304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-5721b1-openshift-default-56c5cbb5cc-987lj"
Nov 02 15:47:52.726318 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:52.726286    5304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-24255b-openshift-default-6687484f49-992gq"
Nov 02 15:47:52.726813 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:52.726610    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-24255b-openshift-default-6687484f49-992gq"
Nov 02 15:47:52.727458 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:52.727437    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-24255b-openshift-default-6687484f49-992gq"
Nov 02 15:47:52.900809 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:52.900774    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-b55da1-7757f8c8bc-l8dpc"]
Nov 02 15:47:53.320991 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:53.320955    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-5721b1-openshift-default-56c5cbb5cc-987lj"
Nov 02 15:47:53.322214 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:53.322186    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-5721b1-openshift-default-56c5cbb5cc-987lj"
Nov 02 15:47:54.802717 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:47:54.802677    5304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k6fjk" containerName="extractor"
Nov 02 15:47:57.901963 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:47:57.901921    5304 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-b55da1-7757f8c8bc-l8dpc" podUID="1ec89e8e-8ec5-497f-b3ad-1570c6d0d3dd" containerName="payload-processing" containerID="cri-o://9eb97d07737b81346510341054528b102c06371546d047c73cdca7b1186e881f" gracePeriod=25
Nov 02 15:48:00.381110 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:48:00.381074    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-d79991-56bd8b5955-jkfrr"]
Nov 02 15:48:00.425370 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:48:00.424120    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-d79991-7765697955-29mq6"]
Nov 02 15:48:05.382048 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:48:05.382005    5304 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-d79991-56bd8b5955-jkfrr" podUID="588da214-2720-47e6-b537-f90977db6a92" containerName="payload-processing" containerID="cri-o://709cc3b9141e57728469a4574d2ab3753af7707308fe8d6b68b464e46bf37910" gracePeriod=25
Nov 02 15:48:05.427646 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:48:05.427602    5304 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d79991-7765697955-29mq6" podUID="ad1b2ff2-4594-4cf3-88a7-199030bae2a5" containerName="payload-pre-processing" containerID="cri-o://dff1da1027757d871240b7d5650073e55bb49a47b2b39b45a14a23a992fb7afc" gracePeriod=25
Nov 02 15:48:05.698450 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:48:05.698417    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-d79991-56bd8b5955-jkfrr"
Nov 02 15:48:05.752801 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:48:05.752768    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d79991-7765697955-29mq6"
Nov 02 15:48:05.830565 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:48:05.830523    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ad1b2ff2-4594-4cf3-88a7-199030bae2a5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad1b2ff2-4594-4cf3-88a7-199030bae2a5-plugins-config-volume\") pod \"ad1b2ff2-4594-4cf3-88a7-199030bae2a5\" (UID: \"ad1b2ff2-4594-4cf3-88a7-199030bae2a5\") "
Nov 02 15:48:05.830758 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:48:05.830708    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/588da214-2720-47e6-b537-f90977db6a92-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/588da214-2720-47e6-b537-f90977db6a92-plugins-config-volume\") pod \"588da214-2720-47e6-b537-f90977db6a92\" (UID: \"588da214-2720-47e6-b537-f90977db6a92\") "
Nov 02 15:48:05.830844 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:48:05.830830    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad1b2ff2-4594-4cf3-88a7-199030bae2a5-kube-api-access-dw7cb\" (UniqueName: \"kubernetes.io/projected/ad1b2ff2-4594-4cf3-88a7-199030bae2a5-kube-api-access-dw7cb\") pod \"ad1b2ff2-4594-4cf3-88a7-199030bae2a5\" (UID: \"ad1b2ff2-4594-4cf3-88a7-199030bae2a5\") "
Nov 02 15:48:05.830926 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:48:05.830892    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad1b2ff2-4594-4cf3-88a7-199030bae2a5-plugins-config-volume" pod "ad1b2ff2-4594-4cf3-88a7-199030bae2a5" (UID: "ad1b2ff2-4594-4cf3-88a7-199030bae2a5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:48:05.830991 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:48:05.830900    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/588da214-2720-47e6-b537-f90977db6a92-kube-api-access-dk45l\" (UniqueName: \"kubernetes.io/projected/588da214-2720-47e6-b537-f90977db6a92-kube-api-access-dk45l\") pod \"588da214-2720-47e6-b537-f90977db6a92\" (UID: \"588da214-2720-47e6-b537-f90977db6a92\") "
Nov 02 15:48:05.831183 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:48:05.831152    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/588da214-2720-47e6-b537-f90977db6a92-plugins-config-volume" pod "588da214-2720-47e6-b537-f90977db6a92" (UID: "588da214-2720-47e6-b537-f90977db6a92"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:48:05.831650 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:48:05.831628    5304 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/588da214-2720-47e6-b537-f90977db6a92-plugins-config-volume\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:48:05.831738 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:48:05.831657    5304 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad1b2ff2-4594-4cf3-88a7-199030bae2a5-plugins-config-volume\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:48:05.833031 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:48:05.833009    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/588da214-2720-47e6-b537-f90977db6a92-kube-api-access-dk45l" pod "588da214-2720-47e6-b537-f90977db6a92" (UID: "588da214-2720-47e6-b537-f90977db6a92"). InnerVolumeSpecName "kube-api-access-dk45l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:48:05.833496 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:48:05.833481    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad1b2ff2-4594-4cf3-88a7-199030bae2a5-kube-api-access-dw7cb" pod "ad1b2ff2-4594-4cf3-88a7-199030bae2a5" (UID: "ad1b2ff2-4594-4cf3-88a7-199030bae2a5"). InnerVolumeSpecName "kube-api-access-dw7cb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:48:05.932694 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:48:05.932610    5304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dw7cb\" (UniqueName: \"kubernetes.io/projected/ad1b2ff2-4594-4cf3-88a7-199030bae2a5-kube-api-access-dw7cb\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:48:05.932694 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:48:05.932641    5304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dk45l\" (UniqueName: \"kubernetes.io/projected/588da214-2720-47e6-b537-f90977db6a92-kube-api-access-dk45l\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:48:06.203880 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:48:06.203790    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-b55da1-7757f8c8bc-l8dpc" podUID="1ec89e8e-8ec5-497f-b3ad-1570c6d0d3dd" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 15:48:06.453623 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:48:06.453574    5304 generic.go:356] "Generic (PLEG): container finished" podID="ad1b2ff2-4594-4cf3-88a7-199030bae2a5" containerID="dff1da1027757d871240b7d5650073e55bb49a47b2b39b45a14a23a992fb7afc" exitCode=0
Nov 02 15:48:06.454205 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:48:06.453646    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d79991-7765697955-29mq6"
Nov 02 15:48:06.454205 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:48:06.453709    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d79991-7765697955-29mq6" event={"ID":"ad1b2ff2-4594-4cf3-88a7-199030bae2a5","Type":"ContainerDied","Data":"dff1da1027757d871240b7d5650073e55bb49a47b2b39b45a14a23a992fb7afc"}
Nov 02 15:48:06.454469 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:48:06.454443    5304 scope.go:122] "RemoveContainer" containerID="dff1da1027757d871240b7d5650073e55bb49a47b2b39b45a14a23a992fb7afc"
Nov 02 15:48:06.454571 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:48:06.454369    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d79991-7765697955-29mq6" event={"ID":"ad1b2ff2-4594-4cf3-88a7-199030bae2a5","Type":"ContainerDied","Data":"c7fed1f55eff2d8af7f1fd4607b3e829a94acb25ff5344bd23ec54e68c68d89e"}
Nov 02 15:48:06.456621 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:48:06.456596    5304 generic.go:356] "Generic (PLEG): container finished" podID="588da214-2720-47e6-b537-f90977db6a92" containerID="709cc3b9141e57728469a4574d2ab3753af7707308fe8d6b68b464e46bf37910" exitCode=0
Nov 02 15:48:06.456826 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:48:06.456792    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-d79991-56bd8b5955-jkfrr" event={"ID":"588da214-2720-47e6-b537-f90977db6a92","Type":"ContainerDied","Data":"709cc3b9141e57728469a4574d2ab3753af7707308fe8d6b68b464e46bf37910"}
Nov 02 15:48:06.456947 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:48:06.456835    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-d79991-56bd8b5955-jkfrr" event={"ID":"588da214-2720-47e6-b537-f90977db6a92","Type":"ContainerDied","Data":"ff79915d62e3fc26515b967f41bbd7256f61008815aeea6ecb50ce315a342dc8"}
Nov 02 15:48:06.456947 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:48:06.456847    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-d79991-56bd8b5955-jkfrr"
Nov 02 15:48:06.466014 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:48:06.465995    5304 scope.go:122] "RemoveContainer" containerID="dff1da1027757d871240b7d5650073e55bb49a47b2b39b45a14a23a992fb7afc"
Nov 02 15:48:06.466421 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:48:06.466399    5304 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dff1da1027757d871240b7d5650073e55bb49a47b2b39b45a14a23a992fb7afc\": container with ID starting with dff1da1027757d871240b7d5650073e55bb49a47b2b39b45a14a23a992fb7afc not found: ID does not exist" containerID="dff1da1027757d871240b7d5650073e55bb49a47b2b39b45a14a23a992fb7afc"
Nov 02 15:48:06.466513 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:48:06.466432    5304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dff1da1027757d871240b7d5650073e55bb49a47b2b39b45a14a23a992fb7afc"} err="failed to get container status \"dff1da1027757d871240b7d5650073e55bb49a47b2b39b45a14a23a992fb7afc\": rpc error: code = NotFound desc = could not find container \"dff1da1027757d871240b7d5650073e55bb49a47b2b39b45a14a23a992fb7afc\": container with ID starting with dff1da1027757d871240b7d5650073e55bb49a47b2b39b45a14a23a992fb7afc not found: ID does not exist"
Nov 02 15:48:06.466513 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:48:06.466453    5304 scope.go:122] "RemoveContainer" containerID="709cc3b9141e57728469a4574d2ab3753af7707308fe8d6b68b464e46bf37910"
Nov 02 15:48:06.475984 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:48:06.475960    5304 scope.go:122] "RemoveContainer" containerID="709cc3b9141e57728469a4574d2ab3753af7707308fe8d6b68b464e46bf37910"
Nov 02 15:48:06.476273 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:48:06.476252    5304 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"709cc3b9141e57728469a4574d2ab3753af7707308fe8d6b68b464e46bf37910\": container with ID starting with 709cc3b9141e57728469a4574d2ab3753af7707308fe8d6b68b464e46bf37910 not found: ID does not exist" containerID="709cc3b9141e57728469a4574d2ab3753af7707308fe8d6b68b464e46bf37910"
Nov 02 15:48:06.476357 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:48:06.476280    5304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"709cc3b9141e57728469a4574d2ab3753af7707308fe8d6b68b464e46bf37910"} err="failed to get container status \"709cc3b9141e57728469a4574d2ab3753af7707308fe8d6b68b464e46bf37910\": rpc error: code = NotFound desc = could not find container \"709cc3b9141e57728469a4574d2ab3753af7707308fe8d6b68b464e46bf37910\": container with ID starting with 709cc3b9141e57728469a4574d2ab3753af7707308fe8d6b68b464e46bf37910 not found: ID does not exist"
Nov 02 15:48:06.505640 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:48:06.505619    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-d79991-56bd8b5955-jkfrr"]
Nov 02 15:48:06.517768 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:48:06.517742    5304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-d79991-56bd8b5955-jkfrr"]
Nov 02 15:48:06.548471 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:48:06.548433    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-d79991-7765697955-29mq6"]
Nov 02 15:48:06.551726 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:48:06.551703    5304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-d79991-7765697955-29mq6"]
Nov 02 15:48:06.914181 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:48:06.909684    5304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-24255b-84c779c487-bnkpw"]
Nov 02 15:48:06.914181 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:48:06.910554    5304 cpu_manager.go:414] "containerMap: removing container" podUID="588da214-2720-47e6-b537-f90977db6a92" containerName="payload-processing"
Nov 02 15:48:06.914181 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:48:06.910575    5304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="588da214-2720-47e6-b537-f90977db6a92" containerName="payload-processing"
Nov 02 15:48:06.914181 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:48:06.910622    5304 cpu_manager.go:414] "containerMap: removing container" podUID="ad1b2ff2-4594-4cf3-88a7-199030bae2a5" containerName="payload-pre-processing"
Nov 02 15:48:06.914181 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:48:06.910631    5304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad1b2ff2-4594-4cf3-88a7-199030bae2a5" containerName="payload-pre-processing"
Nov 02 15:48:06.914181 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:48:06.910650    5304 cpu_manager.go:414] "containerMap: removing container" podUID="783e5ed2-e421-4d98-87e2-6b57a5662b66" containerName="istio-proxy"
Nov 02 15:48:06.914181 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:48:06.910657    5304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="783e5ed2-e421-4d98-87e2-6b57a5662b66" containerName="istio-proxy"
Nov 02 15:48:06.914181 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:48:06.910789    5304 memory_manager.go:365] "RemoveStaleState removing state" podUID="783e5ed2-e421-4d98-87e2-6b57a5662b66" containerName="istio-proxy"
Nov 02 15:48:06.914181 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:48:06.910802    5304 memory_manager.go:365] "RemoveStaleState removing state" podUID="588da214-2720-47e6-b537-f90977db6a92" containerName="payload-processing"
Nov 02 15:48:06.914181 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:48:06.910816    5304 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad1b2ff2-4594-4cf3-88a7-199030bae2a5" containerName="payload-pre-processing"
Nov 02 15:48:06.931240 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:48:06.927898    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-24255b-84c779c487-bnkpw"
Nov 02 15:48:06.950499 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:48:06.950465    5304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-24255b-dockercfg-7x42d\""
Nov 02 15:48:06.950731 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:48:06.950653    5304 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-24255b\""
Nov 02 15:48:06.955947 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:48:06.955074    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-24255b-84c779c487-bnkpw"]
Nov 02 15:48:07.078690 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:48:07.078638    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30cc7413-f0e9-4734-bcea-2a731bd75f88-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-24255b-84c779c487-bnkpw\" (UID: \"30cc7413-f0e9-4734-bcea-2a731bd75f88\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-24255b-84c779c487-bnkpw"
Nov 02 15:48:07.079045 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:48:07.079017    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qsjn4\" (UniqueName: \"kubernetes.io/projected/30cc7413-f0e9-4734-bcea-2a731bd75f88-kube-api-access-qsjn4\") pod \"payload-pre-processing-e2e-ipp-b-24255b-84c779c487-bnkpw\" (UID: \"30cc7413-f0e9-4734-bcea-2a731bd75f88\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-24255b-84c779c487-bnkpw"
Nov 02 15:48:07.180619 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:48:07.180536    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30cc7413-f0e9-4734-bcea-2a731bd75f88-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-24255b-84c779c487-bnkpw\" (UID: \"30cc7413-f0e9-4734-bcea-2a731bd75f88\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-24255b-84c779c487-bnkpw"
Nov 02 15:48:07.180777 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:48:07.180706    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qsjn4\" (UniqueName: \"kubernetes.io/projected/30cc7413-f0e9-4734-bcea-2a731bd75f88-kube-api-access-qsjn4\") pod \"payload-pre-processing-e2e-ipp-b-24255b-84c779c487-bnkpw\" (UID: \"30cc7413-f0e9-4734-bcea-2a731bd75f88\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-24255b-84c779c487-bnkpw"
Nov 02 15:48:07.181269 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:48:07.181248    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30cc7413-f0e9-4734-bcea-2a731bd75f88-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-24255b-84c779c487-bnkpw\" (UID: \"30cc7413-f0e9-4734-bcea-2a731bd75f88\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-24255b-84c779c487-bnkpw"
Nov 02 15:48:07.204714 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:48:07.204680    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qsjn4\" (UniqueName: \"kubernetes.io/projected/30cc7413-f0e9-4734-bcea-2a731bd75f88-kube-api-access-qsjn4\") pod \"payload-pre-processing-e2e-ipp-b-24255b-84c779c487-bnkpw\" (UID: \"30cc7413-f0e9-4734-bcea-2a731bd75f88\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-24255b-84c779c487-bnkpw"
Nov 02 15:48:07.280829 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:48:07.280793    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-24255b-84c779c487-bnkpw"
Nov 02 15:48:07.586657 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:48:07.586542    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-24255b-84c779c487-bnkpw"]
Nov 02 15:48:07.589631 ip-10-0-135-138 kubenswrapper[5304]: W1102 15:48:07.589600    5304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30cc7413_f0e9_4734_bcea_2a731bd75f88.slice/crio-d8ee8a80f543a6b81a5dd5fd5b79fcd8634cb05de62a418bd53e77166cad68a9 WatchSource:0}: Error finding container d8ee8a80f543a6b81a5dd5fd5b79fcd8634cb05de62a418bd53e77166cad68a9: Status 404 returned error can't find the container with id d8ee8a80f543a6b81a5dd5fd5b79fcd8634cb05de62a418bd53e77166cad68a9
Nov 02 15:48:08.119437 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:48:08.119399    5304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="588da214-2720-47e6-b537-f90977db6a92" path="/var/lib/kubelet/pods/588da214-2720-47e6-b537-f90977db6a92/volumes"
Nov 02 15:48:08.119850 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:48:08.119837    5304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ad1b2ff2-4594-4cf3-88a7-199030bae2a5" path="/var/lib/kubelet/pods/ad1b2ff2-4594-4cf3-88a7-199030bae2a5/volumes"
Nov 02 15:48:08.472954 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:48:08.472859    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-24255b-84c779c487-bnkpw" event={"ID":"30cc7413-f0e9-4734-bcea-2a731bd75f88","Type":"ContainerStarted","Data":"4f6b1cbe90d12222a64d90c99a9785a9b4ee7f4659d190a5cf3bb63858b90a39"}
Nov 02 15:48:08.472954 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:48:08.472947    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-24255b-84c779c487-bnkpw" event={"ID":"30cc7413-f0e9-4734-bcea-2a731bd75f88","Type":"ContainerStarted","Data":"d8ee8a80f543a6b81a5dd5fd5b79fcd8634cb05de62a418bd53e77166cad68a9"}
Nov 02 15:48:08.473196 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:48:08.472976    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-24255b-84c779c487-bnkpw"
Nov 02 15:48:08.522728 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:48:08.522672    5304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-24255b-84c779c487-bnkpw" podStartSLOduration=2.522657265 podStartE2EDuration="2.522657265s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:48:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:48:08.519748307 +0000 UTC m=+3088.513572716" watchObservedRunningTime="2026-11-02 15:48:08.522657265 +0000 UTC m=+3088.516481708"
Nov 02 15:48:16.204017 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:48:16.203975    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-b55da1-7757f8c8bc-l8dpc" podUID="1ec89e8e-8ec5-497f-b3ad-1570c6d0d3dd" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 15:48:18.195991 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:48:18.195960    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-b55da1-7757f8c8bc-l8dpc"
Nov 02 15:48:18.278504 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:48:18.278469    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1ec89e8e-8ec5-497f-b3ad-1570c6d0d3dd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ec89e8e-8ec5-497f-b3ad-1570c6d0d3dd-plugins-config-volume\") pod \"1ec89e8e-8ec5-497f-b3ad-1570c6d0d3dd\" (UID: \"1ec89e8e-8ec5-497f-b3ad-1570c6d0d3dd\") "
Nov 02 15:48:18.278709 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:48:18.278604    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ec89e8e-8ec5-497f-b3ad-1570c6d0d3dd-kube-api-access-j95fz\" (UniqueName: \"kubernetes.io/projected/1ec89e8e-8ec5-497f-b3ad-1570c6d0d3dd-kube-api-access-j95fz\") pod \"1ec89e8e-8ec5-497f-b3ad-1570c6d0d3dd\" (UID: \"1ec89e8e-8ec5-497f-b3ad-1570c6d0d3dd\") "
Nov 02 15:48:18.278925 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:48:18.278898    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ec89e8e-8ec5-497f-b3ad-1570c6d0d3dd-plugins-config-volume" pod "1ec89e8e-8ec5-497f-b3ad-1570c6d0d3dd" (UID: "1ec89e8e-8ec5-497f-b3ad-1570c6d0d3dd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:48:18.279213 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:48:18.279193    5304 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ec89e8e-8ec5-497f-b3ad-1570c6d0d3dd-plugins-config-volume\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:48:18.285371 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:48:18.285309    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ec89e8e-8ec5-497f-b3ad-1570c6d0d3dd-kube-api-access-j95fz" pod "1ec89e8e-8ec5-497f-b3ad-1570c6d0d3dd" (UID: "1ec89e8e-8ec5-497f-b3ad-1570c6d0d3dd"). InnerVolumeSpecName "kube-api-access-j95fz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:48:18.403152 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:48:18.402186    5304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j95fz\" (UniqueName: \"kubernetes.io/projected/1ec89e8e-8ec5-497f-b3ad-1570c6d0d3dd-kube-api-access-j95fz\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:48:18.570203 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:48:18.570104    5304 generic.go:356] "Generic (PLEG): container finished" podID="1ec89e8e-8ec5-497f-b3ad-1570c6d0d3dd" containerID="9eb97d07737b81346510341054528b102c06371546d047c73cdca7b1186e881f" exitCode=0
Nov 02 15:48:18.570203 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:48:18.570192    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-b55da1-7757f8c8bc-l8dpc"
Nov 02 15:48:18.570415 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:48:18.570201    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-b55da1-7757f8c8bc-l8dpc" event={"ID":"1ec89e8e-8ec5-497f-b3ad-1570c6d0d3dd","Type":"ContainerDied","Data":"9eb97d07737b81346510341054528b102c06371546d047c73cdca7b1186e881f"}
Nov 02 15:48:18.570415 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:48:18.570259    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-b55da1-7757f8c8bc-l8dpc" event={"ID":"1ec89e8e-8ec5-497f-b3ad-1570c6d0d3dd","Type":"ContainerDied","Data":"62bd5e3dde0b4cb4588504469b7943bde27c5aa33b10f6a142099c91966798a8"}
Nov 02 15:48:18.570415 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:48:18.570285    5304 scope.go:122] "RemoveContainer" containerID="9eb97d07737b81346510341054528b102c06371546d047c73cdca7b1186e881f"
Nov 02 15:48:18.581489 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:48:18.581466    5304 scope.go:122] "RemoveContainer" containerID="9eb97d07737b81346510341054528b102c06371546d047c73cdca7b1186e881f"
Nov 02 15:48:18.581822 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:48:18.581801    5304 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9eb97d07737b81346510341054528b102c06371546d047c73cdca7b1186e881f\": container with ID starting with 9eb97d07737b81346510341054528b102c06371546d047c73cdca7b1186e881f not found: ID does not exist" containerID="9eb97d07737b81346510341054528b102c06371546d047c73cdca7b1186e881f"
Nov 02 15:48:18.581899 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:48:18.581831    5304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9eb97d07737b81346510341054528b102c06371546d047c73cdca7b1186e881f"} err="failed to get container status \"9eb97d07737b81346510341054528b102c06371546d047c73cdca7b1186e881f\": rpc error: code = NotFound desc = could not find container \"9eb97d07737b81346510341054528b102c06371546d047c73cdca7b1186e881f\": container with ID starting with 9eb97d07737b81346510341054528b102c06371546d047c73cdca7b1186e881f not found: ID does not exist"
Nov 02 15:48:18.624426 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:48:18.624394    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-b55da1-7757f8c8bc-l8dpc"]
Nov 02 15:48:18.636234 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:48:18.636201    5304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-b55da1-7757f8c8bc-l8dpc"]
Nov 02 15:48:19.502536 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:48:19.502509    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-24255b-84c779c487-bnkpw"
Nov 02 15:48:20.119524 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:48:20.119491    5304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ec89e8e-8ec5-497f-b3ad-1570c6d0d3dd" path="/var/lib/kubelet/pods/1ec89e8e-8ec5-497f-b3ad-1570c6d0d3dd/volumes"
Nov 02 15:48:20.799677 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:48:20.799638    5304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f0039c-5597f65b87-tqz8b"]
Nov 02 15:48:20.820413 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:48:20.820383    5304 cpu_manager.go:414] "containerMap: removing container" podUID="1ec89e8e-8ec5-497f-b3ad-1570c6d0d3dd" containerName="payload-processing"
Nov 02 15:48:20.820413 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:48:20.820409    5304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ec89e8e-8ec5-497f-b3ad-1570c6d0d3dd" containerName="payload-processing"
Nov 02 15:48:20.820835 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:48:20.820816    5304 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ec89e8e-8ec5-497f-b3ad-1570c6d0d3dd" containerName="payload-processing"
Nov 02 15:48:20.828401 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:48:20.828373    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f0039c-5597f65b87-tqz8b"]
Nov 02 15:48:20.828710 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:48:20.828656    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f0039c-5597f65b87-tqz8b"
Nov 02 15:48:20.856164 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:48:20.856139    5304 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-f0039c\""
Nov 02 15:48:20.856345 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:48:20.856225    5304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-f0039c-dockercfg-d8lkn\""
Nov 02 15:48:20.941611 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:48:20.941575    5304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f0039c-ff4996964-9pzcn"]
Nov 02 15:48:20.943944 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:48:20.943922    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d35d4668-8376-487a-9c91-8b6801eb9cc6-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-f0039c-5597f65b87-tqz8b\" (UID: \"d35d4668-8376-487a-9c91-8b6801eb9cc6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f0039c-5597f65b87-tqz8b"
Nov 02 15:48:20.944235 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:48:20.944206    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8pkxw\" (UniqueName: \"kubernetes.io/projected/d35d4668-8376-487a-9c91-8b6801eb9cc6-kube-api-access-8pkxw\") pod \"payload-pre-processing-e2e-ipp-a-f0039c-5597f65b87-tqz8b\" (UID: \"d35d4668-8376-487a-9c91-8b6801eb9cc6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f0039c-5597f65b87-tqz8b"
Nov 02 15:48:20.944548 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:48:20.944534    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-f0039c-ff4996964-9pzcn"
Nov 02 15:48:20.966657 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:48:20.966627    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f0039c-ff4996964-9pzcn"]
Nov 02 15:48:21.045976 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:48:21.045939    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8pkxw\" (UniqueName: \"kubernetes.io/projected/d35d4668-8376-487a-9c91-8b6801eb9cc6-kube-api-access-8pkxw\") pod \"payload-pre-processing-e2e-ipp-a-f0039c-5597f65b87-tqz8b\" (UID: \"d35d4668-8376-487a-9c91-8b6801eb9cc6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f0039c-5597f65b87-tqz8b"
Nov 02 15:48:21.046230 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:48:21.046010    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d35d4668-8376-487a-9c91-8b6801eb9cc6-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-f0039c-5597f65b87-tqz8b\" (UID: \"d35d4668-8376-487a-9c91-8b6801eb9cc6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f0039c-5597f65b87-tqz8b"
Nov 02 15:48:21.046230 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:48:21.046064    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43c5260b-b089-4c06-ac76-e6e170cac754-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-f0039c-ff4996964-9pzcn\" (UID: \"43c5260b-b089-4c06-ac76-e6e170cac754\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f0039c-ff4996964-9pzcn"
Nov 02 15:48:21.046230 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:48:21.046148    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p4dl8\" (UniqueName: \"kubernetes.io/projected/43c5260b-b089-4c06-ac76-e6e170cac754-kube-api-access-p4dl8\") pod \"payload-processing-e2e-ipp-a-f0039c-ff4996964-9pzcn\" (UID: \"43c5260b-b089-4c06-ac76-e6e170cac754\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f0039c-ff4996964-9pzcn"
Nov 02 15:48:21.046758 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:48:21.046738    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d35d4668-8376-487a-9c91-8b6801eb9cc6-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-f0039c-5597f65b87-tqz8b\" (UID: \"d35d4668-8376-487a-9c91-8b6801eb9cc6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f0039c-5597f65b87-tqz8b"
Nov 02 15:48:21.069250 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:48:21.069188    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8pkxw\" (UniqueName: \"kubernetes.io/projected/d35d4668-8376-487a-9c91-8b6801eb9cc6-kube-api-access-8pkxw\") pod \"payload-pre-processing-e2e-ipp-a-f0039c-5597f65b87-tqz8b\" (UID: \"d35d4668-8376-487a-9c91-8b6801eb9cc6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f0039c-5597f65b87-tqz8b"
Nov 02 15:48:21.147268 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:48:21.147235    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p4dl8\" (UniqueName: \"kubernetes.io/projected/43c5260b-b089-4c06-ac76-e6e170cac754-kube-api-access-p4dl8\") pod \"payload-processing-e2e-ipp-a-f0039c-ff4996964-9pzcn\" (UID: \"43c5260b-b089-4c06-ac76-e6e170cac754\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f0039c-ff4996964-9pzcn"
Nov 02 15:48:21.147461 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:48:21.147418    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43c5260b-b089-4c06-ac76-e6e170cac754-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-f0039c-ff4996964-9pzcn\" (UID: \"43c5260b-b089-4c06-ac76-e6e170cac754\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f0039c-ff4996964-9pzcn"
Nov 02 15:48:21.148101 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:48:21.148079    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43c5260b-b089-4c06-ac76-e6e170cac754-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-f0039c-ff4996964-9pzcn\" (UID: \"43c5260b-b089-4c06-ac76-e6e170cac754\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f0039c-ff4996964-9pzcn"
Nov 02 15:48:21.155759 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:48:21.155737    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f0039c-5597f65b87-tqz8b"
Nov 02 15:48:21.171019 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:48:21.170995    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p4dl8\" (UniqueName: \"kubernetes.io/projected/43c5260b-b089-4c06-ac76-e6e170cac754-kube-api-access-p4dl8\") pod \"payload-processing-e2e-ipp-a-f0039c-ff4996964-9pzcn\" (UID: \"43c5260b-b089-4c06-ac76-e6e170cac754\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f0039c-ff4996964-9pzcn"
Nov 02 15:48:21.257003 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:48:21.256948    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-f0039c-ff4996964-9pzcn"
Nov 02 15:48:21.479155 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:48:21.479108    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f0039c-5597f65b87-tqz8b"]
Nov 02 15:48:21.481727 ip-10-0-135-138 kubenswrapper[5304]: W1102 15:48:21.481697    5304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd35d4668_8376_487a_9c91_8b6801eb9cc6.slice/crio-2118d95c6fc5095d52ca8a55cb216bdc9b4525c1c6f048e647b089fcbaf27336 WatchSource:0}: Error finding container 2118d95c6fc5095d52ca8a55cb216bdc9b4525c1c6f048e647b089fcbaf27336: Status 404 returned error can't find the container with id 2118d95c6fc5095d52ca8a55cb216bdc9b4525c1c6f048e647b089fcbaf27336
Nov 02 15:48:21.579945 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:48:21.579917    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f0039c-ff4996964-9pzcn"]
Nov 02 15:48:21.583055 ip-10-0-135-138 kubenswrapper[5304]: W1102 15:48:21.583032    5304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43c5260b_b089_4c06_ac76_e6e170cac754.slice/crio-61f396b65ff959727476bfd4399eb5f80a014ccf1ea1dc4b9ab593ea489d50a5 WatchSource:0}: Error finding container 61f396b65ff959727476bfd4399eb5f80a014ccf1ea1dc4b9ab593ea489d50a5: Status 404 returned error can't find the container with id 61f396b65ff959727476bfd4399eb5f80a014ccf1ea1dc4b9ab593ea489d50a5
Nov 02 15:48:21.598372 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:48:21.598334    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f0039c-5597f65b87-tqz8b" event={"ID":"d35d4668-8376-487a-9c91-8b6801eb9cc6","Type":"ContainerStarted","Data":"cf002bb8f28bd1c2b000f567c7eafb6aa6b554b5231e6d516ba5b096c60275b4"}
Nov 02 15:48:21.598512 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:48:21.598394    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f0039c-5597f65b87-tqz8b" event={"ID":"d35d4668-8376-487a-9c91-8b6801eb9cc6","Type":"ContainerStarted","Data":"2118d95c6fc5095d52ca8a55cb216bdc9b4525c1c6f048e647b089fcbaf27336"}
Nov 02 15:48:21.598512 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:48:21.598437    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f0039c-5597f65b87-tqz8b"
Nov 02 15:48:21.600570 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:48:21.600543    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-f0039c-ff4996964-9pzcn" event={"ID":"43c5260b-b089-4c06-ac76-e6e170cac754","Type":"ContainerStarted","Data":"61f396b65ff959727476bfd4399eb5f80a014ccf1ea1dc4b9ab593ea489d50a5"}
Nov 02 15:48:21.640703 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:48:21.640657    5304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f0039c-5597f65b87-tqz8b" podStartSLOduration=1.6406418550000001 podStartE2EDuration="1.640641855s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:48:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:48:21.638280924 +0000 UTC m=+3101.632105324" watchObservedRunningTime="2026-11-02 15:48:21.640641855 +0000 UTC m=+3101.634466267"
Nov 02 15:48:22.611684 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:48:22.611646    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-f0039c-ff4996964-9pzcn" event={"ID":"43c5260b-b089-4c06-ac76-e6e170cac754","Type":"ContainerStarted","Data":"0b61a4b3ffd0614509c9c252c532a59aea8ad299ff31a9d866ae255a83d9d02d"}
Nov 02 15:48:22.654509 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:48:22.654442    5304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-f0039c-ff4996964-9pzcn" podStartSLOduration=2.6544229010000002 podStartE2EDuration="2.654422901s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:48:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:48:22.65285629 +0000 UTC m=+3102.646680700" watchObservedRunningTime="2026-11-02 15:48:22.654422901 +0000 UTC m=+3102.648247311"
Nov 02 15:48:23.618859 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:48:23.618828    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-f0039c-ff4996964-9pzcn"
Nov 02 15:48:24.804218 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:48:24.804180    5304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k6fjk" containerName="extractor"
Nov 02 15:48:30.259152 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:48:30.256077    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-b55da1-687f46949-z4mjq"]
Nov 02 15:48:30.259152 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:48:30.256371    5304 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-b55da1-687f46949-z4mjq" podUID="7c093afd-5109-4891-a40c-868a053da719" containerName="maas-api" containerID="cri-o://32f42b3c511838439227acf76e715246eddff1c27695e530f66076e2523648d0" gracePeriod=30
Nov 02 15:48:30.628827 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:48:30.628799    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-b55da1-687f46949-z4mjq"
Nov 02 15:48:30.667742 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:48:30.667701    5304 generic.go:356] "Generic (PLEG): container finished" podID="7c093afd-5109-4891-a40c-868a053da719" containerID="32f42b3c511838439227acf76e715246eddff1c27695e530f66076e2523648d0" exitCode=0
Nov 02 15:48:30.667952 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:48:30.667800    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-b55da1-687f46949-z4mjq" event={"ID":"7c093afd-5109-4891-a40c-868a053da719","Type":"ContainerDied","Data":"32f42b3c511838439227acf76e715246eddff1c27695e530f66076e2523648d0"}
Nov 02 15:48:30.667952 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:48:30.667852    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-b55da1-687f46949-z4mjq" event={"ID":"7c093afd-5109-4891-a40c-868a053da719","Type":"ContainerDied","Data":"13ca995481f3e26b142a145d40807892a880f6567ee1890d049267949b9f2c72"}
Nov 02 15:48:30.667952 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:48:30.667882    5304 scope.go:122] "RemoveContainer" containerID="32f42b3c511838439227acf76e715246eddff1c27695e530f66076e2523648d0"
Nov 02 15:48:30.668152 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:48:30.668048    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-b55da1-687f46949-z4mjq"
Nov 02 15:48:30.699471 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:48:30.695347    5304 scope.go:122] "RemoveContainer" containerID="32f42b3c511838439227acf76e715246eddff1c27695e530f66076e2523648d0"
Nov 02 15:48:30.701637 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:48:30.701558    5304 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"32f42b3c511838439227acf76e715246eddff1c27695e530f66076e2523648d0\": container with ID starting with 32f42b3c511838439227acf76e715246eddff1c27695e530f66076e2523648d0 not found: ID does not exist" containerID="32f42b3c511838439227acf76e715246eddff1c27695e530f66076e2523648d0"
Nov 02 15:48:30.701637 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:48:30.701607    5304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32f42b3c511838439227acf76e715246eddff1c27695e530f66076e2523648d0"} err="failed to get container status \"32f42b3c511838439227acf76e715246eddff1c27695e530f66076e2523648d0\": rpc error: code = NotFound desc = could not find container \"32f42b3c511838439227acf76e715246eddff1c27695e530f66076e2523648d0\": container with ID starting with 32f42b3c511838439227acf76e715246eddff1c27695e530f66076e2523648d0 not found: ID does not exist"
Nov 02 15:48:30.850372 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:48:30.850285    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7c093afd-5109-4891-a40c-868a053da719-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7c093afd-5109-4891-a40c-868a053da719-maas-api-tls\") pod \"7c093afd-5109-4891-a40c-868a053da719\" (UID: \"7c093afd-5109-4891-a40c-868a053da719\") "
Nov 02 15:48:30.850553 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:48:30.850487    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c093afd-5109-4891-a40c-868a053da719-kube-api-access-m8wxw\" (UniqueName: \"kubernetes.io/projected/7c093afd-5109-4891-a40c-868a053da719-kube-api-access-m8wxw\") pod \"7c093afd-5109-4891-a40c-868a053da719\" (UID: \"7c093afd-5109-4891-a40c-868a053da719\") "
Nov 02 15:48:30.850643 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:48:30.850611    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7c093afd-5109-4891-a40c-868a053da719-metrics-certs\" (UniqueName: \"kubernetes.io/secret/7c093afd-5109-4891-a40c-868a053da719-metrics-certs\") pod \"7c093afd-5109-4891-a40c-868a053da719\" (UID: \"7c093afd-5109-4891-a40c-868a053da719\") "
Nov 02 15:48:30.850775 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:48:30.850760    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7c093afd-5109-4891-a40c-868a053da719-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c093afd-5109-4891-a40c-868a053da719-openshift-service-ca\") pod \"7c093afd-5109-4891-a40c-868a053da719\" (UID: \"7c093afd-5109-4891-a40c-868a053da719\") "
Nov 02 15:48:30.851178 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:48:30.851145    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c093afd-5109-4891-a40c-868a053da719-openshift-service-ca" pod "7c093afd-5109-4891-a40c-868a053da719" (UID: "7c093afd-5109-4891-a40c-868a053da719"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:48:30.851358 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:48:30.851341    5304 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c093afd-5109-4891-a40c-868a053da719-openshift-service-ca\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:48:30.852722 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:48:30.852691    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7c093afd-5109-4891-a40c-868a053da719-maas-api-tls" pod "7c093afd-5109-4891-a40c-868a053da719" (UID: "7c093afd-5109-4891-a40c-868a053da719"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:48:30.852722 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:48:30.852711    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7c093afd-5109-4891-a40c-868a053da719-metrics-certs" pod "7c093afd-5109-4891-a40c-868a053da719" (UID: "7c093afd-5109-4891-a40c-868a053da719"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:48:30.852875 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:48:30.852734    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c093afd-5109-4891-a40c-868a053da719-kube-api-access-m8wxw" pod "7c093afd-5109-4891-a40c-868a053da719" (UID: "7c093afd-5109-4891-a40c-868a053da719"). InnerVolumeSpecName "kube-api-access-m8wxw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:48:30.953396 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:48:30.953356    5304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m8wxw\" (UniqueName: \"kubernetes.io/projected/7c093afd-5109-4891-a40c-868a053da719-kube-api-access-m8wxw\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:48:30.953648 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:48:30.953627    5304 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7c093afd-5109-4891-a40c-868a053da719-metrics-certs\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:48:30.953771 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:48:30.953760    5304 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7c093afd-5109-4891-a40c-868a053da719-maas-api-tls\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:48:31.031555 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:48:31.031514    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-b55da1-687f46949-z4mjq"]
Nov 02 15:48:31.038109 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:48:31.038079    5304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-b55da1-687f46949-z4mjq"]
Nov 02 15:48:32.119886 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:48:32.119849    5304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7c093afd-5109-4891-a40c-868a053da719" path="/var/lib/kubelet/pods/7c093afd-5109-4891-a40c-868a053da719/volumes"
Nov 02 15:48:32.614346 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:48:32.614314    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f0039c-5597f65b87-tqz8b"
Nov 02 15:48:34.627694 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:48:34.627660    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-f0039c-ff4996964-9pzcn"
Nov 02 15:48:37.775149 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:48:37.771067    5304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-d79991-6d4b6d8c48-ptv5r"]
Nov 02 15:48:37.775149 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:48:37.771976    5304 cpu_manager.go:414] "containerMap: removing container" podUID="7c093afd-5109-4891-a40c-868a053da719" containerName="maas-api"
Nov 02 15:48:37.775149 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:48:37.772000    5304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c093afd-5109-4891-a40c-868a053da719" containerName="maas-api"
Nov 02 15:48:37.775149 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:48:37.772240    5304 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c093afd-5109-4891-a40c-868a053da719" containerName="maas-api"
Nov 02 15:48:37.787187 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:48:37.776892    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-d79991-6d4b6d8c48-ptv5r"
Nov 02 15:48:37.802070 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:48:37.802038    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-d79991-6d4b6d8c48-ptv5r"]
Nov 02 15:48:37.804723 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:48:37.804696    5304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-d79991-dockercfg-6z7pb\""
Nov 02 15:48:37.816264 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:48:37.816230    5304 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-d79991\""
Nov 02 15:48:37.876979 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:48:37.876620    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe452842-bf9b-4129-a1dc-4a5b98d2672f-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-d79991-6d4b6d8c48-ptv5r\" (UID: \"fe452842-bf9b-4129-a1dc-4a5b98d2672f\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-d79991-6d4b6d8c48-ptv5r"
Nov 02 15:48:37.876979 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:48:37.876703    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mjtcm\" (UniqueName: \"kubernetes.io/projected/fe452842-bf9b-4129-a1dc-4a5b98d2672f-kube-api-access-mjtcm\") pod \"payload-processing-e2e-ipp-a-d79991-6d4b6d8c48-ptv5r\" (UID: \"fe452842-bf9b-4129-a1dc-4a5b98d2672f\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-d79991-6d4b6d8c48-ptv5r"
Nov 02 15:48:37.977723 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:48:37.977686    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe452842-bf9b-4129-a1dc-4a5b98d2672f-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-d79991-6d4b6d8c48-ptv5r\" (UID: \"fe452842-bf9b-4129-a1dc-4a5b98d2672f\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-d79991-6d4b6d8c48-ptv5r"
Nov 02 15:48:37.977950 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:48:37.977750    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mjtcm\" (UniqueName: \"kubernetes.io/projected/fe452842-bf9b-4129-a1dc-4a5b98d2672f-kube-api-access-mjtcm\") pod \"payload-processing-e2e-ipp-a-d79991-6d4b6d8c48-ptv5r\" (UID: \"fe452842-bf9b-4129-a1dc-4a5b98d2672f\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-d79991-6d4b6d8c48-ptv5r"
Nov 02 15:48:37.978476 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:48:37.978452    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe452842-bf9b-4129-a1dc-4a5b98d2672f-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-d79991-6d4b6d8c48-ptv5r\" (UID: \"fe452842-bf9b-4129-a1dc-4a5b98d2672f\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-d79991-6d4b6d8c48-ptv5r"
Nov 02 15:48:38.005495 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:48:38.005458    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mjtcm\" (UniqueName: \"kubernetes.io/projected/fe452842-bf9b-4129-a1dc-4a5b98d2672f-kube-api-access-mjtcm\") pod \"payload-processing-e2e-ipp-a-d79991-6d4b6d8c48-ptv5r\" (UID: \"fe452842-bf9b-4129-a1dc-4a5b98d2672f\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-d79991-6d4b6d8c48-ptv5r"
Nov 02 15:48:38.106713 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:48:38.106676    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-d79991-6d4b6d8c48-ptv5r"
Nov 02 15:48:38.811714 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:48:38.811687    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-d79991-6d4b6d8c48-ptv5r"]
Nov 02 15:48:38.827818 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:48:38.827780    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-d79991-6d4b6d8c48-ptv5r" event={"ID":"fe452842-bf9b-4129-a1dc-4a5b98d2672f","Type":"ContainerStarted","Data":"120da66418d003200617c06ae7eea5dca3611719993f1b35433a24b24cddc00c"}
Nov 02 15:48:39.115120 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:48:39.114035    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-b55da1-openshift-default-787ff9f87c-cmxhv"]
Nov 02 15:48:39.115120 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:48:39.114253    5304 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w6-b55da1-openshift-default-787ff9f87c-cmxhv" podUID="67da2b58-a0f0-44f6-9258-b418a21e73bc" containerName="istio-proxy" containerID="cri-o://efa3a256bfcda39d706219796dc57612df582f4045ec78771cb86c6cd80eb067" gracePeriod=30
Nov 02 15:48:39.427205 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:48:39.423048    5304 patch_prober.go:29] interesting pod/e2e-shared-b-w6-b55da1-openshift-default-787ff9f87c-cmxhv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 15:48:39.427205 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:48:39.423106    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w6-b55da1-openshift-default-787ff9f87c-cmxhv" podUID="67da2b58-a0f0-44f6-9258-b418a21e73bc" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 15:48:39.870893 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:48:39.870855    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-d79991-6d4b6d8c48-ptv5r" event={"ID":"fe452842-bf9b-4129-a1dc-4a5b98d2672f","Type":"ContainerStarted","Data":"01aa731f1b95615f39e11bf514b6c741f59446ce23c18bb0334572291fab4459"}
Nov 02 15:48:39.871438 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:48:39.871215    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-d79991-6d4b6d8c48-ptv5r"
Nov 02 15:48:39.915026 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:48:39.914961    5304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-d79991-6d4b6d8c48-ptv5r" podStartSLOduration=2.9149412789999998 podStartE2EDuration="2.914941279s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:48:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:48:39.912798133 +0000 UTC m=+3119.906622543" watchObservedRunningTime="2026-11-02 15:48:39.914941279 +0000 UTC m=+3119.908765689"
Nov 02 15:48:42.687179 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:48:42.683777    5304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-8537e9-b666465c7-58z69"]
Nov 02 15:48:42.771151 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:48:42.760829    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-8537e9-b666465c7-58z69"]
Nov 02 15:48:42.771151 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:48:42.760992    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-8537e9-b666465c7-58z69"
Nov 02 15:48:42.814381 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:48:42.814348    5304 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-8537e9\""
Nov 02 15:48:42.814740 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:48:42.814725    5304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-8537e9-dockercfg-f8f6x\""
Nov 02 15:48:42.854266 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:48:42.854211    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f366f832-d3cd-402e-875c-a23629f7c347-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-8537e9-b666465c7-58z69\" (UID: \"f366f832-d3cd-402e-875c-a23629f7c347\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-8537e9-b666465c7-58z69"
Nov 02 15:48:42.854459 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:48:42.854414    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2h9tz\" (UniqueName: \"kubernetes.io/projected/f366f832-d3cd-402e-875c-a23629f7c347-kube-api-access-2h9tz\") pod \"payload-processing-e2e-ipp-b-8537e9-b666465c7-58z69\" (UID: \"f366f832-d3cd-402e-875c-a23629f7c347\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-8537e9-b666465c7-58z69"
Nov 02 15:48:42.956180 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:48:42.956036    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f366f832-d3cd-402e-875c-a23629f7c347-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-8537e9-b666465c7-58z69\" (UID: \"f366f832-d3cd-402e-875c-a23629f7c347\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-8537e9-b666465c7-58z69"
Nov 02 15:48:42.956606 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:48:42.956574    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2h9tz\" (UniqueName: \"kubernetes.io/projected/f366f832-d3cd-402e-875c-a23629f7c347-kube-api-access-2h9tz\") pod \"payload-processing-e2e-ipp-b-8537e9-b666465c7-58z69\" (UID: \"f366f832-d3cd-402e-875c-a23629f7c347\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-8537e9-b666465c7-58z69"
Nov 02 15:48:42.957802 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:48:42.957775    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f366f832-d3cd-402e-875c-a23629f7c347-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-8537e9-b666465c7-58z69\" (UID: \"f366f832-d3cd-402e-875c-a23629f7c347\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-8537e9-b666465c7-58z69"
Nov 02 15:48:42.980047 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:48:42.980019    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2h9tz\" (UniqueName: \"kubernetes.io/projected/f366f832-d3cd-402e-875c-a23629f7c347-kube-api-access-2h9tz\") pod \"payload-processing-e2e-ipp-b-8537e9-b666465c7-58z69\" (UID: \"f366f832-d3cd-402e-875c-a23629f7c347\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-8537e9-b666465c7-58z69"
Nov 02 15:48:43.127730 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:48:43.127694    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-8537e9-b666465c7-58z69"
Nov 02 15:48:43.466285 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:48:43.466118    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-8537e9-b666465c7-58z69"]
Nov 02 15:48:43.471469 ip-10-0-135-138 kubenswrapper[5304]: W1102 15:48:43.471443    5304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf366f832_d3cd_402e_875c_a23629f7c347.slice/crio-77c4fcb92c80da4394f8ea0c36784a45841bbb9401291b4dcb7cf5b8484ba778 WatchSource:0}: Error finding container 77c4fcb92c80da4394f8ea0c36784a45841bbb9401291b4dcb7cf5b8484ba778: Status 404 returned error can't find the container with id 77c4fcb92c80da4394f8ea0c36784a45841bbb9401291b4dcb7cf5b8484ba778
Nov 02 15:48:43.900667 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:48:43.900632    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-8537e9-b666465c7-58z69" event={"ID":"f366f832-d3cd-402e-875c-a23629f7c347","Type":"ContainerStarted","Data":"66087bdb01289a7b8670c3b983ef04416f9b38c84f5c869b3e8031561b7f414b"}
Nov 02 15:48:43.901086 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:48:43.900686    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-8537e9-b666465c7-58z69" event={"ID":"f366f832-d3cd-402e-875c-a23629f7c347","Type":"ContainerStarted","Data":"77c4fcb92c80da4394f8ea0c36784a45841bbb9401291b4dcb7cf5b8484ba778"}
Nov 02 15:48:43.901086 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:48:43.900766    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-8537e9-b666465c7-58z69"
Nov 02 15:48:43.944242 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:48:43.944172    5304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-8537e9-b666465c7-58z69" podStartSLOduration=1.944152603 podStartE2EDuration="1.944152603s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:48:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:48:43.94108448 +0000 UTC m=+3123.934908888" watchObservedRunningTime="2026-11-02 15:48:43.944152603 +0000 UTC m=+3123.937977016"
Nov 02 15:48:44.406588 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:48:44.406557    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-b55da1-openshift-default-787ff9f87c-cmxhv"
Nov 02 15:48:44.485680 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:48:44.485643    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67da2b58-a0f0-44f6-9258-b418a21e73bc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67da2b58-a0f0-44f6-9258-b418a21e73bc-istio-envoy\") pod \"67da2b58-a0f0-44f6-9258-b418a21e73bc\" (UID: \"67da2b58-a0f0-44f6-9258-b418a21e73bc\") "
Nov 02 15:48:44.485893 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:48:44.485765    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67da2b58-a0f0-44f6-9258-b418a21e73bc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67da2b58-a0f0-44f6-9258-b418a21e73bc-workload-certs\") pod \"67da2b58-a0f0-44f6-9258-b418a21e73bc\" (UID: \"67da2b58-a0f0-44f6-9258-b418a21e73bc\") "
Nov 02 15:48:44.485893 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:48:44.485820    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67da2b58-a0f0-44f6-9258-b418a21e73bc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67da2b58-a0f0-44f6-9258-b418a21e73bc-istio-data\") pod \"67da2b58-a0f0-44f6-9258-b418a21e73bc\" (UID: \"67da2b58-a0f0-44f6-9258-b418a21e73bc\") "
Nov 02 15:48:44.485893 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:48:44.485856    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67da2b58-a0f0-44f6-9258-b418a21e73bc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67da2b58-a0f0-44f6-9258-b418a21e73bc-credential-socket\") pod \"67da2b58-a0f0-44f6-9258-b418a21e73bc\" (UID: \"67da2b58-a0f0-44f6-9258-b418a21e73bc\") "
Nov 02 15:48:44.486077 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:48:44.485962    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/67da2b58-a0f0-44f6-9258-b418a21e73bc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67da2b58-a0f0-44f6-9258-b418a21e73bc-istio-podinfo\") pod \"67da2b58-a0f0-44f6-9258-b418a21e73bc\" (UID: \"67da2b58-a0f0-44f6-9258-b418a21e73bc\") "
Nov 02 15:48:44.486077 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:48:44.486037    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67da2b58-a0f0-44f6-9258-b418a21e73bc-istio-token\" (UniqueName: \"kubernetes.io/projected/67da2b58-a0f0-44f6-9258-b418a21e73bc-istio-token\") pod \"67da2b58-a0f0-44f6-9258-b418a21e73bc\" (UID: \"67da2b58-a0f0-44f6-9258-b418a21e73bc\") "
Nov 02 15:48:44.486247 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:48:44.486080    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/67da2b58-a0f0-44f6-9258-b418a21e73bc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67da2b58-a0f0-44f6-9258-b418a21e73bc-istiod-ca-cert\") pod \"67da2b58-a0f0-44f6-9258-b418a21e73bc\" (UID: \"67da2b58-a0f0-44f6-9258-b418a21e73bc\") "
Nov 02 15:48:44.486247 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:48:44.486118    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67da2b58-a0f0-44f6-9258-b418a21e73bc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67da2b58-a0f0-44f6-9258-b418a21e73bc-workload-socket\") pod \"67da2b58-a0f0-44f6-9258-b418a21e73bc\" (UID: \"67da2b58-a0f0-44f6-9258-b418a21e73bc\") "
Nov 02 15:48:44.486247 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:48:44.486176    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67da2b58-a0f0-44f6-9258-b418a21e73bc-kube-api-access-sr82w\" (UniqueName: \"kubernetes.io/projected/67da2b58-a0f0-44f6-9258-b418a21e73bc-kube-api-access-sr82w\") pod \"67da2b58-a0f0-44f6-9258-b418a21e73bc\" (UID: \"67da2b58-a0f0-44f6-9258-b418a21e73bc\") "
Nov 02 15:48:44.486816 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:48:44.486774    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67da2b58-a0f0-44f6-9258-b418a21e73bc-credential-socket" pod "67da2b58-a0f0-44f6-9258-b418a21e73bc" (UID: "67da2b58-a0f0-44f6-9258-b418a21e73bc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:48:44.489999 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:48:44.489921    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67da2b58-a0f0-44f6-9258-b418a21e73bc-workload-certs" pod "67da2b58-a0f0-44f6-9258-b418a21e73bc" (UID: "67da2b58-a0f0-44f6-9258-b418a21e73bc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:48:44.490502 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:48:44.490469    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67da2b58-a0f0-44f6-9258-b418a21e73bc-istio-data" pod "67da2b58-a0f0-44f6-9258-b418a21e73bc" (UID: "67da2b58-a0f0-44f6-9258-b418a21e73bc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:48:44.490848 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:48:44.490826    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67da2b58-a0f0-44f6-9258-b418a21e73bc-workload-socket" pod "67da2b58-a0f0-44f6-9258-b418a21e73bc" (UID: "67da2b58-a0f0-44f6-9258-b418a21e73bc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:48:44.491075 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:48:44.491053    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67da2b58-a0f0-44f6-9258-b418a21e73bc-kube-api-access-sr82w" pod "67da2b58-a0f0-44f6-9258-b418a21e73bc" (UID: "67da2b58-a0f0-44f6-9258-b418a21e73bc"). InnerVolumeSpecName "kube-api-access-sr82w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:48:44.491207 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:48:44.491086    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67da2b58-a0f0-44f6-9258-b418a21e73bc-istiod-ca-cert" pod "67da2b58-a0f0-44f6-9258-b418a21e73bc" (UID: "67da2b58-a0f0-44f6-9258-b418a21e73bc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:48:44.492078 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:48:44.492055    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67da2b58-a0f0-44f6-9258-b418a21e73bc-istio-envoy" pod "67da2b58-a0f0-44f6-9258-b418a21e73bc" (UID: "67da2b58-a0f0-44f6-9258-b418a21e73bc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:48:44.493367 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:48:44.493331    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/67da2b58-a0f0-44f6-9258-b418a21e73bc-istio-podinfo" pod "67da2b58-a0f0-44f6-9258-b418a21e73bc" (UID: "67da2b58-a0f0-44f6-9258-b418a21e73bc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 15:48:44.494783 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:48:44.494756    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67da2b58-a0f0-44f6-9258-b418a21e73bc-istio-token" pod "67da2b58-a0f0-44f6-9258-b418a21e73bc" (UID: "67da2b58-a0f0-44f6-9258-b418a21e73bc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:48:44.519761 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:48:44.519722    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-5113c2/premium-w6-5113c2-kserve-b84945cb5-pc6f8"]
Nov 02 15:48:44.588094 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:48:44.588054    5304 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67da2b58-a0f0-44f6-9258-b418a21e73bc-istio-podinfo\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:48:44.588094 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:48:44.588091    5304 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67da2b58-a0f0-44f6-9258-b418a21e73bc-istio-token\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:48:44.588401 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:48:44.588107    5304 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67da2b58-a0f0-44f6-9258-b418a21e73bc-istiod-ca-cert\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:48:44.588401 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:48:44.588120    5304 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67da2b58-a0f0-44f6-9258-b418a21e73bc-workload-socket\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:48:44.588401 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:48:44.588171    5304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sr82w\" (UniqueName: \"kubernetes.io/projected/67da2b58-a0f0-44f6-9258-b418a21e73bc-kube-api-access-sr82w\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:48:44.588401 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:48:44.588188    5304 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67da2b58-a0f0-44f6-9258-b418a21e73bc-istio-envoy\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:48:44.588401 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:48:44.588201    5304 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67da2b58-a0f0-44f6-9258-b418a21e73bc-workload-certs\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:48:44.588401 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:48:44.588212    5304 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67da2b58-a0f0-44f6-9258-b418a21e73bc-istio-data\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:48:44.588401 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:48:44.588223    5304 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67da2b58-a0f0-44f6-9258-b418a21e73bc-credential-socket\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:48:44.912534 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:48:44.912488    5304 generic.go:356] "Generic (PLEG): container finished" podID="67da2b58-a0f0-44f6-9258-b418a21e73bc" containerID="efa3a256bfcda39d706219796dc57612df582f4045ec78771cb86c6cd80eb067" exitCode=0
Nov 02 15:48:44.913732 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:48:44.913708    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-b55da1-openshift-default-787ff9f87c-cmxhv"
Nov 02 15:48:44.914329 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:48:44.914186    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-b55da1-openshift-default-787ff9f87c-cmxhv" event={"ID":"67da2b58-a0f0-44f6-9258-b418a21e73bc","Type":"ContainerDied","Data":"efa3a256bfcda39d706219796dc57612df582f4045ec78771cb86c6cd80eb067"}
Nov 02 15:48:44.914456 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:48:44.914377    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-b55da1-openshift-default-787ff9f87c-cmxhv" event={"ID":"67da2b58-a0f0-44f6-9258-b418a21e73bc","Type":"ContainerDied","Data":"a4c4ccc2c238a149676473f8f48d78673a61198c242464fd45d351410b2c1127"}
Nov 02 15:48:44.914456 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:48:44.914413    5304 scope.go:122] "RemoveContainer" containerID="efa3a256bfcda39d706219796dc57612df582f4045ec78771cb86c6cd80eb067"
Nov 02 15:48:44.928197 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:48:44.928080    5304 scope.go:122] "RemoveContainer" containerID="aa7cb5d0093ee870a139ebc974db546ad5b647c091f54f48cddf10d8ddfc7646"
Nov 02 15:48:44.940594 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:48:44.940575    5304 scope.go:122] "RemoveContainer" containerID="efa3a256bfcda39d706219796dc57612df582f4045ec78771cb86c6cd80eb067"
Nov 02 15:48:44.940931 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:48:44.940910    5304 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"efa3a256bfcda39d706219796dc57612df582f4045ec78771cb86c6cd80eb067\": container with ID starting with efa3a256bfcda39d706219796dc57612df582f4045ec78771cb86c6cd80eb067 not found: ID does not exist" containerID="efa3a256bfcda39d706219796dc57612df582f4045ec78771cb86c6cd80eb067"
Nov 02 15:48:44.941015 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:48:44.940939    5304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"efa3a256bfcda39d706219796dc57612df582f4045ec78771cb86c6cd80eb067"} err="failed to get container status \"efa3a256bfcda39d706219796dc57612df582f4045ec78771cb86c6cd80eb067\": rpc error: code = NotFound desc = could not find container \"efa3a256bfcda39d706219796dc57612df582f4045ec78771cb86c6cd80eb067\": container with ID starting with efa3a256bfcda39d706219796dc57612df582f4045ec78771cb86c6cd80eb067 not found: ID does not exist"
Nov 02 15:48:44.941015 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:48:44.940956    5304 scope.go:122] "RemoveContainer" containerID="aa7cb5d0093ee870a139ebc974db546ad5b647c091f54f48cddf10d8ddfc7646"
Nov 02 15:48:44.941378 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:48:44.941355    5304 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa7cb5d0093ee870a139ebc974db546ad5b647c091f54f48cddf10d8ddfc7646\": container with ID starting with aa7cb5d0093ee870a139ebc974db546ad5b647c091f54f48cddf10d8ddfc7646 not found: ID does not exist" containerID="aa7cb5d0093ee870a139ebc974db546ad5b647c091f54f48cddf10d8ddfc7646"
Nov 02 15:48:44.941475 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:48:44.941384    5304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa7cb5d0093ee870a139ebc974db546ad5b647c091f54f48cddf10d8ddfc7646"} err="failed to get container status \"aa7cb5d0093ee870a139ebc974db546ad5b647c091f54f48cddf10d8ddfc7646\": rpc error: code = NotFound desc = could not find container \"aa7cb5d0093ee870a139ebc974db546ad5b647c091f54f48cddf10d8ddfc7646\": container with ID starting with aa7cb5d0093ee870a139ebc974db546ad5b647c091f54f48cddf10d8ddfc7646 not found: ID does not exist"
Nov 02 15:48:44.970738 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:48:44.970700    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-b55da1-openshift-default-787ff9f87c-cmxhv"]
Nov 02 15:48:44.982263 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:48:44.982232    5304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-b55da1-openshift-default-787ff9f87c-cmxhv"]
Nov 02 15:48:46.118771 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:48:46.118737    5304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="67da2b58-a0f0-44f6-9258-b418a21e73bc" path="/var/lib/kubelet/pods/67da2b58-a0f0-44f6-9258-b418a21e73bc/volumes"
Nov 02 15:48:47.940821 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:48:47.940787    5304 generic.go:356] "Generic (PLEG): container finished" podID="1020808c-e760-46e7-99e8-66932fca1956" containerID="554b43614667dd9cb4b1f94d5281367be991545f9ad45480363d8879d52c5001" exitCode=0
Nov 02 15:48:47.941308 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:48:47.940858    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-d79991-openshift-default-74c857f7cc-p2snd" event={"ID":"1020808c-e760-46e7-99e8-66932fca1956","Type":"ContainerDied","Data":"554b43614667dd9cb4b1f94d5281367be991545f9ad45480363d8879d52c5001"}
Nov 02 15:48:47.941308 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:48:47.941285    5304 scope.go:122] "RemoveContainer" containerID="554b43614667dd9cb4b1f94d5281367be991545f9ad45480363d8879d52c5001"
Nov 02 15:48:47.943043 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:48:47.943010    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:48:47.943165 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:48:47.943079    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:48:47.943165 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:48:47.943113    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:48:48.950063 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:48:48.950024    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-d79991-openshift-default-74c857f7cc-p2snd" event={"ID":"1020808c-e760-46e7-99e8-66932fca1956","Type":"ContainerStarted","Data":"4f574ad5724d4e728c194a366add175a69e11f44da92c080ed4c3dcaad1512a1"}
Nov 02 15:48:49.214010 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:48:49.213923    5304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-d79991-openshift-default-74c857f7cc-p2snd"
Nov 02 15:48:49.214204 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:48:49.214109    5304 patch_prober.go:29] interesting pod/e2e-ipp-a-d79991-openshift-default-74c857f7cc-p2snd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused" start-of-body=
Nov 02 15:48:49.214271 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:48:49.214200    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-d79991-openshift-default-74c857f7cc-p2snd" podUID="1020808c-e760-46e7-99e8-66932fca1956" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused"
Nov 02 15:48:50.213970 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:48:50.213933    5304 patch_prober.go:29] interesting pod/e2e-ipp-a-d79991-openshift-default-74c857f7cc-p2snd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused" start-of-body=
Nov 02 15:48:50.214396 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:48:50.214008    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-d79991-openshift-default-74c857f7cc-p2snd" podUID="1020808c-e760-46e7-99e8-66932fca1956" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused"
Nov 02 15:48:50.881238 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:48:50.881207    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-d79991-6d4b6d8c48-ptv5r"
Nov 02 15:48:51.214341 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:48:51.214249    5304 patch_prober.go:29] interesting pod/e2e-ipp-a-d79991-openshift-default-74c857f7cc-p2snd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused" start-of-body=
Nov 02 15:48:51.214341 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:48:51.214317    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-d79991-openshift-default-74c857f7cc-p2snd" podUID="1020808c-e760-46e7-99e8-66932fca1956" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused"
Nov 02 15:48:52.214344 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:48:52.214302    5304 patch_prober.go:29] interesting pod/e2e-ipp-a-d79991-openshift-default-74c857f7cc-p2snd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused" start-of-body=
Nov 02 15:48:52.214847 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:48:52.214361    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-d79991-openshift-default-74c857f7cc-p2snd" podUID="1020808c-e760-46e7-99e8-66932fca1956" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused"
Nov 02 15:48:53.214689 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:48:53.214643    5304 patch_prober.go:29] interesting pod/e2e-ipp-a-d79991-openshift-default-74c857f7cc-p2snd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused" start-of-body=
Nov 02 15:48:53.215112 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:48:53.214704    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-d79991-openshift-default-74c857f7cc-p2snd" podUID="1020808c-e760-46e7-99e8-66932fca1956" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused"
Nov 02 15:48:54.214212 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:48:54.214176    5304 patch_prober.go:29] interesting pod/e2e-ipp-a-d79991-openshift-default-74c857f7cc-p2snd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused" start-of-body=
Nov 02 15:48:54.214459 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:48:54.214235    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-d79991-openshift-default-74c857f7cc-p2snd" podUID="1020808c-e760-46e7-99e8-66932fca1956" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused"
Nov 02 15:48:54.800368 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:48:54.800318    5304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k6fjk" containerName="extractor"
Nov 02 15:48:54.917399 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:48:54.917371    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-8537e9-b666465c7-58z69"
Nov 02 15:48:55.219259 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:48:55.219222    5304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-d79991-openshift-default-74c857f7cc-p2snd"
Nov 02 15:48:55.219635 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:48:55.219616    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-d79991-openshift-default-74c857f7cc-p2snd"
Nov 02 15:48:55.220350 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:48:55.220330    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-d79991-openshift-default-74c857f7cc-p2snd"
Nov 02 15:48:59.586431 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:48:59.586364    5304 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-5113c2/premium-w6-5113c2-kserve-b84945cb5-pc6f8" podUID="9fb3e992-7bb0-497a-a82c-6e396ba241c1" containerName="main" containerID="cri-o://df0454f8110d738a12ee04ac4b3f682c922588634410c0ac973c7be000df702e" gracePeriod=45
Nov 02 15:48:59.643648 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:48:59.627483    5304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-af33a6-59b4d9dc95-v9pl6"]
Nov 02 15:48:59.643648 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:48:59.628382    5304 cpu_manager.go:414] "containerMap: removing container" podUID="67da2b58-a0f0-44f6-9258-b418a21e73bc" containerName="istio-proxy"
Nov 02 15:48:59.643648 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:48:59.628404    5304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="67da2b58-a0f0-44f6-9258-b418a21e73bc" containerName="istio-proxy"
Nov 02 15:48:59.643648 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:48:59.628434    5304 cpu_manager.go:414] "containerMap: removing container" podUID="67da2b58-a0f0-44f6-9258-b418a21e73bc" containerName="istio-proxy"
Nov 02 15:48:59.643648 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:48:59.628442    5304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="67da2b58-a0f0-44f6-9258-b418a21e73bc" containerName="istio-proxy"
Nov 02 15:48:59.643648 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:48:59.628584    5304 memory_manager.go:365] "RemoveStaleState removing state" podUID="67da2b58-a0f0-44f6-9258-b418a21e73bc" containerName="istio-proxy"
Nov 02 15:48:59.661835 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:48:59.661802    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-af33a6-59b4d9dc95-v9pl6"]
Nov 02 15:48:59.662230 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:48:59.662145    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-af33a6-59b4d9dc95-v9pl6"
Nov 02 15:48:59.743156 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:48:59.735474    5304 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-af33a6\""
Nov 02 15:48:59.754196 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:48:59.742369    5304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-af33a6-dockercfg-6f8pj\""
Nov 02 15:48:59.766209 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:48:59.753225    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vngdw\" (UniqueName: \"kubernetes.io/projected/888a0ac9-4645-4fca-bc53-022beb9f484f-kube-api-access-vngdw\") pod \"payload-processing-e2e-ipp-b-af33a6-59b4d9dc95-v9pl6\" (UID: \"888a0ac9-4645-4fca-bc53-022beb9f484f\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-af33a6-59b4d9dc95-v9pl6"
Nov 02 15:48:59.788393 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:48:59.788356    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/888a0ac9-4645-4fca-bc53-022beb9f484f-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-af33a6-59b4d9dc95-v9pl6\" (UID: \"888a0ac9-4645-4fca-bc53-022beb9f484f\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-af33a6-59b4d9dc95-v9pl6"
Nov 02 15:48:59.891549 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:48:59.890833    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vngdw\" (UniqueName: \"kubernetes.io/projected/888a0ac9-4645-4fca-bc53-022beb9f484f-kube-api-access-vngdw\") pod \"payload-processing-e2e-ipp-b-af33a6-59b4d9dc95-v9pl6\" (UID: \"888a0ac9-4645-4fca-bc53-022beb9f484f\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-af33a6-59b4d9dc95-v9pl6"
Nov 02 15:48:59.891549 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:48:59.890888    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/888a0ac9-4645-4fca-bc53-022beb9f484f-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-af33a6-59b4d9dc95-v9pl6\" (UID: \"888a0ac9-4645-4fca-bc53-022beb9f484f\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-af33a6-59b4d9dc95-v9pl6"
Nov 02 15:48:59.894716 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:48:59.894689    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/888a0ac9-4645-4fca-bc53-022beb9f484f-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-af33a6-59b4d9dc95-v9pl6\" (UID: \"888a0ac9-4645-4fca-bc53-022beb9f484f\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-af33a6-59b4d9dc95-v9pl6"
Nov 02 15:48:59.918704 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:48:59.918677    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vngdw\" (UniqueName: \"kubernetes.io/projected/888a0ac9-4645-4fca-bc53-022beb9f484f-kube-api-access-vngdw\") pod \"payload-processing-e2e-ipp-b-af33a6-59b4d9dc95-v9pl6\" (UID: \"888a0ac9-4645-4fca-bc53-022beb9f484f\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-af33a6-59b4d9dc95-v9pl6"
Nov 02 15:49:00.206197 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:49:00.200315    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5113c2/premium-w6-5113c2-kserve-b84945cb5-pc6f8" podUID="9fb3e992-7bb0-497a-a82c-6e396ba241c1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": dial tcp 10.133.0.47:8000: connect: connection refused"
Nov 02 15:49:00.206197 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:49:00.205245    5304 generic.go:356] "Generic (PLEG): container finished" podID="9fb3e992-7bb0-497a-a82c-6e396ba241c1" containerID="df0454f8110d738a12ee04ac4b3f682c922588634410c0ac973c7be000df702e" exitCode=0
Nov 02 15:49:00.212177 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:49:00.212081    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-af33a6-59b4d9dc95-v9pl6"
Nov 02 15:49:00.228685 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:49:00.228294    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-5113c2/premium-w6-5113c2-kserve-b84945cb5-pc6f8" event={"ID":"9fb3e992-7bb0-497a-a82c-6e396ba241c1","Type":"ContainerDied","Data":"df0454f8110d738a12ee04ac4b3f682c922588634410c0ac973c7be000df702e"}
Nov 02 15:49:00.509381 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:49:00.509351    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-5113c2/premium-w6-5113c2-kserve-b84945cb5-pc6f8"
Nov 02 15:49:00.696217 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:49:00.696188    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-af33a6-59b4d9dc95-v9pl6"]
Nov 02 15:49:00.701542 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:49:00.701507    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9fb3e992-7bb0-497a-a82c-6e396ba241c1-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9fb3e992-7bb0-497a-a82c-6e396ba241c1-model-cache\") pod \"9fb3e992-7bb0-497a-a82c-6e396ba241c1\" (UID: \"9fb3e992-7bb0-497a-a82c-6e396ba241c1\") "
Nov 02 15:49:00.701670 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:49:00.701578    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9fb3e992-7bb0-497a-a82c-6e396ba241c1-dshm\" (UniqueName: \"kubernetes.io/empty-dir/9fb3e992-7bb0-497a-a82c-6e396ba241c1-dshm\") pod \"9fb3e992-7bb0-497a-a82c-6e396ba241c1\" (UID: \"9fb3e992-7bb0-497a-a82c-6e396ba241c1\") "
Nov 02 15:49:00.701743 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:49:00.701674    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9fb3e992-7bb0-497a-a82c-6e396ba241c1-tls-certs\" (UniqueName: \"kubernetes.io/secret/9fb3e992-7bb0-497a-a82c-6e396ba241c1-tls-certs\") pod \"9fb3e992-7bb0-497a-a82c-6e396ba241c1\" (UID: \"9fb3e992-7bb0-497a-a82c-6e396ba241c1\") "
Nov 02 15:49:00.701743 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:49:00.701708    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9fb3e992-7bb0-497a-a82c-6e396ba241c1-home\" (UniqueName: \"kubernetes.io/empty-dir/9fb3e992-7bb0-497a-a82c-6e396ba241c1-home\") pod \"9fb3e992-7bb0-497a-a82c-6e396ba241c1\" (UID: \"9fb3e992-7bb0-497a-a82c-6e396ba241c1\") "
Nov 02 15:49:00.701865 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:49:00.701765    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fb3e992-7bb0-497a-a82c-6e396ba241c1-model-cache" pod "9fb3e992-7bb0-497a-a82c-6e396ba241c1" (UID: "9fb3e992-7bb0-497a-a82c-6e396ba241c1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:49:00.701865 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:49:00.701798    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9fb3e992-7bb0-497a-a82c-6e396ba241c1-kube-api-access-k8rhb\" (UniqueName: \"kubernetes.io/projected/9fb3e992-7bb0-497a-a82c-6e396ba241c1-kube-api-access-k8rhb\") pod \"9fb3e992-7bb0-497a-a82c-6e396ba241c1\" (UID: \"9fb3e992-7bb0-497a-a82c-6e396ba241c1\") "
Nov 02 15:49:00.701978 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:49:00.701877    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9fb3e992-7bb0-497a-a82c-6e396ba241c1-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9fb3e992-7bb0-497a-a82c-6e396ba241c1-tmp-dir\") pod \"9fb3e992-7bb0-497a-a82c-6e396ba241c1\" (UID: \"9fb3e992-7bb0-497a-a82c-6e396ba241c1\") "
Nov 02 15:49:00.702387 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:49:00.702361    5304 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9fb3e992-7bb0-497a-a82c-6e396ba241c1-model-cache\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:49:00.702582 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:49:00.702548    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fb3e992-7bb0-497a-a82c-6e396ba241c1-home" pod "9fb3e992-7bb0-497a-a82c-6e396ba241c1" (UID: "9fb3e992-7bb0-497a-a82c-6e396ba241c1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:49:00.702666 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:49:00.702586    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fb3e992-7bb0-497a-a82c-6e396ba241c1-tmp-dir" pod "9fb3e992-7bb0-497a-a82c-6e396ba241c1" (UID: "9fb3e992-7bb0-497a-a82c-6e396ba241c1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:49:00.705017 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:49:00.704990    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fb3e992-7bb0-497a-a82c-6e396ba241c1-dshm" pod "9fb3e992-7bb0-497a-a82c-6e396ba241c1" (UID: "9fb3e992-7bb0-497a-a82c-6e396ba241c1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:49:00.705181 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:49:00.705073    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9fb3e992-7bb0-497a-a82c-6e396ba241c1-tls-certs" pod "9fb3e992-7bb0-497a-a82c-6e396ba241c1" (UID: "9fb3e992-7bb0-497a-a82c-6e396ba241c1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:49:00.706233 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:49:00.706207    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9fb3e992-7bb0-497a-a82c-6e396ba241c1-kube-api-access-k8rhb" pod "9fb3e992-7bb0-497a-a82c-6e396ba241c1" (UID: "9fb3e992-7bb0-497a-a82c-6e396ba241c1"). InnerVolumeSpecName "kube-api-access-k8rhb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:49:00.803293 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:49:00.803217    5304 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9fb3e992-7bb0-497a-a82c-6e396ba241c1-tmp-dir\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:49:00.803293 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:49:00.803242    5304 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9fb3e992-7bb0-497a-a82c-6e396ba241c1-dshm\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:49:00.803293 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:49:00.803250    5304 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9fb3e992-7bb0-497a-a82c-6e396ba241c1-tls-certs\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:49:00.803293 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:49:00.803258    5304 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9fb3e992-7bb0-497a-a82c-6e396ba241c1-home\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:49:00.803293 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:49:00.803267    5304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k8rhb\" (UniqueName: \"kubernetes.io/projected/9fb3e992-7bb0-497a-a82c-6e396ba241c1-kube-api-access-k8rhb\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:49:01.213138 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:49:01.213099    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-5113c2/premium-w6-5113c2-kserve-b84945cb5-pc6f8"
Nov 02 15:49:01.213791 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:49:01.213675    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-5113c2/premium-w6-5113c2-kserve-b84945cb5-pc6f8" event={"ID":"9fb3e992-7bb0-497a-a82c-6e396ba241c1","Type":"ContainerDied","Data":"3c228ada8b883005eebe18993d69e96b95249a253b8a7774aa8ba821ac49c390"}
Nov 02 15:49:01.213791 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:49:01.213750    5304 scope.go:122] "RemoveContainer" containerID="df0454f8110d738a12ee04ac4b3f682c922588634410c0ac973c7be000df702e"
Nov 02 15:49:01.216291 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:49:01.216266    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-af33a6-59b4d9dc95-v9pl6" event={"ID":"888a0ac9-4645-4fca-bc53-022beb9f484f","Type":"ContainerStarted","Data":"dd08f36bab9e1ecad2bc2d0e595679a301474467ed6f145ee6d41a25b91ded49"}
Nov 02 15:49:01.216403 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:49:01.216336    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-af33a6-59b4d9dc95-v9pl6" event={"ID":"888a0ac9-4645-4fca-bc53-022beb9f484f","Type":"ContainerStarted","Data":"c6ee1cf6632089dda0357a20d3efea8238d8046cc5f1b3a437ab68f3e684273d"}
Nov 02 15:49:01.216512 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:49:01.216498    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-af33a6-59b4d9dc95-v9pl6"
Nov 02 15:49:01.284802 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:49:01.284756    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-5113c2/premium-w6-5113c2-kserve-b84945cb5-pc6f8"]
Nov 02 15:49:01.284966 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:49:01.284814    5304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-5113c2/premium-w6-5113c2-kserve-b84945cb5-pc6f8"]
Nov 02 15:49:01.327198 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:49:01.324315    5304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-af33a6-59b4d9dc95-v9pl6" podStartSLOduration=2.3242954080000002 podStartE2EDuration="2.324295408s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:48:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:49:01.316960334 +0000 UTC m=+3141.310784853" watchObservedRunningTime="2026-11-02 15:49:01.324295408 +0000 UTC m=+3141.318119819"
Nov 02 15:49:02.118532 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:49:02.118489    5304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9fb3e992-7bb0-497a-a82c-6e396ba241c1" path="/var/lib/kubelet/pods/9fb3e992-7bb0-497a-a82c-6e396ba241c1/volumes"
Nov 02 15:49:08.792342 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:49:08.745882    5304 patch_prober.go:29] interesting pod/e2e-ipp-b-5721b1-openshift-default-56c5cbb5cc-987lj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.210:15021/healthz/ready\": read tcp 10.133.0.2:34110->10.133.0.210:15021: read: connection reset by peer" start-of-body=
Nov 02 15:49:08.792342 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:49:08.745965    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-5721b1-openshift-default-56c5cbb5cc-987lj" podUID="1285186b-0b54-48b3-84c6-b640966bb312" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.210:15021/healthz/ready\": read tcp 10.133.0.2:34110->10.133.0.210:15021: read: connection reset by peer"
Nov 02 15:49:09.299163 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:49:09.299092    5304 generic.go:356] "Generic (PLEG): container finished" podID="1285186b-0b54-48b3-84c6-b640966bb312" containerID="4eb2e692475edc363f1e913a7bc8179f0ccb216421f28c4a087c0e4a9bcd6285" exitCode=0
Nov 02 15:49:09.299475 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:49:09.299180    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-5721b1-openshift-default-56c5cbb5cc-987lj" event={"ID":"1285186b-0b54-48b3-84c6-b640966bb312","Type":"ContainerDied","Data":"4eb2e692475edc363f1e913a7bc8179f0ccb216421f28c4a087c0e4a9bcd6285"}
Nov 02 15:49:09.299709 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:49:09.299684    5304 scope.go:122] "RemoveContainer" containerID="4eb2e692475edc363f1e913a7bc8179f0ccb216421f28c4a087c0e4a9bcd6285"
Nov 02 15:49:09.301684 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:49:09.301654    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:49:09.301784 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:49:09.301728    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:49:09.301784 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:49:09.301762    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:49:10.309192 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:49:10.309157    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-5721b1-openshift-default-56c5cbb5cc-987lj" event={"ID":"1285186b-0b54-48b3-84c6-b640966bb312","Type":"ContainerStarted","Data":"69030f9b3280d9c4cc9be8d8541281ac5da89159dba6f823a5d22e19d15d1cec"}
Nov 02 15:49:10.394791 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:49:10.394762    5304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-5721b1-openshift-default-56c5cbb5cc-987lj"
Nov 02 15:49:10.395038 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:49:10.395014    5304 patch_prober.go:29] interesting pod/e2e-ipp-b-5721b1-openshift-default-56c5cbb5cc-987lj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused" start-of-body=
Nov 02 15:49:10.395092 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:49:10.395067    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-5721b1-openshift-default-56c5cbb5cc-987lj" podUID="1285186b-0b54-48b3-84c6-b640966bb312" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused"
Nov 02 15:49:11.320943 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:49:11.320853    5304 generic.go:356] "Generic (PLEG): container finished" podID="d7ab8a29-4410-4fde-8975-49c13b3ff1fb" containerID="c4fc082d0ebbceccdf88601b58447c83a6bc5dd2fe8ba878d8d75c78d9d8729f" exitCode=0
Nov 02 15:49:11.320943 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:49:11.320921    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-e40940-openshift-default-6fb44b465f-75t4b" event={"ID":"d7ab8a29-4410-4fde-8975-49c13b3ff1fb","Type":"ContainerDied","Data":"c4fc082d0ebbceccdf88601b58447c83a6bc5dd2fe8ba878d8d75c78d9d8729f"}
Nov 02 15:49:11.321458 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:49:11.321411    5304 scope.go:122] "RemoveContainer" containerID="c4fc082d0ebbceccdf88601b58447c83a6bc5dd2fe8ba878d8d75c78d9d8729f"
Nov 02 15:49:11.323174 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:49:11.323107    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:49:11.323241 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:49:11.323217    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:49:11.323283 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:49:11.323249    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:49:11.395302 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:49:11.395273    5304 patch_prober.go:29] interesting pod/e2e-ipp-b-5721b1-openshift-default-56c5cbb5cc-987lj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused" start-of-body=
Nov 02 15:49:11.395445 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:49:11.395329    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-5721b1-openshift-default-56c5cbb5cc-987lj" podUID="1285186b-0b54-48b3-84c6-b640966bb312" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused"
Nov 02 15:49:12.227404 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:49:12.227382    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-af33a6-59b4d9dc95-v9pl6"
Nov 02 15:49:12.351152 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:49:12.330278    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-e40940-openshift-default-6fb44b465f-75t4b" event={"ID":"d7ab8a29-4410-4fde-8975-49c13b3ff1fb","Type":"ContainerStarted","Data":"334e92c952bf360c8eca5a791970798152aaf9ca276333fe8a507a4a7e643043"}
Nov 02 15:49:12.395514 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:49:12.395476    5304 patch_prober.go:29] interesting pod/e2e-ipp-b-5721b1-openshift-default-56c5cbb5cc-987lj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused" start-of-body=
Nov 02 15:49:12.395715 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:49:12.395537    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-5721b1-openshift-default-56c5cbb5cc-987lj" podUID="1285186b-0b54-48b3-84c6-b640966bb312" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused"
Nov 02 15:49:13.090354 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:49:13.090322    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-e40940-openshift-default-6fb44b465f-75t4b"
Nov 02 15:49:13.163150 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:49:13.162500    5304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-e40940-openshift-default-6fb44b465f-75t4b"
Nov 02 15:49:13.172305 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:49:13.172265    5304 patch_prober.go:29] interesting pod/e2e-ipp-a-e40940-openshift-default-6fb44b465f-75t4b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused" start-of-body=
Nov 02 15:49:13.172643 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:49:13.172609    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-e40940-openshift-default-6fb44b465f-75t4b" podUID="d7ab8a29-4410-4fde-8975-49c13b3ff1fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused"
Nov 02 15:49:13.394818 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:49:13.394718    5304 patch_prober.go:29] interesting pod/e2e-ipp-b-5721b1-openshift-default-56c5cbb5cc-987lj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused" start-of-body=
Nov 02 15:49:13.394818 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:49:13.394788    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-5721b1-openshift-default-56c5cbb5cc-987lj" podUID="1285186b-0b54-48b3-84c6-b640966bb312" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused"
Nov 02 15:49:14.172150 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:49:14.168542    5304 patch_prober.go:29] interesting pod/e2e-ipp-a-e40940-openshift-default-6fb44b465f-75t4b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused" start-of-body=
Nov 02 15:49:14.172150 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:49:14.168603    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-e40940-openshift-default-6fb44b465f-75t4b" podUID="d7ab8a29-4410-4fde-8975-49c13b3ff1fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused"
Nov 02 15:49:14.394835 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:49:14.394793    5304 patch_prober.go:29] interesting pod/e2e-ipp-b-5721b1-openshift-default-56c5cbb5cc-987lj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused" start-of-body=
Nov 02 15:49:14.395376 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:49:14.394868    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-5721b1-openshift-default-56c5cbb5cc-987lj" podUID="1285186b-0b54-48b3-84c6-b640966bb312" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused"
Nov 02 15:49:15.162151 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:49:15.161563    5304 patch_prober.go:29] interesting pod/e2e-ipp-a-e40940-openshift-default-6fb44b465f-75t4b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused" start-of-body=
Nov 02 15:49:15.162151 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:49:15.161645    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-e40940-openshift-default-6fb44b465f-75t4b" podUID="d7ab8a29-4410-4fde-8975-49c13b3ff1fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused"
Nov 02 15:49:15.395291 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:49:15.395250    5304 patch_prober.go:29] interesting pod/e2e-ipp-b-5721b1-openshift-default-56c5cbb5cc-987lj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused" start-of-body=
Nov 02 15:49:15.395731 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:49:15.395338    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-5721b1-openshift-default-56c5cbb5cc-987lj" podUID="1285186b-0b54-48b3-84c6-b640966bb312" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused"
Nov 02 15:49:16.161528 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:49:16.161489    5304 patch_prober.go:29] interesting pod/e2e-ipp-a-e40940-openshift-default-6fb44b465f-75t4b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused" start-of-body=
Nov 02 15:49:16.161753 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:49:16.161566    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-e40940-openshift-default-6fb44b465f-75t4b" podUID="d7ab8a29-4410-4fde-8975-49c13b3ff1fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused"
Nov 02 15:49:16.399227 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:49:16.399195    5304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-5721b1-openshift-default-56c5cbb5cc-987lj"
Nov 02 15:49:16.399667 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:49:16.399591    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-5721b1-openshift-default-56c5cbb5cc-987lj"
Nov 02 15:49:16.400564 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:49:16.400544    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-5721b1-openshift-default-56c5cbb5cc-987lj"
Nov 02 15:49:17.161009 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:49:17.160971    5304 patch_prober.go:29] interesting pod/e2e-ipp-a-e40940-openshift-default-6fb44b465f-75t4b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused" start-of-body=
Nov 02 15:49:17.161261 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:49:17.161035    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-e40940-openshift-default-6fb44b465f-75t4b" podUID="d7ab8a29-4410-4fde-8975-49c13b3ff1fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused"
Nov 02 15:49:18.165943 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:49:18.165911    5304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-e40940-openshift-default-6fb44b465f-75t4b"
Nov 02 15:49:18.166877 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:49:18.166860    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-e40940-openshift-default-6fb44b465f-75t4b"
Nov 02 15:49:24.799105 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:49:24.799017    5304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k6fjk" containerName="extractor"
Nov 02 15:49:31.460281 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:49:31.460238    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-24255b-5555cd4dc9-jmck4"]
Nov 02 15:49:31.460724 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:49:31.460556    5304 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-24255b-5555cd4dc9-jmck4" podUID="774e4952-c994-4374-aa62-3d67cc777d5a" containerName="maas-api" containerID="cri-o://16e425e93f9a7e78b486668ef7f7db3b4a2bc1744569f2bbc3ca5e57798348b2" gracePeriod=30
Nov 02 15:49:31.581706 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:49:31.581666    5304 generic.go:356] "Generic (PLEG): container finished" podID="774e4952-c994-4374-aa62-3d67cc777d5a" containerID="16e425e93f9a7e78b486668ef7f7db3b4a2bc1744569f2bbc3ca5e57798348b2" exitCode=0
Nov 02 15:49:31.581834 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:49:31.581719    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-24255b-5555cd4dc9-jmck4" event={"ID":"774e4952-c994-4374-aa62-3d67cc777d5a","Type":"ContainerDied","Data":"16e425e93f9a7e78b486668ef7f7db3b4a2bc1744569f2bbc3ca5e57798348b2"}
Nov 02 15:49:31.643690 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:49:31.643647    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-24255b-84c779c487-bnkpw"]
Nov 02 15:49:31.643887 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:49:31.643856    5304 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-24255b-84c779c487-bnkpw" podUID="30cc7413-f0e9-4734-bcea-2a731bd75f88" containerName="payload-pre-processing" containerID="cri-o://4f6b1cbe90d12222a64d90c99a9785a9b4ee7f4659d190a5cf3bb63858b90a39" gracePeriod=30
Nov 02 15:49:31.805477 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:49:31.805440    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-24255b-5555cd4dc9-jmck4"
Nov 02 15:49:31.923609 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:49:31.923575    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-24255b-84c779c487-bnkpw"
Nov 02 15:49:32.095110 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:49:32.093643    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/774e4952-c994-4374-aa62-3d67cc777d5a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/774e4952-c994-4374-aa62-3d67cc777d5a-maas-api-tls\") pod \"774e4952-c994-4374-aa62-3d67cc777d5a\" (UID: \"774e4952-c994-4374-aa62-3d67cc777d5a\") "
Nov 02 15:49:32.095110 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:49:32.093699    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/774e4952-c994-4374-aa62-3d67cc777d5a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/774e4952-c994-4374-aa62-3d67cc777d5a-openshift-service-ca\") pod \"774e4952-c994-4374-aa62-3d67cc777d5a\" (UID: \"774e4952-c994-4374-aa62-3d67cc777d5a\") "
Nov 02 15:49:32.095110 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:49:32.093797    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/774e4952-c994-4374-aa62-3d67cc777d5a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/774e4952-c994-4374-aa62-3d67cc777d5a-metrics-certs\") pod \"774e4952-c994-4374-aa62-3d67cc777d5a\" (UID: \"774e4952-c994-4374-aa62-3d67cc777d5a\") "
Nov 02 15:49:32.095110 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:49:32.093867    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/774e4952-c994-4374-aa62-3d67cc777d5a-kube-api-access-d6ccq\" (UniqueName: \"kubernetes.io/projected/774e4952-c994-4374-aa62-3d67cc777d5a-kube-api-access-d6ccq\") pod \"774e4952-c994-4374-aa62-3d67cc777d5a\" (UID: \"774e4952-c994-4374-aa62-3d67cc777d5a\") "
Nov 02 15:49:32.099909 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:49:32.096223    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/774e4952-c994-4374-aa62-3d67cc777d5a-openshift-service-ca" pod "774e4952-c994-4374-aa62-3d67cc777d5a" (UID: "774e4952-c994-4374-aa62-3d67cc777d5a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:49:32.106196 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:49:32.105392    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/774e4952-c994-4374-aa62-3d67cc777d5a-metrics-certs" pod "774e4952-c994-4374-aa62-3d67cc777d5a" (UID: "774e4952-c994-4374-aa62-3d67cc777d5a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:49:32.112308 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:49:32.112275    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/774e4952-c994-4374-aa62-3d67cc777d5a-maas-api-tls" pod "774e4952-c994-4374-aa62-3d67cc777d5a" (UID: "774e4952-c994-4374-aa62-3d67cc777d5a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:49:32.112476 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:49:32.112389    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/774e4952-c994-4374-aa62-3d67cc777d5a-kube-api-access-d6ccq" pod "774e4952-c994-4374-aa62-3d67cc777d5a" (UID: "774e4952-c994-4374-aa62-3d67cc777d5a"). InnerVolumeSpecName "kube-api-access-d6ccq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:49:32.195154 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:49:32.195106    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/30cc7413-f0e9-4734-bcea-2a731bd75f88-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30cc7413-f0e9-4734-bcea-2a731bd75f88-plugins-config-volume\") pod \"30cc7413-f0e9-4734-bcea-2a731bd75f88\" (UID: \"30cc7413-f0e9-4734-bcea-2a731bd75f88\") "
Nov 02 15:49:32.195364 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:49:32.195251    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30cc7413-f0e9-4734-bcea-2a731bd75f88-kube-api-access-qsjn4\" (UniqueName: \"kubernetes.io/projected/30cc7413-f0e9-4734-bcea-2a731bd75f88-kube-api-access-qsjn4\") pod \"30cc7413-f0e9-4734-bcea-2a731bd75f88\" (UID: \"30cc7413-f0e9-4734-bcea-2a731bd75f88\") "
Nov 02 15:49:32.195555 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:49:32.195531    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30cc7413-f0e9-4734-bcea-2a731bd75f88-plugins-config-volume" pod "30cc7413-f0e9-4734-bcea-2a731bd75f88" (UID: "30cc7413-f0e9-4734-bcea-2a731bd75f88"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:49:32.195852 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:49:32.195834    5304 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30cc7413-f0e9-4734-bcea-2a731bd75f88-plugins-config-volume\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:49:32.195901 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:49:32.195855    5304 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/774e4952-c994-4374-aa62-3d67cc777d5a-maas-api-tls\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:49:32.195901 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:49:32.195867    5304 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/774e4952-c994-4374-aa62-3d67cc777d5a-openshift-service-ca\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:49:32.195901 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:49:32.195875    5304 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/774e4952-c994-4374-aa62-3d67cc777d5a-metrics-certs\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:49:32.195901 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:49:32.195883    5304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d6ccq\" (UniqueName: \"kubernetes.io/projected/774e4952-c994-4374-aa62-3d67cc777d5a-kube-api-access-d6ccq\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:49:32.197465 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:49:32.197438    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30cc7413-f0e9-4734-bcea-2a731bd75f88-kube-api-access-qsjn4" pod "30cc7413-f0e9-4734-bcea-2a731bd75f88" (UID: "30cc7413-f0e9-4734-bcea-2a731bd75f88"). InnerVolumeSpecName "kube-api-access-qsjn4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:49:32.297296 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:49:32.297261    5304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qsjn4\" (UniqueName: \"kubernetes.io/projected/30cc7413-f0e9-4734-bcea-2a731bd75f88-kube-api-access-qsjn4\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:49:32.589960 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:49:32.589928    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-24255b-5555cd4dc9-jmck4"
Nov 02 15:49:32.590513 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:49:32.589964    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-24255b-5555cd4dc9-jmck4" event={"ID":"774e4952-c994-4374-aa62-3d67cc777d5a","Type":"ContainerDied","Data":"1a5b8f14764643709ecd03c9cd785d37a5367c7666d116105dea2d3aa5522a05"}
Nov 02 15:49:32.590513 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:49:32.590034    5304 scope.go:122] "RemoveContainer" containerID="16e425e93f9a7e78b486668ef7f7db3b4a2bc1744569f2bbc3ca5e57798348b2"
Nov 02 15:49:32.591613 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:49:32.591583    5304 generic.go:356] "Generic (PLEG): container finished" podID="30cc7413-f0e9-4734-bcea-2a731bd75f88" containerID="4f6b1cbe90d12222a64d90c99a9785a9b4ee7f4659d190a5cf3bb63858b90a39" exitCode=0
Nov 02 15:49:32.591764 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:49:32.591638    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-24255b-84c779c487-bnkpw"
Nov 02 15:49:32.591764 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:49:32.591638    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-24255b-84c779c487-bnkpw" event={"ID":"30cc7413-f0e9-4734-bcea-2a731bd75f88","Type":"ContainerDied","Data":"4f6b1cbe90d12222a64d90c99a9785a9b4ee7f4659d190a5cf3bb63858b90a39"}
Nov 02 15:49:32.591868 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:49:32.591807    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-24255b-84c779c487-bnkpw" event={"ID":"30cc7413-f0e9-4734-bcea-2a731bd75f88","Type":"ContainerDied","Data":"d8ee8a80f543a6b81a5dd5fd5b79fcd8634cb05de62a418bd53e77166cad68a9"}
Nov 02 15:49:32.602412 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:49:32.602388    5304 scope.go:122] "RemoveContainer" containerID="4f6b1cbe90d12222a64d90c99a9785a9b4ee7f4659d190a5cf3bb63858b90a39"
Nov 02 15:49:32.613927 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:49:32.613908    5304 scope.go:122] "RemoveContainer" containerID="4f6b1cbe90d12222a64d90c99a9785a9b4ee7f4659d190a5cf3bb63858b90a39"
Nov 02 15:49:32.614270 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:49:32.614243    5304 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f6b1cbe90d12222a64d90c99a9785a9b4ee7f4659d190a5cf3bb63858b90a39\": container with ID starting with 4f6b1cbe90d12222a64d90c99a9785a9b4ee7f4659d190a5cf3bb63858b90a39 not found: ID does not exist" containerID="4f6b1cbe90d12222a64d90c99a9785a9b4ee7f4659d190a5cf3bb63858b90a39"
Nov 02 15:49:32.614368 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:49:32.614277    5304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f6b1cbe90d12222a64d90c99a9785a9b4ee7f4659d190a5cf3bb63858b90a39"} err="failed to get container status \"4f6b1cbe90d12222a64d90c99a9785a9b4ee7f4659d190a5cf3bb63858b90a39\": rpc error: code = NotFound desc = could not find container \"4f6b1cbe90d12222a64d90c99a9785a9b4ee7f4659d190a5cf3bb63858b90a39\": container with ID starting with 4f6b1cbe90d12222a64d90c99a9785a9b4ee7f4659d190a5cf3bb63858b90a39 not found: ID does not exist"
Nov 02 15:49:32.645511 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:49:32.645482    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-24255b-5555cd4dc9-jmck4"]
Nov 02 15:49:32.657304 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:49:32.657277    5304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-24255b-5555cd4dc9-jmck4"]
Nov 02 15:49:32.697000 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:49:32.696963    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-24255b-84c779c487-bnkpw"]
Nov 02 15:49:32.700040 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:49:32.700014    5304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-24255b-84c779c487-bnkpw"]
Nov 02 15:49:34.119399 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:49:34.119367    5304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="30cc7413-f0e9-4734-bcea-2a731bd75f88" path="/var/lib/kubelet/pods/30cc7413-f0e9-4734-bcea-2a731bd75f88/volumes"
Nov 02 15:49:34.119857 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:49:34.119843    5304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="774e4952-c994-4374-aa62-3d67cc777d5a" path="/var/lib/kubelet/pods/774e4952-c994-4374-aa62-3d67cc777d5a/volumes"
Nov 02 15:49:40.157740 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:49:40.157701    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-24255b-openshift-default-6687484f49-992gq"]
Nov 02 15:49:40.158371 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:49:40.157955    5304 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-24255b-openshift-default-6687484f49-992gq" podUID="c19acd3a-5dbf-4da3-b88f-e36284b22243" containerName="istio-proxy" containerID="cri-o://4128ca1e0694d81d32c0ef9ddbec96c80ea57847126a2ebb01bc306d74548a25" gracePeriod=30
Nov 02 15:49:45.430325 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:49:45.430261    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-24255b-openshift-default-6687484f49-992gq"
Nov 02 15:49:45.531462 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:49:45.531426    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c19acd3a-5dbf-4da3-b88f-e36284b22243-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c19acd3a-5dbf-4da3-b88f-e36284b22243-workload-certs\") pod \"c19acd3a-5dbf-4da3-b88f-e36284b22243\" (UID: \"c19acd3a-5dbf-4da3-b88f-e36284b22243\") "
Nov 02 15:49:45.531640 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:49:45.531491    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c19acd3a-5dbf-4da3-b88f-e36284b22243-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c19acd3a-5dbf-4da3-b88f-e36284b22243-istiod-ca-cert\") pod \"c19acd3a-5dbf-4da3-b88f-e36284b22243\" (UID: \"c19acd3a-5dbf-4da3-b88f-e36284b22243\") "
Nov 02 15:49:45.531640 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:49:45.531517    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c19acd3a-5dbf-4da3-b88f-e36284b22243-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c19acd3a-5dbf-4da3-b88f-e36284b22243-istio-podinfo\") pod \"c19acd3a-5dbf-4da3-b88f-e36284b22243\" (UID: \"c19acd3a-5dbf-4da3-b88f-e36284b22243\") "
Nov 02 15:49:45.531640 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:49:45.531623    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c19acd3a-5dbf-4da3-b88f-e36284b22243-istio-token\" (UniqueName: \"kubernetes.io/projected/c19acd3a-5dbf-4da3-b88f-e36284b22243-istio-token\") pod \"c19acd3a-5dbf-4da3-b88f-e36284b22243\" (UID: \"c19acd3a-5dbf-4da3-b88f-e36284b22243\") "
Nov 02 15:49:45.531841 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:49:45.531704    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c19acd3a-5dbf-4da3-b88f-e36284b22243-kube-api-access-cn7fd\" (UniqueName: \"kubernetes.io/projected/c19acd3a-5dbf-4da3-b88f-e36284b22243-kube-api-access-cn7fd\") pod \"c19acd3a-5dbf-4da3-b88f-e36284b22243\" (UID: \"c19acd3a-5dbf-4da3-b88f-e36284b22243\") "
Nov 02 15:49:45.531841 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:49:45.531706    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c19acd3a-5dbf-4da3-b88f-e36284b22243-workload-certs" pod "c19acd3a-5dbf-4da3-b88f-e36284b22243" (UID: "c19acd3a-5dbf-4da3-b88f-e36284b22243"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:49:45.531841 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:49:45.531810    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c19acd3a-5dbf-4da3-b88f-e36284b22243-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c19acd3a-5dbf-4da3-b88f-e36284b22243-workload-socket\") pod \"c19acd3a-5dbf-4da3-b88f-e36284b22243\" (UID: \"c19acd3a-5dbf-4da3-b88f-e36284b22243\") "
Nov 02 15:49:45.532013 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:49:45.531936    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c19acd3a-5dbf-4da3-b88f-e36284b22243-istiod-ca-cert" pod "c19acd3a-5dbf-4da3-b88f-e36284b22243" (UID: "c19acd3a-5dbf-4da3-b88f-e36284b22243"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:49:45.532084 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:49:45.532060    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c19acd3a-5dbf-4da3-b88f-e36284b22243-workload-socket" pod "c19acd3a-5dbf-4da3-b88f-e36284b22243" (UID: "c19acd3a-5dbf-4da3-b88f-e36284b22243"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:49:45.532084 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:49:45.532077    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c19acd3a-5dbf-4da3-b88f-e36284b22243-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c19acd3a-5dbf-4da3-b88f-e36284b22243-istio-data\") pod \"c19acd3a-5dbf-4da3-b88f-e36284b22243\" (UID: \"c19acd3a-5dbf-4da3-b88f-e36284b22243\") "
Nov 02 15:49:45.532233 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:49:45.532177    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c19acd3a-5dbf-4da3-b88f-e36284b22243-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c19acd3a-5dbf-4da3-b88f-e36284b22243-credential-socket\") pod \"c19acd3a-5dbf-4da3-b88f-e36284b22243\" (UID: \"c19acd3a-5dbf-4da3-b88f-e36284b22243\") "
Nov 02 15:49:45.532233 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:49:45.532224    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c19acd3a-5dbf-4da3-b88f-e36284b22243-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c19acd3a-5dbf-4da3-b88f-e36284b22243-istio-envoy\") pod \"c19acd3a-5dbf-4da3-b88f-e36284b22243\" (UID: \"c19acd3a-5dbf-4da3-b88f-e36284b22243\") "
Nov 02 15:49:45.532349 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:49:45.532293    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c19acd3a-5dbf-4da3-b88f-e36284b22243-istio-data" pod "c19acd3a-5dbf-4da3-b88f-e36284b22243" (UID: "c19acd3a-5dbf-4da3-b88f-e36284b22243"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:49:45.532489 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:49:45.532467    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c19acd3a-5dbf-4da3-b88f-e36284b22243-credential-socket" pod "c19acd3a-5dbf-4da3-b88f-e36284b22243" (UID: "c19acd3a-5dbf-4da3-b88f-e36284b22243"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:49:45.532929 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:49:45.532837    5304 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c19acd3a-5dbf-4da3-b88f-e36284b22243-workload-socket\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:49:45.532929 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:49:45.532865    5304 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c19acd3a-5dbf-4da3-b88f-e36284b22243-istio-data\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:49:45.532929 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:49:45.532879    5304 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c19acd3a-5dbf-4da3-b88f-e36284b22243-credential-socket\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:49:45.532929 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:49:45.532893    5304 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c19acd3a-5dbf-4da3-b88f-e36284b22243-workload-certs\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:49:45.532929 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:49:45.532906    5304 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c19acd3a-5dbf-4da3-b88f-e36284b22243-istiod-ca-cert\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:49:45.534506 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:49:45.534335    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c19acd3a-5dbf-4da3-b88f-e36284b22243-istio-envoy" pod "c19acd3a-5dbf-4da3-b88f-e36284b22243" (UID: "c19acd3a-5dbf-4da3-b88f-e36284b22243"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:49:45.534506 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:49:45.534438    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c19acd3a-5dbf-4da3-b88f-e36284b22243-istio-token" pod "c19acd3a-5dbf-4da3-b88f-e36284b22243" (UID: "c19acd3a-5dbf-4da3-b88f-e36284b22243"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:49:45.535344 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:49:45.534684    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c19acd3a-5dbf-4da3-b88f-e36284b22243-istio-podinfo" pod "c19acd3a-5dbf-4da3-b88f-e36284b22243" (UID: "c19acd3a-5dbf-4da3-b88f-e36284b22243"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 15:49:45.535344 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:49:45.534827    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c19acd3a-5dbf-4da3-b88f-e36284b22243-kube-api-access-cn7fd" pod "c19acd3a-5dbf-4da3-b88f-e36284b22243" (UID: "c19acd3a-5dbf-4da3-b88f-e36284b22243"). InnerVolumeSpecName "kube-api-access-cn7fd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:49:45.633956 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:49:45.633924    5304 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c19acd3a-5dbf-4da3-b88f-e36284b22243-istio-envoy\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:49:45.633956 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:49:45.633952    5304 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c19acd3a-5dbf-4da3-b88f-e36284b22243-istio-podinfo\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:49:45.633956 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:49:45.633961    5304 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c19acd3a-5dbf-4da3-b88f-e36284b22243-istio-token\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:49:45.634243 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:49:45.633972    5304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cn7fd\" (UniqueName: \"kubernetes.io/projected/c19acd3a-5dbf-4da3-b88f-e36284b22243-kube-api-access-cn7fd\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:49:45.685935 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:49:45.685853    5304 generic.go:356] "Generic (PLEG): container finished" podID="c19acd3a-5dbf-4da3-b88f-e36284b22243" containerID="4128ca1e0694d81d32c0ef9ddbec96c80ea57847126a2ebb01bc306d74548a25" exitCode=0
Nov 02 15:49:45.685935 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:49:45.685902    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-24255b-openshift-default-6687484f49-992gq"
Nov 02 15:49:45.685935 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:49:45.685918    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-24255b-openshift-default-6687484f49-992gq" event={"ID":"c19acd3a-5dbf-4da3-b88f-e36284b22243","Type":"ContainerDied","Data":"4128ca1e0694d81d32c0ef9ddbec96c80ea57847126a2ebb01bc306d74548a25"}
Nov 02 15:49:45.686202 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:49:45.685985    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-24255b-openshift-default-6687484f49-992gq" event={"ID":"c19acd3a-5dbf-4da3-b88f-e36284b22243","Type":"ContainerDied","Data":"4f8dd28ee9dc312e3de29e0ec68a15de89453d8c7272e98b0882723acd7e26e2"}
Nov 02 15:49:45.686202 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:49:45.686009    5304 scope.go:122] "RemoveContainer" containerID="4128ca1e0694d81d32c0ef9ddbec96c80ea57847126a2ebb01bc306d74548a25"
Nov 02 15:49:45.696671 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:49:45.696652    5304 scope.go:122] "RemoveContainer" containerID="2a334a4d6be2f27cefd44840ccc5c33c95658bbdb551ba895da7970f96112d02"
Nov 02 15:49:45.705846 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:49:45.705828    5304 scope.go:122] "RemoveContainer" containerID="4128ca1e0694d81d32c0ef9ddbec96c80ea57847126a2ebb01bc306d74548a25"
Nov 02 15:49:45.706107 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:49:45.706085    5304 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4128ca1e0694d81d32c0ef9ddbec96c80ea57847126a2ebb01bc306d74548a25\": container with ID starting with 4128ca1e0694d81d32c0ef9ddbec96c80ea57847126a2ebb01bc306d74548a25 not found: ID does not exist" containerID="4128ca1e0694d81d32c0ef9ddbec96c80ea57847126a2ebb01bc306d74548a25"
Nov 02 15:49:45.706209 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:49:45.706136    5304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4128ca1e0694d81d32c0ef9ddbec96c80ea57847126a2ebb01bc306d74548a25"} err="failed to get container status \"4128ca1e0694d81d32c0ef9ddbec96c80ea57847126a2ebb01bc306d74548a25\": rpc error: code = NotFound desc = could not find container \"4128ca1e0694d81d32c0ef9ddbec96c80ea57847126a2ebb01bc306d74548a25\": container with ID starting with 4128ca1e0694d81d32c0ef9ddbec96c80ea57847126a2ebb01bc306d74548a25 not found: ID does not exist"
Nov 02 15:49:45.706209 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:49:45.706161    5304 scope.go:122] "RemoveContainer" containerID="2a334a4d6be2f27cefd44840ccc5c33c95658bbdb551ba895da7970f96112d02"
Nov 02 15:49:45.706406 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:49:45.706388    5304 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a334a4d6be2f27cefd44840ccc5c33c95658bbdb551ba895da7970f96112d02\": container with ID starting with 2a334a4d6be2f27cefd44840ccc5c33c95658bbdb551ba895da7970f96112d02 not found: ID does not exist" containerID="2a334a4d6be2f27cefd44840ccc5c33c95658bbdb551ba895da7970f96112d02"
Nov 02 15:49:45.706447 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:49:45.706413    5304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a334a4d6be2f27cefd44840ccc5c33c95658bbdb551ba895da7970f96112d02"} err="failed to get container status \"2a334a4d6be2f27cefd44840ccc5c33c95658bbdb551ba895da7970f96112d02\": rpc error: code = NotFound desc = could not find container \"2a334a4d6be2f27cefd44840ccc5c33c95658bbdb551ba895da7970f96112d02\": container with ID starting with 2a334a4d6be2f27cefd44840ccc5c33c95658bbdb551ba895da7970f96112d02 not found: ID does not exist"
Nov 02 15:49:45.739542 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:49:45.739509    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-24255b-openshift-default-6687484f49-992gq"]
Nov 02 15:49:45.747464 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:49:45.747437    5304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-24255b-openshift-default-6687484f49-992gq"]
Nov 02 15:49:46.119692 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:49:46.119660    5304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c19acd3a-5dbf-4da3-b88f-e36284b22243" path="/var/lib/kubelet/pods/c19acd3a-5dbf-4da3-b88f-e36284b22243/volumes"
Nov 02 15:49:50.657774 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:49:50.650748    5304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-e40940-7c4db584c5-jjf2w"]
Nov 02 15:49:50.657774 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:49:50.651647    5304 cpu_manager.go:414] "containerMap: removing container" podUID="c19acd3a-5dbf-4da3-b88f-e36284b22243" containerName="istio-proxy"
Nov 02 15:49:50.657774 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:49:50.651668    5304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c19acd3a-5dbf-4da3-b88f-e36284b22243" containerName="istio-proxy"
Nov 02 15:49:50.657774 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:49:50.651681    5304 cpu_manager.go:414] "containerMap: removing container" podUID="774e4952-c994-4374-aa62-3d67cc777d5a" containerName="maas-api"
Nov 02 15:49:50.657774 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:49:50.651691    5304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="774e4952-c994-4374-aa62-3d67cc777d5a" containerName="maas-api"
Nov 02 15:49:50.657774 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:49:50.651727    5304 cpu_manager.go:414] "containerMap: removing container" podUID="30cc7413-f0e9-4734-bcea-2a731bd75f88" containerName="payload-pre-processing"
Nov 02 15:49:50.657774 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:49:50.651738    5304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30cc7413-f0e9-4734-bcea-2a731bd75f88" containerName="payload-pre-processing"
Nov 02 15:49:50.657774 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:49:50.651763    5304 cpu_manager.go:414] "containerMap: removing container" podUID="9fb3e992-7bb0-497a-a82c-6e396ba241c1" containerName="main"
Nov 02 15:49:50.657774 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:49:50.651770    5304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9fb3e992-7bb0-497a-a82c-6e396ba241c1" containerName="main"
Nov 02 15:49:50.657774 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:49:50.651941    5304 memory_manager.go:365] "RemoveStaleState removing state" podUID="c19acd3a-5dbf-4da3-b88f-e36284b22243" containerName="istio-proxy"
Nov 02 15:49:50.657774 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:49:50.651952    5304 memory_manager.go:365] "RemoveStaleState removing state" podUID="774e4952-c994-4374-aa62-3d67cc777d5a" containerName="maas-api"
Nov 02 15:49:50.657774 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:49:50.651965    5304 memory_manager.go:365] "RemoveStaleState removing state" podUID="30cc7413-f0e9-4734-bcea-2a731bd75f88" containerName="payload-pre-processing"
Nov 02 15:49:50.657774 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:49:50.651977    5304 memory_manager.go:365] "RemoveStaleState removing state" podUID="9fb3e992-7bb0-497a-a82c-6e396ba241c1" containerName="main"
Nov 02 15:49:50.657774 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:49:50.651987    5304 memory_manager.go:365] "RemoveStaleState removing state" podUID="67da2b58-a0f0-44f6-9258-b418a21e73bc" containerName="istio-proxy"
Nov 02 15:49:50.657774 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:49:50.651997    5304 memory_manager.go:365] "RemoveStaleState removing state" podUID="c19acd3a-5dbf-4da3-b88f-e36284b22243" containerName="istio-proxy"
Nov 02 15:49:50.659887 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:49:50.659861    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e40940-7c4db584c5-jjf2w"
Nov 02 15:49:50.676195 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:49:50.675799    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-e40940-7c4db584c5-jjf2w"]
Nov 02 15:49:50.683940 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:49:50.683913    5304 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-e40940\""
Nov 02 15:49:50.684086 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:49:50.683942    5304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-e40940-dockercfg-v87m7\""
Nov 02 15:49:50.815456 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:49:50.815410    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5pz76\" (UniqueName: \"kubernetes.io/projected/e04e8ce8-dec7-47b8-9eba-a8f87a539790-kube-api-access-5pz76\") pod \"payload-pre-processing-e2e-ipp-a-e40940-7c4db584c5-jjf2w\" (UID: \"e04e8ce8-dec7-47b8-9eba-a8f87a539790\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e40940-7c4db584c5-jjf2w"
Nov 02 15:49:50.815660 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:49:50.815636    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e04e8ce8-dec7-47b8-9eba-a8f87a539790-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-e40940-7c4db584c5-jjf2w\" (UID: \"e04e8ce8-dec7-47b8-9eba-a8f87a539790\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e40940-7c4db584c5-jjf2w"
Nov 02 15:49:50.823761 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:49:50.823720    5304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-e40940-5dc9c99d5d-dwtqt"]
Nov 02 15:49:50.824529 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:49:50.824503    5304 cpu_manager.go:414] "containerMap: removing container" podUID="c19acd3a-5dbf-4da3-b88f-e36284b22243" containerName="istio-proxy"
Nov 02 15:49:50.824529 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:49:50.824527    5304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c19acd3a-5dbf-4da3-b88f-e36284b22243" containerName="istio-proxy"
Nov 02 15:49:50.827299 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:49:50.827275    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-e40940-5dc9c99d5d-dwtqt"
Nov 02 15:49:50.848943 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:49:50.848914    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-e40940-5dc9c99d5d-dwtqt"]
Nov 02 15:49:50.916798 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:49:50.916689    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5pz76\" (UniqueName: \"kubernetes.io/projected/e04e8ce8-dec7-47b8-9eba-a8f87a539790-kube-api-access-5pz76\") pod \"payload-pre-processing-e2e-ipp-a-e40940-7c4db584c5-jjf2w\" (UID: \"e04e8ce8-dec7-47b8-9eba-a8f87a539790\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e40940-7c4db584c5-jjf2w"
Nov 02 15:49:50.916995 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:49:50.916915    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e04e8ce8-dec7-47b8-9eba-a8f87a539790-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-e40940-7c4db584c5-jjf2w\" (UID: \"e04e8ce8-dec7-47b8-9eba-a8f87a539790\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e40940-7c4db584c5-jjf2w"
Nov 02 15:49:50.917629 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:49:50.917610    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e04e8ce8-dec7-47b8-9eba-a8f87a539790-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-e40940-7c4db584c5-jjf2w\" (UID: \"e04e8ce8-dec7-47b8-9eba-a8f87a539790\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e40940-7c4db584c5-jjf2w"
Nov 02 15:49:50.941096 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:49:50.941059    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5pz76\" (UniqueName: \"kubernetes.io/projected/e04e8ce8-dec7-47b8-9eba-a8f87a539790-kube-api-access-5pz76\") pod \"payload-pre-processing-e2e-ipp-a-e40940-7c4db584c5-jjf2w\" (UID: \"e04e8ce8-dec7-47b8-9eba-a8f87a539790\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e40940-7c4db584c5-jjf2w"
Nov 02 15:49:50.985715 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:49:50.985676    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e40940-7c4db584c5-jjf2w"
Nov 02 15:49:51.018029 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:49:51.017976    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wmw6l\" (UniqueName: \"kubernetes.io/projected/99313de5-a08c-4544-86ec-54aeedfe4ec2-kube-api-access-wmw6l\") pod \"payload-processing-e2e-ipp-a-e40940-5dc9c99d5d-dwtqt\" (UID: \"99313de5-a08c-4544-86ec-54aeedfe4ec2\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-e40940-5dc9c99d5d-dwtqt"
Nov 02 15:49:51.018252 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:49:51.018232    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99313de5-a08c-4544-86ec-54aeedfe4ec2-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-e40940-5dc9c99d5d-dwtqt\" (UID: \"99313de5-a08c-4544-86ec-54aeedfe4ec2\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-e40940-5dc9c99d5d-dwtqt"
Nov 02 15:49:51.119788 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:49:51.119751    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wmw6l\" (UniqueName: \"kubernetes.io/projected/99313de5-a08c-4544-86ec-54aeedfe4ec2-kube-api-access-wmw6l\") pod \"payload-processing-e2e-ipp-a-e40940-5dc9c99d5d-dwtqt\" (UID: \"99313de5-a08c-4544-86ec-54aeedfe4ec2\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-e40940-5dc9c99d5d-dwtqt"
Nov 02 15:49:51.119972 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:49:51.119860    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99313de5-a08c-4544-86ec-54aeedfe4ec2-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-e40940-5dc9c99d5d-dwtqt\" (UID: \"99313de5-a08c-4544-86ec-54aeedfe4ec2\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-e40940-5dc9c99d5d-dwtqt"
Nov 02 15:49:51.120671 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:49:51.120650    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99313de5-a08c-4544-86ec-54aeedfe4ec2-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-e40940-5dc9c99d5d-dwtqt\" (UID: \"99313de5-a08c-4544-86ec-54aeedfe4ec2\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-e40940-5dc9c99d5d-dwtqt"
Nov 02 15:49:51.143700 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:49:51.143665    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wmw6l\" (UniqueName: \"kubernetes.io/projected/99313de5-a08c-4544-86ec-54aeedfe4ec2-kube-api-access-wmw6l\") pod \"payload-processing-e2e-ipp-a-e40940-5dc9c99d5d-dwtqt\" (UID: \"99313de5-a08c-4544-86ec-54aeedfe4ec2\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-e40940-5dc9c99d5d-dwtqt"
Nov 02 15:49:51.146082 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:49:51.146057    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-e40940-5dc9c99d5d-dwtqt"
Nov 02 15:49:51.291620 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:49:51.291583    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-e40940-7c4db584c5-jjf2w"]
Nov 02 15:49:51.295079 ip-10-0-135-138 kubenswrapper[5304]: W1102 15:49:51.295048    5304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode04e8ce8_dec7_47b8_9eba_a8f87a539790.slice/crio-0bd8b5ed7ce54fa6e95e75444012cebf7ea7f5b8d6cbece15a496ca3a5b16d62 WatchSource:0}: Error finding container 0bd8b5ed7ce54fa6e95e75444012cebf7ea7f5b8d6cbece15a496ca3a5b16d62: Status 404 returned error can't find the container with id 0bd8b5ed7ce54fa6e95e75444012cebf7ea7f5b8d6cbece15a496ca3a5b16d62
Nov 02 15:49:51.451147 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:49:51.450955    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-e40940-5dc9c99d5d-dwtqt"]
Nov 02 15:49:51.454163 ip-10-0-135-138 kubenswrapper[5304]: W1102 15:49:51.454117    5304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99313de5_a08c_4544_86ec_54aeedfe4ec2.slice/crio-a6705673c26520a7ce70104144fd45dbcc62d23c88f51ff4722d300142f1df06 WatchSource:0}: Error finding container a6705673c26520a7ce70104144fd45dbcc62d23c88f51ff4722d300142f1df06: Status 404 returned error can't find the container with id a6705673c26520a7ce70104144fd45dbcc62d23c88f51ff4722d300142f1df06
Nov 02 15:49:51.739368 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:49:51.739273    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-e40940-5dc9c99d5d-dwtqt" event={"ID":"99313de5-a08c-4544-86ec-54aeedfe4ec2","Type":"ContainerStarted","Data":"0c1481f6c3b5b393817cbb1bb02ff827c5918c40deaeb074ace187c0b17bf0da"}
Nov 02 15:49:51.739368 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:49:51.739348    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-e40940-5dc9c99d5d-dwtqt"
Nov 02 15:49:51.739893 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:49:51.739371    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-e40940-5dc9c99d5d-dwtqt" event={"ID":"99313de5-a08c-4544-86ec-54aeedfe4ec2","Type":"ContainerStarted","Data":"a6705673c26520a7ce70104144fd45dbcc62d23c88f51ff4722d300142f1df06"}
Nov 02 15:49:51.741247 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:49:51.741212    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e40940-7c4db584c5-jjf2w" event={"ID":"e04e8ce8-dec7-47b8-9eba-a8f87a539790","Type":"ContainerStarted","Data":"88f474c9c2c74216db260321e1bab527903d86129dd2a5298ee3c518b266fab9"}
Nov 02 15:49:51.741406 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:49:51.741256    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e40940-7c4db584c5-jjf2w" event={"ID":"e04e8ce8-dec7-47b8-9eba-a8f87a539790","Type":"ContainerStarted","Data":"0bd8b5ed7ce54fa6e95e75444012cebf7ea7f5b8d6cbece15a496ca3a5b16d62"}
Nov 02 15:49:51.741406 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:49:51.741362    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e40940-7c4db584c5-jjf2w"
Nov 02 15:49:51.781611 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:49:51.781554    5304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-e40940-5dc9c99d5d-dwtqt" podStartSLOduration=1.78154005 podStartE2EDuration="1.78154005s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:49:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:49:51.780299692 +0000 UTC m=+3191.774124101" watchObservedRunningTime="2026-11-02 15:49:51.78154005 +0000 UTC m=+3191.775364459"
Nov 02 15:49:51.822386 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:49:51.822318    5304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e40940-7c4db584c5-jjf2w" podStartSLOduration=1.8222970360000001 podStartE2EDuration="1.822297036s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:49:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:49:51.820321418 +0000 UTC m=+3191.814145830" watchObservedRunningTime="2026-11-02 15:49:51.822297036 +0000 UTC m=+3191.816121445"
Nov 02 15:49:54.799406 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:49:54.799364    5304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k6fjk" containerName="extractor"
Nov 02 15:50:02.750391 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:50:02.750351    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e40940-7c4db584c5-jjf2w"
Nov 02 15:50:02.751017 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:50:02.750750    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-e40940-5dc9c99d5d-dwtqt"
Nov 02 15:50:07.448626 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:50:07.448580    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-8537e9-b666465c7-58z69"]
Nov 02 15:50:10.030281 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:50:10.030244    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d30c0729-9597d"]
Nov 02 15:50:10.041809 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:50:10.041777    5304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d30c0729-9597d"]
Nov 02 15:50:10.119520 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:50:10.119481    5304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="094c52e5-8049-44dc-a80d-80b5444933ed" path="/var/lib/kubelet/pods/094c52e5-8049-44dc-a80d-80b5444933ed/volumes"
Nov 02 15:50:11.933863 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:50:11.933820    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-0a7163-8dd8847c5-5s25z"]
Nov 02 15:50:12.449532 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:50:12.449488    5304 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-8537e9-b666465c7-58z69" podUID="f366f832-d3cd-402e-875c-a23629f7c347" containerName="payload-processing" containerID="cri-o://66087bdb01289a7b8670c3b983ef04416f9b38c84f5c869b3e8031561b7f414b" gracePeriod=25
Nov 02 15:50:12.776296 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:50:12.776269    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-8537e9-b666465c7-58z69"
Nov 02 15:50:12.879375 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:50:12.879338    5304 generic.go:356] "Generic (PLEG): container finished" podID="f366f832-d3cd-402e-875c-a23629f7c347" containerID="66087bdb01289a7b8670c3b983ef04416f9b38c84f5c869b3e8031561b7f414b" exitCode=0
Nov 02 15:50:12.879532 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:50:12.879396    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-8537e9-b666465c7-58z69"
Nov 02 15:50:12.879532 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:50:12.879422    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-8537e9-b666465c7-58z69" event={"ID":"f366f832-d3cd-402e-875c-a23629f7c347","Type":"ContainerDied","Data":"66087bdb01289a7b8670c3b983ef04416f9b38c84f5c869b3e8031561b7f414b"}
Nov 02 15:50:12.879532 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:50:12.879496    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-8537e9-b666465c7-58z69" event={"ID":"f366f832-d3cd-402e-875c-a23629f7c347","Type":"ContainerDied","Data":"77c4fcb92c80da4394f8ea0c36784a45841bbb9401291b4dcb7cf5b8484ba778"}
Nov 02 15:50:12.879532 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:50:12.879530    5304 scope.go:122] "RemoveContainer" containerID="66087bdb01289a7b8670c3b983ef04416f9b38c84f5c869b3e8031561b7f414b"
Nov 02 15:50:12.888761 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:50:12.888726    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f366f832-d3cd-402e-875c-a23629f7c347-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f366f832-d3cd-402e-875c-a23629f7c347-plugins-config-volume\") pod \"f366f832-d3cd-402e-875c-a23629f7c347\" (UID: \"f366f832-d3cd-402e-875c-a23629f7c347\") "
Nov 02 15:50:12.888979 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:50:12.888960    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f366f832-d3cd-402e-875c-a23629f7c347-kube-api-access-2h9tz\" (UniqueName: \"kubernetes.io/projected/f366f832-d3cd-402e-875c-a23629f7c347-kube-api-access-2h9tz\") pod \"f366f832-d3cd-402e-875c-a23629f7c347\" (UID: \"f366f832-d3cd-402e-875c-a23629f7c347\") "
Nov 02 15:50:12.889150 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:50:12.889107    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f366f832-d3cd-402e-875c-a23629f7c347-plugins-config-volume" pod "f366f832-d3cd-402e-875c-a23629f7c347" (UID: "f366f832-d3cd-402e-875c-a23629f7c347"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:50:12.889511 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:50:12.889490    5304 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f366f832-d3cd-402e-875c-a23629f7c347-plugins-config-volume\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:50:12.890598 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:50:12.890580    5304 scope.go:122] "RemoveContainer" containerID="66087bdb01289a7b8670c3b983ef04416f9b38c84f5c869b3e8031561b7f414b"
Nov 02 15:50:12.890938 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:50:12.890915    5304 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"66087bdb01289a7b8670c3b983ef04416f9b38c84f5c869b3e8031561b7f414b\": container with ID starting with 66087bdb01289a7b8670c3b983ef04416f9b38c84f5c869b3e8031561b7f414b not found: ID does not exist" containerID="66087bdb01289a7b8670c3b983ef04416f9b38c84f5c869b3e8031561b7f414b"
Nov 02 15:50:12.891001 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:50:12.890947    5304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66087bdb01289a7b8670c3b983ef04416f9b38c84f5c869b3e8031561b7f414b"} err="failed to get container status \"66087bdb01289a7b8670c3b983ef04416f9b38c84f5c869b3e8031561b7f414b\": rpc error: code = NotFound desc = could not find container \"66087bdb01289a7b8670c3b983ef04416f9b38c84f5c869b3e8031561b7f414b\": container with ID starting with 66087bdb01289a7b8670c3b983ef04416f9b38c84f5c869b3e8031561b7f414b not found: ID does not exist"
Nov 02 15:50:12.891688 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:50:12.891672    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f366f832-d3cd-402e-875c-a23629f7c347-kube-api-access-2h9tz" pod "f366f832-d3cd-402e-875c-a23629f7c347" (UID: "f366f832-d3cd-402e-875c-a23629f7c347"). InnerVolumeSpecName "kube-api-access-2h9tz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:50:12.990908 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:50:12.990876    5304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2h9tz\" (UniqueName: \"kubernetes.io/projected/f366f832-d3cd-402e-875c-a23629f7c347-kube-api-access-2h9tz\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:50:13.247116 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:50:13.247080    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-8537e9-b666465c7-58z69"]
Nov 02 15:50:13.249959 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:50:13.249932    5304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-8537e9-b666465c7-58z69"]
Nov 02 15:50:14.218622 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:50:14.218581    5304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f366f832-d3cd-402e-875c-a23629f7c347" path="/var/lib/kubelet/pods/f366f832-d3cd-402e-875c-a23629f7c347/volumes"
Nov 02 15:50:16.935175 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:50:16.935109    5304 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-0a7163-8dd8847c5-5s25z" podUID="8e3715e3-1d26-4abf-ad77-6cb9cb70c349" containerName="payload-pre-processing" containerID="cri-o://7dbe891d97b673b8c7dbbf6e8c33f8470fb41bf72e172021845dcf6c75f881ce" gracePeriod=25
Nov 02 15:50:22.030276 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:50:22.030241    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-16c3c8ee-l2z4h"]
Nov 02 15:50:22.042098 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:50:22.042064    5304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-16c3c8ee-l2z4h"]
Nov 02 15:50:22.119322 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:50:22.119289    5304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7cc94d9d-b85d-4bce-910e-471f19291c5f" path="/var/lib/kubelet/pods/7cc94d9d-b85d-4bce-910e-471f19291c5f/volumes"
Nov 02 15:50:23.795057 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:50:23.795018    5304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-af33a6-openshift-default-759865cbb9-rzw5f"]
Nov 02 15:50:23.796549 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:50:23.796516    5304 cpu_manager.go:414] "containerMap: removing container" podUID="f366f832-d3cd-402e-875c-a23629f7c347" containerName="payload-processing"
Nov 02 15:50:23.796549 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:50:23.796542    5304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f366f832-d3cd-402e-875c-a23629f7c347" containerName="payload-processing"
Nov 02 15:50:23.796808 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:50:23.796793    5304 memory_manager.go:365] "RemoveStaleState removing state" podUID="f366f832-d3cd-402e-875c-a23629f7c347" containerName="payload-processing"
Nov 02 15:50:23.803142 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:50:23.803105    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-af33a6-openshift-default-759865cbb9-rzw5f"
Nov 02 15:50:23.823081 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:50:23.823053    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-af33a6-openshift-default-759865cbb9-rzw5f"]
Nov 02 15:50:23.831488 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:50:23.831449    5304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-af33a6-openshift-default-dockercfg-9jk49\""
Nov 02 15:50:23.890259 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:50:23.890222    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bdab310f-f084-4e60-9b19-3fd5e0b6a953-workload-certs\") pod \"e2e-ipp-b-af33a6-openshift-default-759865cbb9-rzw5f\" (UID: \"bdab310f-f084-4e60-9b19-3fd5e0b6a953\") " pod="openshift-ingress/e2e-ipp-b-af33a6-openshift-default-759865cbb9-rzw5f"
Nov 02 15:50:23.890471 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:50:23.890279    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bdab310f-f084-4e60-9b19-3fd5e0b6a953-istio-envoy\") pod \"e2e-ipp-b-af33a6-openshift-default-759865cbb9-rzw5f\" (UID: \"bdab310f-f084-4e60-9b19-3fd5e0b6a953\") " pod="openshift-ingress/e2e-ipp-b-af33a6-openshift-default-759865cbb9-rzw5f"
Nov 02 15:50:23.890471 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:50:23.890380    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdab310f-f084-4e60-9b19-3fd5e0b6a953-workload-socket\") pod \"e2e-ipp-b-af33a6-openshift-default-759865cbb9-rzw5f\" (UID: \"bdab310f-f084-4e60-9b19-3fd5e0b6a953\") " pod="openshift-ingress/e2e-ipp-b-af33a6-openshift-default-759865cbb9-rzw5f"
Nov 02 15:50:23.890471 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:50:23.890425    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bdab310f-f084-4e60-9b19-3fd5e0b6a953-istio-data\") pod \"e2e-ipp-b-af33a6-openshift-default-759865cbb9-rzw5f\" (UID: \"bdab310f-f084-4e60-9b19-3fd5e0b6a953\") " pod="openshift-ingress/e2e-ipp-b-af33a6-openshift-default-759865cbb9-rzw5f"
Nov 02 15:50:23.890658 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:50:23.890534    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bdab310f-f084-4e60-9b19-3fd5e0b6a953-istiod-ca-cert\") pod \"e2e-ipp-b-af33a6-openshift-default-759865cbb9-rzw5f\" (UID: \"bdab310f-f084-4e60-9b19-3fd5e0b6a953\") " pod="openshift-ingress/e2e-ipp-b-af33a6-openshift-default-759865cbb9-rzw5f"
Nov 02 15:50:23.890658 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:50:23.890591    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sw5hb\" (UniqueName: \"kubernetes.io/projected/bdab310f-f084-4e60-9b19-3fd5e0b6a953-kube-api-access-sw5hb\") pod \"e2e-ipp-b-af33a6-openshift-default-759865cbb9-rzw5f\" (UID: \"bdab310f-f084-4e60-9b19-3fd5e0b6a953\") " pod="openshift-ingress/e2e-ipp-b-af33a6-openshift-default-759865cbb9-rzw5f"
Nov 02 15:50:23.890658 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:50:23.890639    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bdab310f-f084-4e60-9b19-3fd5e0b6a953-istio-podinfo\") pod \"e2e-ipp-b-af33a6-openshift-default-759865cbb9-rzw5f\" (UID: \"bdab310f-f084-4e60-9b19-3fd5e0b6a953\") " pod="openshift-ingress/e2e-ipp-b-af33a6-openshift-default-759865cbb9-rzw5f"
Nov 02 15:50:23.890828 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:50:23.890693    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdab310f-f084-4e60-9b19-3fd5e0b6a953-credential-socket\") pod \"e2e-ipp-b-af33a6-openshift-default-759865cbb9-rzw5f\" (UID: \"bdab310f-f084-4e60-9b19-3fd5e0b6a953\") " pod="openshift-ingress/e2e-ipp-b-af33a6-openshift-default-759865cbb9-rzw5f"
Nov 02 15:50:23.890828 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:50:23.890753    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bdab310f-f084-4e60-9b19-3fd5e0b6a953-istio-token\") pod \"e2e-ipp-b-af33a6-openshift-default-759865cbb9-rzw5f\" (UID: \"bdab310f-f084-4e60-9b19-3fd5e0b6a953\") " pod="openshift-ingress/e2e-ipp-b-af33a6-openshift-default-759865cbb9-rzw5f"
Nov 02 15:50:23.992100 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:50:23.992066    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bdab310f-f084-4e60-9b19-3fd5e0b6a953-istio-podinfo\") pod \"e2e-ipp-b-af33a6-openshift-default-759865cbb9-rzw5f\" (UID: \"bdab310f-f084-4e60-9b19-3fd5e0b6a953\") " pod="openshift-ingress/e2e-ipp-b-af33a6-openshift-default-759865cbb9-rzw5f"
Nov 02 15:50:23.992317 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:50:23.992162    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdab310f-f084-4e60-9b19-3fd5e0b6a953-credential-socket\") pod \"e2e-ipp-b-af33a6-openshift-default-759865cbb9-rzw5f\" (UID: \"bdab310f-f084-4e60-9b19-3fd5e0b6a953\") " pod="openshift-ingress/e2e-ipp-b-af33a6-openshift-default-759865cbb9-rzw5f"
Nov 02 15:50:23.992317 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:50:23.992197    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bdab310f-f084-4e60-9b19-3fd5e0b6a953-istio-token\") pod \"e2e-ipp-b-af33a6-openshift-default-759865cbb9-rzw5f\" (UID: \"bdab310f-f084-4e60-9b19-3fd5e0b6a953\") " pod="openshift-ingress/e2e-ipp-b-af33a6-openshift-default-759865cbb9-rzw5f"
Nov 02 15:50:23.992317 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:50:23.992261    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bdab310f-f084-4e60-9b19-3fd5e0b6a953-workload-certs\") pod \"e2e-ipp-b-af33a6-openshift-default-759865cbb9-rzw5f\" (UID: \"bdab310f-f084-4e60-9b19-3fd5e0b6a953\") " pod="openshift-ingress/e2e-ipp-b-af33a6-openshift-default-759865cbb9-rzw5f"
Nov 02 15:50:23.992487 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:50:23.992323    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bdab310f-f084-4e60-9b19-3fd5e0b6a953-istio-envoy\") pod \"e2e-ipp-b-af33a6-openshift-default-759865cbb9-rzw5f\" (UID: \"bdab310f-f084-4e60-9b19-3fd5e0b6a953\") " pod="openshift-ingress/e2e-ipp-b-af33a6-openshift-default-759865cbb9-rzw5f"
Nov 02 15:50:23.992487 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:50:23.992377    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdab310f-f084-4e60-9b19-3fd5e0b6a953-workload-socket\") pod \"e2e-ipp-b-af33a6-openshift-default-759865cbb9-rzw5f\" (UID: \"bdab310f-f084-4e60-9b19-3fd5e0b6a953\") " pod="openshift-ingress/e2e-ipp-b-af33a6-openshift-default-759865cbb9-rzw5f"
Nov 02 15:50:23.992487 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:50:23.992420    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bdab310f-f084-4e60-9b19-3fd5e0b6a953-istio-data\") pod \"e2e-ipp-b-af33a6-openshift-default-759865cbb9-rzw5f\" (UID: \"bdab310f-f084-4e60-9b19-3fd5e0b6a953\") " pod="openshift-ingress/e2e-ipp-b-af33a6-openshift-default-759865cbb9-rzw5f"
Nov 02 15:50:23.992675 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:50:23.992525    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bdab310f-f084-4e60-9b19-3fd5e0b6a953-istiod-ca-cert\") pod \"e2e-ipp-b-af33a6-openshift-default-759865cbb9-rzw5f\" (UID: \"bdab310f-f084-4e60-9b19-3fd5e0b6a953\") " pod="openshift-ingress/e2e-ipp-b-af33a6-openshift-default-759865cbb9-rzw5f"
Nov 02 15:50:23.992675 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:50:23.992621    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sw5hb\" (UniqueName: \"kubernetes.io/projected/bdab310f-f084-4e60-9b19-3fd5e0b6a953-kube-api-access-sw5hb\") pod \"e2e-ipp-b-af33a6-openshift-default-759865cbb9-rzw5f\" (UID: \"bdab310f-f084-4e60-9b19-3fd5e0b6a953\") " pod="openshift-ingress/e2e-ipp-b-af33a6-openshift-default-759865cbb9-rzw5f"
Nov 02 15:50:23.992789 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:50:23.992695    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdab310f-f084-4e60-9b19-3fd5e0b6a953-credential-socket\") pod \"e2e-ipp-b-af33a6-openshift-default-759865cbb9-rzw5f\" (UID: \"bdab310f-f084-4e60-9b19-3fd5e0b6a953\") " pod="openshift-ingress/e2e-ipp-b-af33a6-openshift-default-759865cbb9-rzw5f"
Nov 02 15:50:23.992789 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:50:23.992735    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bdab310f-f084-4e60-9b19-3fd5e0b6a953-workload-certs\") pod \"e2e-ipp-b-af33a6-openshift-default-759865cbb9-rzw5f\" (UID: \"bdab310f-f084-4e60-9b19-3fd5e0b6a953\") " pod="openshift-ingress/e2e-ipp-b-af33a6-openshift-default-759865cbb9-rzw5f"
Nov 02 15:50:23.992964 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:50:23.992942    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bdab310f-f084-4e60-9b19-3fd5e0b6a953-istio-data\") pod \"e2e-ipp-b-af33a6-openshift-default-759865cbb9-rzw5f\" (UID: \"bdab310f-f084-4e60-9b19-3fd5e0b6a953\") " pod="openshift-ingress/e2e-ipp-b-af33a6-openshift-default-759865cbb9-rzw5f"
Nov 02 15:50:23.993028 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:50:23.993000    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdab310f-f084-4e60-9b19-3fd5e0b6a953-workload-socket\") pod \"e2e-ipp-b-af33a6-openshift-default-759865cbb9-rzw5f\" (UID: \"bdab310f-f084-4e60-9b19-3fd5e0b6a953\") " pod="openshift-ingress/e2e-ipp-b-af33a6-openshift-default-759865cbb9-rzw5f"
Nov 02 15:50:23.993316 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:50:23.993296    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bdab310f-f084-4e60-9b19-3fd5e0b6a953-istiod-ca-cert\") pod \"e2e-ipp-b-af33a6-openshift-default-759865cbb9-rzw5f\" (UID: \"bdab310f-f084-4e60-9b19-3fd5e0b6a953\") " pod="openshift-ingress/e2e-ipp-b-af33a6-openshift-default-759865cbb9-rzw5f"
Nov 02 15:50:23.994607 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:50:23.994582    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bdab310f-f084-4e60-9b19-3fd5e0b6a953-istio-envoy\") pod \"e2e-ipp-b-af33a6-openshift-default-759865cbb9-rzw5f\" (UID: \"bdab310f-f084-4e60-9b19-3fd5e0b6a953\") " pod="openshift-ingress/e2e-ipp-b-af33a6-openshift-default-759865cbb9-rzw5f"
Nov 02 15:50:23.994855 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:50:23.994836    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bdab310f-f084-4e60-9b19-3fd5e0b6a953-istio-podinfo\") pod \"e2e-ipp-b-af33a6-openshift-default-759865cbb9-rzw5f\" (UID: \"bdab310f-f084-4e60-9b19-3fd5e0b6a953\") " pod="openshift-ingress/e2e-ipp-b-af33a6-openshift-default-759865cbb9-rzw5f"
Nov 02 15:50:24.035109 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:50:24.035064    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-0a7163-8dd8847c5-5s25z" podUID="8e3715e3-1d26-4abf-ad77-6cb9cb70c349" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 15:50:24.036143 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:50:24.036098    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bdab310f-f084-4e60-9b19-3fd5e0b6a953-istio-token\") pod \"e2e-ipp-b-af33a6-openshift-default-759865cbb9-rzw5f\" (UID: \"bdab310f-f084-4e60-9b19-3fd5e0b6a953\") " pod="openshift-ingress/e2e-ipp-b-af33a6-openshift-default-759865cbb9-rzw5f"
Nov 02 15:50:24.036912 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:50:24.036886    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sw5hb\" (UniqueName: \"kubernetes.io/projected/bdab310f-f084-4e60-9b19-3fd5e0b6a953-kube-api-access-sw5hb\") pod \"e2e-ipp-b-af33a6-openshift-default-759865cbb9-rzw5f\" (UID: \"bdab310f-f084-4e60-9b19-3fd5e0b6a953\") " pod="openshift-ingress/e2e-ipp-b-af33a6-openshift-default-759865cbb9-rzw5f"
Nov 02 15:50:24.118344 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:50:24.118312    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-af33a6-openshift-default-759865cbb9-rzw5f"
Nov 02 15:50:24.286597 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:50:24.286558    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-af33a6-59b4d9dc95-v9pl6"]
Nov 02 15:50:24.456429 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:50:24.456389    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-af33a6-openshift-default-759865cbb9-rzw5f"]
Nov 02 15:50:24.458329 ip-10-0-135-138 kubenswrapper[5304]: W1102 15:50:24.458288    5304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbdab310f_f084_4e60_9b19_3fd5e0b6a953.slice/crio-f8a04414ae93734bf8a4f73850c089df32d00998d80b631b6b74fdafae59a025 WatchSource:0}: Error finding container f8a04414ae93734bf8a4f73850c089df32d00998d80b631b6b74fdafae59a025: Status 404 returned error can't find the container with id f8a04414ae93734bf8a4f73850c089df32d00998d80b631b6b74fdafae59a025
Nov 02 15:50:24.461283 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:50:24.461246    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:50:24.461370 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:50:24.461316    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:50:24.461370 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:50:24.461345    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:50:24.807639 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:50:24.807533    5304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k6fjk" containerName="extractor"
Nov 02 15:50:25.035803 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:50:25.035754    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-af33a6-openshift-default-759865cbb9-rzw5f" event={"ID":"bdab310f-f084-4e60-9b19-3fd5e0b6a953","Type":"ContainerStarted","Data":"9d0983157d56b071b0f8f59c7711c0c35e654feb059fa2e2b5150c445b1b25b5"}
Nov 02 15:50:25.035995 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:50:25.035843    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-af33a6-openshift-default-759865cbb9-rzw5f" event={"ID":"bdab310f-f084-4e60-9b19-3fd5e0b6a953","Type":"ContainerStarted","Data":"f8a04414ae93734bf8a4f73850c089df32d00998d80b631b6b74fdafae59a025"}
Nov 02 15:50:25.105442 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:50:25.103731    5304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-af33a6-openshift-default-759865cbb9-rzw5f" podStartSLOduration=2.103713669 podStartE2EDuration="2.103713669s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:50:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:50:25.093195173 +0000 UTC m=+3225.087019585" watchObservedRunningTime="2026-11-02 15:50:25.103713669 +0000 UTC m=+3225.097538080"
Nov 02 15:50:25.123354 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:50:25.123317    5304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-af33a6-openshift-default-759865cbb9-rzw5f"
Nov 02 15:50:25.125568 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:50:25.125449    5304 patch_prober.go:29] interesting pod/e2e-ipp-b-af33a6-openshift-default-759865cbb9-rzw5f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.219:15021/healthz/ready\": dial tcp 10.133.0.219:15021: connect: connection refused" start-of-body=
Nov 02 15:50:25.125568 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:50:25.125502    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-af33a6-openshift-default-759865cbb9-rzw5f" podUID="bdab310f-f084-4e60-9b19-3fd5e0b6a953" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.219:15021/healthz/ready\": dial tcp 10.133.0.219:15021: connect: connection refused"
Nov 02 15:50:26.119403 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:50:26.119370    5304 patch_prober.go:29] interesting pod/e2e-ipp-b-af33a6-openshift-default-759865cbb9-rzw5f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.219:15021/healthz/ready\": dial tcp 10.133.0.219:15021: connect: connection refused" start-of-body=
Nov 02 15:50:26.119812 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:50:26.119424    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-af33a6-openshift-default-759865cbb9-rzw5f" podUID="bdab310f-f084-4e60-9b19-3fd5e0b6a953" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.219:15021/healthz/ready\": dial tcp 10.133.0.219:15021: connect: connection refused"
Nov 02 15:50:27.061157 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:50:27.061083    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3bbd4709-nbnqr"]
Nov 02 15:50:27.070896 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:50:27.070860    5304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3bbd4709-nbnqr"]
Nov 02 15:50:27.119528 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:50:27.119495    5304 patch_prober.go:29] interesting pod/e2e-ipp-b-af33a6-openshift-default-759865cbb9-rzw5f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.219:15021/healthz/ready\": dial tcp 10.133.0.219:15021: connect: connection refused" start-of-body=
Nov 02 15:50:27.120013 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:50:27.119555    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-af33a6-openshift-default-759865cbb9-rzw5f" podUID="bdab310f-f084-4e60-9b19-3fd5e0b6a953" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.219:15021/healthz/ready\": dial tcp 10.133.0.219:15021: connect: connection refused"
Nov 02 15:50:28.118830 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:50:28.118793    5304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="96b6b761-e07b-41be-8460-f0b68cd55d4a" path="/var/lib/kubelet/pods/96b6b761-e07b-41be-8460-f0b68cd55d4a/volumes"
Nov 02 15:50:28.119465 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:50:28.119445    5304 patch_prober.go:29] interesting pod/e2e-ipp-b-af33a6-openshift-default-759865cbb9-rzw5f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.219:15021/healthz/ready\": dial tcp 10.133.0.219:15021: connect: connection refused" start-of-body=
Nov 02 15:50:28.119522 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:50:28.119498    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-af33a6-openshift-default-759865cbb9-rzw5f" podUID="bdab310f-f084-4e60-9b19-3fd5e0b6a953" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.219:15021/healthz/ready\": dial tcp 10.133.0.219:15021: connect: connection refused"
Nov 02 15:50:29.119557 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:50:29.119525    5304 patch_prober.go:29] interesting pod/e2e-ipp-b-af33a6-openshift-default-759865cbb9-rzw5f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.219:15021/healthz/ready\": dial tcp 10.133.0.219:15021: connect: connection refused" start-of-body=
Nov 02 15:50:29.119960 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:50:29.119577    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-af33a6-openshift-default-759865cbb9-rzw5f" podUID="bdab310f-f084-4e60-9b19-3fd5e0b6a953" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.219:15021/healthz/ready\": dial tcp 10.133.0.219:15021: connect: connection refused"
Nov 02 15:50:29.289043 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:50:29.288065    5304 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-af33a6-59b4d9dc95-v9pl6" podUID="888a0ac9-4645-4fca-bc53-022beb9f484f" containerName="payload-processing" containerID="cri-o://dd08f36bab9e1ecad2bc2d0e595679a301474467ed6f145ee6d41a25b91ded49" gracePeriod=25
Nov 02 15:50:29.600305 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:50:29.600273    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-af33a6-59b4d9dc95-v9pl6"
Nov 02 15:50:29.709114 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:50:29.709025    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/888a0ac9-4645-4fca-bc53-022beb9f484f-kube-api-access-vngdw\" (UniqueName: \"kubernetes.io/projected/888a0ac9-4645-4fca-bc53-022beb9f484f-kube-api-access-vngdw\") pod \"888a0ac9-4645-4fca-bc53-022beb9f484f\" (UID: \"888a0ac9-4645-4fca-bc53-022beb9f484f\") "
Nov 02 15:50:29.709343 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:50:29.709195    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/888a0ac9-4645-4fca-bc53-022beb9f484f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/888a0ac9-4645-4fca-bc53-022beb9f484f-plugins-config-volume\") pod \"888a0ac9-4645-4fca-bc53-022beb9f484f\" (UID: \"888a0ac9-4645-4fca-bc53-022beb9f484f\") "
Nov 02 15:50:29.709640 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:50:29.709613    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/888a0ac9-4645-4fca-bc53-022beb9f484f-plugins-config-volume" pod "888a0ac9-4645-4fca-bc53-022beb9f484f" (UID: "888a0ac9-4645-4fca-bc53-022beb9f484f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:50:29.709734 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:50:29.709697    5304 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/888a0ac9-4645-4fca-bc53-022beb9f484f-plugins-config-volume\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:50:29.711424 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:50:29.711389    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/888a0ac9-4645-4fca-bc53-022beb9f484f-kube-api-access-vngdw" pod "888a0ac9-4645-4fca-bc53-022beb9f484f" (UID: "888a0ac9-4645-4fca-bc53-022beb9f484f"). InnerVolumeSpecName "kube-api-access-vngdw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:50:29.811585 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:50:29.811540    5304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vngdw\" (UniqueName: \"kubernetes.io/projected/888a0ac9-4645-4fca-bc53-022beb9f484f-kube-api-access-vngdw\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:50:30.094959 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:50:30.094922    5304 generic.go:356] "Generic (PLEG): container finished" podID="888a0ac9-4645-4fca-bc53-022beb9f484f" containerID="dd08f36bab9e1ecad2bc2d0e595679a301474467ed6f145ee6d41a25b91ded49" exitCode=0
Nov 02 15:50:30.095192 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:50:30.094978    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-af33a6-59b4d9dc95-v9pl6"
Nov 02 15:50:30.095192 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:50:30.095001    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-af33a6-59b4d9dc95-v9pl6" event={"ID":"888a0ac9-4645-4fca-bc53-022beb9f484f","Type":"ContainerDied","Data":"dd08f36bab9e1ecad2bc2d0e595679a301474467ed6f145ee6d41a25b91ded49"}
Nov 02 15:50:30.095192 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:50:30.095053    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-af33a6-59b4d9dc95-v9pl6" event={"ID":"888a0ac9-4645-4fca-bc53-022beb9f484f","Type":"ContainerDied","Data":"c6ee1cf6632089dda0357a20d3efea8238d8046cc5f1b3a437ab68f3e684273d"}
Nov 02 15:50:30.095192 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:50:30.095075    5304 scope.go:122] "RemoveContainer" containerID="dd08f36bab9e1ecad2bc2d0e595679a301474467ed6f145ee6d41a25b91ded49"
Nov 02 15:50:30.110263 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:50:30.106857    5304 scope.go:122] "RemoveContainer" containerID="dd08f36bab9e1ecad2bc2d0e595679a301474467ed6f145ee6d41a25b91ded49"
Nov 02 15:50:30.110263 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:50:30.107106    5304 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dd08f36bab9e1ecad2bc2d0e595679a301474467ed6f145ee6d41a25b91ded49\": container with ID starting with dd08f36bab9e1ecad2bc2d0e595679a301474467ed6f145ee6d41a25b91ded49 not found: ID does not exist" containerID="dd08f36bab9e1ecad2bc2d0e595679a301474467ed6f145ee6d41a25b91ded49"
Nov 02 15:50:30.110263 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:50:30.107158    5304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd08f36bab9e1ecad2bc2d0e595679a301474467ed6f145ee6d41a25b91ded49"} err="failed to get container status \"dd08f36bab9e1ecad2bc2d0e595679a301474467ed6f145ee6d41a25b91ded49\": rpc error: code = NotFound desc = could not find container \"dd08f36bab9e1ecad2bc2d0e595679a301474467ed6f145ee6d41a25b91ded49\": container with ID starting with dd08f36bab9e1ecad2bc2d0e595679a301474467ed6f145ee6d41a25b91ded49 not found: ID does not exist"
Nov 02 15:50:30.119362 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:50:30.119261    5304 patch_prober.go:29] interesting pod/e2e-ipp-b-af33a6-openshift-default-759865cbb9-rzw5f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.219:15021/healthz/ready\": dial tcp 10.133.0.219:15021: connect: connection refused" start-of-body=
Nov 02 15:50:30.119362 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:50:30.119297    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-af33a6-openshift-default-759865cbb9-rzw5f" podUID="bdab310f-f084-4e60-9b19-3fd5e0b6a953" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.219:15021/healthz/ready\": dial tcp 10.133.0.219:15021: connect: connection refused"
Nov 02 15:50:30.155898 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:50:30.155865    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-af33a6-59b4d9dc95-v9pl6"]
Nov 02 15:50:30.162117 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:50:30.162087    5304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-af33a6-59b4d9dc95-v9pl6"]
Nov 02 15:50:31.123986 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:50:31.123954    5304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-af33a6-openshift-default-759865cbb9-rzw5f"
Nov 02 15:50:31.124416 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:50:31.124395    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-af33a6-openshift-default-759865cbb9-rzw5f"
Nov 02 15:50:31.125062 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:50:31.125046    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-af33a6-openshift-default-759865cbb9-rzw5f"
Nov 02 15:50:32.119091 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:50:32.119052    5304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="888a0ac9-4645-4fca-bc53-022beb9f484f" path="/var/lib/kubelet/pods/888a0ac9-4645-4fca-bc53-022beb9f484f/volumes"
Nov 02 15:50:34.029230 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:50:34.029186    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-0a7163-8dd8847c5-5s25z" podUID="8e3715e3-1d26-4abf-ad77-6cb9cb70c349" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 15:50:37.143329 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:50:37.143284    5304 generic.go:356] "Generic (PLEG): container finished" podID="8e3715e3-1d26-4abf-ad77-6cb9cb70c349" containerID="7dbe891d97b673b8c7dbbf6e8c33f8470fb41bf72e172021845dcf6c75f881ce" exitCode=0
Nov 02 15:50:37.143854 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:50:37.143356    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-0a7163-8dd8847c5-5s25z" event={"ID":"8e3715e3-1d26-4abf-ad77-6cb9cb70c349","Type":"ContainerDied","Data":"7dbe891d97b673b8c7dbbf6e8c33f8470fb41bf72e172021845dcf6c75f881ce"}
Nov 02 15:50:37.214871 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:50:37.214842    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-0a7163-8dd8847c5-5s25z"
Nov 02 15:50:37.313515 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:50:37.313482    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e3715e3-1d26-4abf-ad77-6cb9cb70c349-kube-api-access-zw8gd\" (UniqueName: \"kubernetes.io/projected/8e3715e3-1d26-4abf-ad77-6cb9cb70c349-kube-api-access-zw8gd\") pod \"8e3715e3-1d26-4abf-ad77-6cb9cb70c349\" (UID: \"8e3715e3-1d26-4abf-ad77-6cb9cb70c349\") "
Nov 02 15:50:37.313737 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:50:37.313608    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e3715e3-1d26-4abf-ad77-6cb9cb70c349-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e3715e3-1d26-4abf-ad77-6cb9cb70c349-plugins-config-volume\") pod \"8e3715e3-1d26-4abf-ad77-6cb9cb70c349\" (UID: \"8e3715e3-1d26-4abf-ad77-6cb9cb70c349\") "
Nov 02 15:50:37.313983 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:50:37.313958    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e3715e3-1d26-4abf-ad77-6cb9cb70c349-plugins-config-volume" pod "8e3715e3-1d26-4abf-ad77-6cb9cb70c349" (UID: "8e3715e3-1d26-4abf-ad77-6cb9cb70c349"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:50:37.314152 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:50:37.314136    5304 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e3715e3-1d26-4abf-ad77-6cb9cb70c349-plugins-config-volume\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:50:37.315696 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:50:37.315673    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e3715e3-1d26-4abf-ad77-6cb9cb70c349-kube-api-access-zw8gd" pod "8e3715e3-1d26-4abf-ad77-6cb9cb70c349" (UID: "8e3715e3-1d26-4abf-ad77-6cb9cb70c349"). InnerVolumeSpecName "kube-api-access-zw8gd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:50:37.415409 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:50:37.415373    5304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zw8gd\" (UniqueName: \"kubernetes.io/projected/8e3715e3-1d26-4abf-ad77-6cb9cb70c349-kube-api-access-zw8gd\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:50:38.191697 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:50:38.191654    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-0a7163-8dd8847c5-5s25z" event={"ID":"8e3715e3-1d26-4abf-ad77-6cb9cb70c349","Type":"ContainerDied","Data":"45acd77c5ba763c2ce0f6389b936e06e0a18c23ef2de18a9c23dd5fcf9381ea6"}
Nov 02 15:50:38.192302 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:50:38.192270    5304 scope.go:122] "RemoveContainer" containerID="7dbe891d97b673b8c7dbbf6e8c33f8470fb41bf72e172021845dcf6c75f881ce"
Nov 02 15:50:38.192533 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:50:38.192513    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-0a7163-8dd8847c5-5s25z"
Nov 02 15:50:38.246278 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:50:38.246245    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-0a7163-8dd8847c5-5s25z"]
Nov 02 15:50:38.257862 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:50:38.257831    5304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-0a7163-8dd8847c5-5s25z"]
Nov 02 15:50:40.121217 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:50:40.121185    5304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e3715e3-1d26-4abf-ad77-6cb9cb70c349" path="/var/lib/kubelet/pods/8e3715e3-1d26-4abf-ad77-6cb9cb70c349/volumes"
Nov 02 15:50:48.977564 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:50:48.977519    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-0a7163-6c989cd9bd-xhgf5"]
Nov 02 15:50:48.978070 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:50:48.977795    5304 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-0a7163-6c989cd9bd-xhgf5" podUID="34a8288b-adc6-424b-a30b-38f023185c86" containerName="maas-api" containerID="cri-o://d9acb6dbd5f54b30c15ce3b101055c388a0b8a35443bf5ac415ca57978ae962b" gracePeriod=30
Nov 02 15:50:49.320469 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:50:49.320379    5304 generic.go:356] "Generic (PLEG): container finished" podID="34a8288b-adc6-424b-a30b-38f023185c86" containerID="d9acb6dbd5f54b30c15ce3b101055c388a0b8a35443bf5ac415ca57978ae962b" exitCode=0
Nov 02 15:50:49.320469 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:50:49.320433    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-0a7163-6c989cd9bd-xhgf5" event={"ID":"34a8288b-adc6-424b-a30b-38f023185c86","Type":"ContainerDied","Data":"d9acb6dbd5f54b30c15ce3b101055c388a0b8a35443bf5ac415ca57978ae962b"}
Nov 02 15:50:49.320469 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:50:49.320481    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-0a7163-6c989cd9bd-xhgf5" event={"ID":"34a8288b-adc6-424b-a30b-38f023185c86","Type":"ContainerDied","Data":"21b8135b8f2ca6ab05e72ed55b4420802c7b68fc52763309066acff413109c35"}
Nov 02 15:50:49.320728 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:50:49.320515    5304 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="21b8135b8f2ca6ab05e72ed55b4420802c7b68fc52763309066acff413109c35"
Nov 02 15:50:49.321111 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:50:49.321090    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-0a7163-6c989cd9bd-xhgf5"
Nov 02 15:50:49.403148 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:50:49.402635    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/34a8288b-adc6-424b-a30b-38f023185c86-metrics-certs\" (UniqueName: \"kubernetes.io/secret/34a8288b-adc6-424b-a30b-38f023185c86-metrics-certs\") pod \"34a8288b-adc6-424b-a30b-38f023185c86\" (UID: \"34a8288b-adc6-424b-a30b-38f023185c86\") "
Nov 02 15:50:49.403148 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:50:49.402796    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/34a8288b-adc6-424b-a30b-38f023185c86-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/34a8288b-adc6-424b-a30b-38f023185c86-openshift-service-ca\") pod \"34a8288b-adc6-424b-a30b-38f023185c86\" (UID: \"34a8288b-adc6-424b-a30b-38f023185c86\") "
Nov 02 15:50:49.403148 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:50:49.402915    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/34a8288b-adc6-424b-a30b-38f023185c86-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/34a8288b-adc6-424b-a30b-38f023185c86-maas-api-tls\") pod \"34a8288b-adc6-424b-a30b-38f023185c86\" (UID: \"34a8288b-adc6-424b-a30b-38f023185c86\") "
Nov 02 15:50:49.403148 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:50:49.402965    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34a8288b-adc6-424b-a30b-38f023185c86-kube-api-access-mzd78\" (UniqueName: \"kubernetes.io/projected/34a8288b-adc6-424b-a30b-38f023185c86-kube-api-access-mzd78\") pod \"34a8288b-adc6-424b-a30b-38f023185c86\" (UID: \"34a8288b-adc6-424b-a30b-38f023185c86\") "
Nov 02 15:50:49.404915 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:50:49.404869    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34a8288b-adc6-424b-a30b-38f023185c86-openshift-service-ca" pod "34a8288b-adc6-424b-a30b-38f023185c86" (UID: "34a8288b-adc6-424b-a30b-38f023185c86"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:50:49.415405 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:50:49.415370    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/34a8288b-adc6-424b-a30b-38f023185c86-metrics-certs" pod "34a8288b-adc6-424b-a30b-38f023185c86" (UID: "34a8288b-adc6-424b-a30b-38f023185c86"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:50:49.458703 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:50:49.455291    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/34a8288b-adc6-424b-a30b-38f023185c86-maas-api-tls" pod "34a8288b-adc6-424b-a30b-38f023185c86" (UID: "34a8288b-adc6-424b-a30b-38f023185c86"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:50:49.459049 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:50:49.459013    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34a8288b-adc6-424b-a30b-38f023185c86-kube-api-access-mzd78" pod "34a8288b-adc6-424b-a30b-38f023185c86" (UID: "34a8288b-adc6-424b-a30b-38f023185c86"). InnerVolumeSpecName "kube-api-access-mzd78". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:50:49.505175 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:50:49.505064    5304 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/34a8288b-adc6-424b-a30b-38f023185c86-maas-api-tls\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:50:49.505175 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:50:49.505101    5304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mzd78\" (UniqueName: \"kubernetes.io/projected/34a8288b-adc6-424b-a30b-38f023185c86-kube-api-access-mzd78\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:50:49.505175 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:50:49.505115    5304 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/34a8288b-adc6-424b-a30b-38f023185c86-metrics-certs\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:50:49.505175 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:50:49.505141    5304 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/34a8288b-adc6-424b-a30b-38f023185c86-openshift-service-ca\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:50:50.326744 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:50:50.326661    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-0a7163-6c989cd9bd-xhgf5"
Nov 02 15:50:50.379106 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:50:50.379065    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-0a7163-6c989cd9bd-xhgf5"]
Nov 02 15:50:50.390640 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:50:50.390610    5304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-0a7163-6c989cd9bd-xhgf5"]
Nov 02 15:50:52.120363 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:50:52.120289    5304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="34a8288b-adc6-424b-a30b-38f023185c86" path="/var/lib/kubelet/pods/34a8288b-adc6-424b-a30b-38f023185c86/volumes"
Nov 02 15:50:54.800000 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:50:54.799958    5304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k6fjk" containerName="extractor"
Nov 02 15:50:56.067665 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:50:56.067624    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-a55a55-59b6968f7d-c496x"]
Nov 02 15:50:56.068222 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:50:56.067916    5304 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-a55a55-59b6968f7d-c496x" podUID="eb8e624a-b173-4fc3-b5b5-4183ebf2eda7" containerName="payload-processing" containerID="cri-o://faba378e91ad9c15eb6272580b5d3f1169f5438d9d07d54300bb8c43aecc74a7" gracePeriod=30
Nov 02 15:50:56.102842 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:50:56.102812    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-a55a55-59db66d795-crnzc"]
Nov 02 15:50:56.103072 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:50:56.103048    5304 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a55a55-59db66d795-crnzc" podUID="d77e8cf8-6cdc-4458-8c0c-564144f06220" containerName="payload-pre-processing" containerID="cri-o://bcd04148bb8837a6d6fb432a8fcc29856dabbddb303cf6ae68f421f1e351de43" gracePeriod=30
Nov 02 15:50:56.369680 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:50:56.369634    5304 generic.go:356] "Generic (PLEG): container finished" podID="d77e8cf8-6cdc-4458-8c0c-564144f06220" containerID="bcd04148bb8837a6d6fb432a8fcc29856dabbddb303cf6ae68f421f1e351de43" exitCode=0
Nov 02 15:50:56.371716 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:50:56.369684    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a55a55-59db66d795-crnzc" event={"ID":"d77e8cf8-6cdc-4458-8c0c-564144f06220","Type":"ContainerDied","Data":"bcd04148bb8837a6d6fb432a8fcc29856dabbddb303cf6ae68f421f1e351de43"}
Nov 02 15:50:56.371838 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:50:56.371804    5304 generic.go:356] "Generic (PLEG): container finished" podID="eb8e624a-b173-4fc3-b5b5-4183ebf2eda7" containerID="faba378e91ad9c15eb6272580b5d3f1169f5438d9d07d54300bb8c43aecc74a7" exitCode=0
Nov 02 15:50:56.371914 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:50:56.371865    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-a55a55-59b6968f7d-c496x" event={"ID":"eb8e624a-b173-4fc3-b5b5-4183ebf2eda7","Type":"ContainerDied","Data":"faba378e91ad9c15eb6272580b5d3f1169f5438d9d07d54300bb8c43aecc74a7"}
Nov 02 15:50:56.475224 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:50:56.472915    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-a55a55-59b6968f7d-c496x"
Nov 02 15:50:56.494687 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:50:56.494650    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a55a55-59db66d795-crnzc"
Nov 02 15:50:56.615073 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:50:56.614992    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d77e8cf8-6cdc-4458-8c0c-564144f06220-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d77e8cf8-6cdc-4458-8c0c-564144f06220-plugins-config-volume\") pod \"d77e8cf8-6cdc-4458-8c0c-564144f06220\" (UID: \"d77e8cf8-6cdc-4458-8c0c-564144f06220\") "
Nov 02 15:50:56.615273 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:50:56.615098    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eb8e624a-b173-4fc3-b5b5-4183ebf2eda7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb8e624a-b173-4fc3-b5b5-4183ebf2eda7-plugins-config-volume\") pod \"eb8e624a-b173-4fc3-b5b5-4183ebf2eda7\" (UID: \"eb8e624a-b173-4fc3-b5b5-4183ebf2eda7\") "
Nov 02 15:50:56.615273 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:50:56.615193    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d77e8cf8-6cdc-4458-8c0c-564144f06220-kube-api-access-ns8sq\" (UniqueName: \"kubernetes.io/projected/d77e8cf8-6cdc-4458-8c0c-564144f06220-kube-api-access-ns8sq\") pod \"d77e8cf8-6cdc-4458-8c0c-564144f06220\" (UID: \"d77e8cf8-6cdc-4458-8c0c-564144f06220\") "
Nov 02 15:50:56.615419 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:50:56.615332    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb8e624a-b173-4fc3-b5b5-4183ebf2eda7-kube-api-access-9sbq5\" (UniqueName: \"kubernetes.io/projected/eb8e624a-b173-4fc3-b5b5-4183ebf2eda7-kube-api-access-9sbq5\") pod \"eb8e624a-b173-4fc3-b5b5-4183ebf2eda7\" (UID: \"eb8e624a-b173-4fc3-b5b5-4183ebf2eda7\") "
Nov 02 15:50:56.615547 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:50:56.615518    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d77e8cf8-6cdc-4458-8c0c-564144f06220-plugins-config-volume" pod "d77e8cf8-6cdc-4458-8c0c-564144f06220" (UID: "d77e8cf8-6cdc-4458-8c0c-564144f06220"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:50:56.615547 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:50:56.615530    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb8e624a-b173-4fc3-b5b5-4183ebf2eda7-plugins-config-volume" pod "eb8e624a-b173-4fc3-b5b5-4183ebf2eda7" (UID: "eb8e624a-b173-4fc3-b5b5-4183ebf2eda7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:50:56.615852 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:50:56.615826    5304 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb8e624a-b173-4fc3-b5b5-4183ebf2eda7-plugins-config-volume\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:50:56.615939 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:50:56.615883    5304 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d77e8cf8-6cdc-4458-8c0c-564144f06220-plugins-config-volume\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:50:56.617855 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:50:56.617830    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb8e624a-b173-4fc3-b5b5-4183ebf2eda7-kube-api-access-9sbq5" pod "eb8e624a-b173-4fc3-b5b5-4183ebf2eda7" (UID: "eb8e624a-b173-4fc3-b5b5-4183ebf2eda7"). InnerVolumeSpecName "kube-api-access-9sbq5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:50:56.618835 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:50:56.618812    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d77e8cf8-6cdc-4458-8c0c-564144f06220-kube-api-access-ns8sq" pod "d77e8cf8-6cdc-4458-8c0c-564144f06220" (UID: "d77e8cf8-6cdc-4458-8c0c-564144f06220"). InnerVolumeSpecName "kube-api-access-ns8sq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:50:56.717680 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:50:56.717646    5304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ns8sq\" (UniqueName: \"kubernetes.io/projected/d77e8cf8-6cdc-4458-8c0c-564144f06220-kube-api-access-ns8sq\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:50:56.717680 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:50:56.717676    5304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9sbq5\" (UniqueName: \"kubernetes.io/projected/eb8e624a-b173-4fc3-b5b5-4183ebf2eda7-kube-api-access-9sbq5\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:50:57.379113 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:50:57.379027    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-a55a55-59b6968f7d-c496x"
Nov 02 15:50:57.379113 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:50:57.379070    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-a55a55-59b6968f7d-c496x" event={"ID":"eb8e624a-b173-4fc3-b5b5-4183ebf2eda7","Type":"ContainerDied","Data":"29efbd66a8785f9eb305b0d086705dd6897d3c64696d1112c6638d437af4fc52"}
Nov 02 15:50:57.379654 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:50:57.379159    5304 scope.go:122] "RemoveContainer" containerID="faba378e91ad9c15eb6272580b5d3f1169f5438d9d07d54300bb8c43aecc74a7"
Nov 02 15:50:57.380592 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:50:57.380565    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a55a55-59db66d795-crnzc" event={"ID":"d77e8cf8-6cdc-4458-8c0c-564144f06220","Type":"ContainerDied","Data":"fb9d61a2520bbd33c35133df75aac14bb5ee36a76373c459bed53c91699287e0"}
Nov 02 15:50:57.380592 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:50:57.380580    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a55a55-59db66d795-crnzc"
Nov 02 15:50:57.391191 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:50:57.391167    5304 scope.go:122] "RemoveContainer" containerID="bcd04148bb8837a6d6fb432a8fcc29856dabbddb303cf6ae68f421f1e351de43"
Nov 02 15:50:57.437286 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:50:57.437252    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-a55a55-59db66d795-crnzc"]
Nov 02 15:50:57.449573 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:50:57.449532    5304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-a55a55-59db66d795-crnzc"]
Nov 02 15:50:57.475159 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:50:57.475113    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-a55a55-59b6968f7d-c496x"]
Nov 02 15:50:57.490070 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:50:57.490039    5304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-a55a55-59b6968f7d-c496x"]
Nov 02 15:50:58.139784 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:50:58.139747    5304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d77e8cf8-6cdc-4458-8c0c-564144f06220" path="/var/lib/kubelet/pods/d77e8cf8-6cdc-4458-8c0c-564144f06220/volumes"
Nov 02 15:50:58.140174 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:50:58.140161    5304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eb8e624a-b173-4fc3-b5b5-4183ebf2eda7" path="/var/lib/kubelet/pods/eb8e624a-b173-4fc3-b5b5-4183ebf2eda7/volumes"
Nov 02 15:50:58.521286 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:50:58.521174    5304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-af33a6-846475bc47-4lq5l"]
Nov 02 15:50:58.522028 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:50:58.521999    5304 cpu_manager.go:414] "containerMap: removing container" podUID="eb8e624a-b173-4fc3-b5b5-4183ebf2eda7" containerName="payload-processing"
Nov 02 15:50:58.522118 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:50:58.522030    5304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb8e624a-b173-4fc3-b5b5-4183ebf2eda7" containerName="payload-processing"
Nov 02 15:50:58.522118 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:50:58.522046    5304 cpu_manager.go:414] "containerMap: removing container" podUID="34a8288b-adc6-424b-a30b-38f023185c86" containerName="maas-api"
Nov 02 15:50:58.522118 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:50:58.522054    5304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34a8288b-adc6-424b-a30b-38f023185c86" containerName="maas-api"
Nov 02 15:50:58.522118 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:50:58.522068    5304 cpu_manager.go:414] "containerMap: removing container" podUID="888a0ac9-4645-4fca-bc53-022beb9f484f" containerName="payload-processing"
Nov 02 15:50:58.522118 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:50:58.522077    5304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="888a0ac9-4645-4fca-bc53-022beb9f484f" containerName="payload-processing"
Nov 02 15:50:58.522118 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:50:58.522118    5304 cpu_manager.go:414] "containerMap: removing container" podUID="8e3715e3-1d26-4abf-ad77-6cb9cb70c349" containerName="payload-pre-processing"
Nov 02 15:50:58.522118 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:50:58.522166    5304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e3715e3-1d26-4abf-ad77-6cb9cb70c349" containerName="payload-pre-processing"
Nov 02 15:50:58.522118 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:50:58.522183    5304 cpu_manager.go:414] "containerMap: removing container" podUID="d77e8cf8-6cdc-4458-8c0c-564144f06220" containerName="payload-pre-processing"
Nov 02 15:50:58.522649 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:50:58.522191    5304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d77e8cf8-6cdc-4458-8c0c-564144f06220" containerName="payload-pre-processing"
Nov 02 15:50:58.522649 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:50:58.522470    5304 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e3715e3-1d26-4abf-ad77-6cb9cb70c349" containerName="payload-pre-processing"
Nov 02 15:50:58.522649 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:50:58.522486    5304 memory_manager.go:365] "RemoveStaleState removing state" podUID="d77e8cf8-6cdc-4458-8c0c-564144f06220" containerName="payload-pre-processing"
Nov 02 15:50:58.522649 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:50:58.522504    5304 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb8e624a-b173-4fc3-b5b5-4183ebf2eda7" containerName="payload-processing"
Nov 02 15:50:58.522649 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:50:58.522516    5304 memory_manager.go:365] "RemoveStaleState removing state" podUID="34a8288b-adc6-424b-a30b-38f023185c86" containerName="maas-api"
Nov 02 15:50:58.522649 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:50:58.522526    5304 memory_manager.go:365] "RemoveStaleState removing state" podUID="888a0ac9-4645-4fca-bc53-022beb9f484f" containerName="payload-processing"
Nov 02 15:50:58.528776 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:50:58.528741    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-af33a6-846475bc47-4lq5l"
Nov 02 15:50:58.545777 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:50:58.545741    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-af33a6-846475bc47-4lq5l"]
Nov 02 15:50:58.547242 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:50:58.547211    5304 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-af33a6\""
Nov 02 15:50:58.547367 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:50:58.547312    5304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-af33a6-dockercfg-sr8c9\""
Nov 02 15:50:58.642882 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:50:58.642840    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b7znk\" (UniqueName: \"kubernetes.io/projected/69df020e-436a-4ae9-b07b-b138bb2ff1da-kube-api-access-b7znk\") pod \"payload-pre-processing-e2e-ipp-b-af33a6-846475bc47-4lq5l\" (UID: \"69df020e-436a-4ae9-b07b-b138bb2ff1da\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-af33a6-846475bc47-4lq5l"
Nov 02 15:50:58.643108 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:50:58.643081    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69df020e-436a-4ae9-b07b-b138bb2ff1da-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-af33a6-846475bc47-4lq5l\" (UID: \"69df020e-436a-4ae9-b07b-b138bb2ff1da\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-af33a6-846475bc47-4lq5l"
Nov 02 15:50:58.677143 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:50:58.677085    5304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-af33a6-7db8978884-rnjtx"]
Nov 02 15:50:58.696675 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:50:58.696642    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-af33a6-7db8978884-rnjtx"
Nov 02 15:50:58.710340 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:50:58.710308    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-af33a6-7db8978884-rnjtx"]
Nov 02 15:50:58.745045 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:50:58.745000    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b7znk\" (UniqueName: \"kubernetes.io/projected/69df020e-436a-4ae9-b07b-b138bb2ff1da-kube-api-access-b7znk\") pod \"payload-pre-processing-e2e-ipp-b-af33a6-846475bc47-4lq5l\" (UID: \"69df020e-436a-4ae9-b07b-b138bb2ff1da\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-af33a6-846475bc47-4lq5l"
Nov 02 15:50:58.745326 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:50:58.745106    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c18605a-898a-41ca-ba2d-e7f48597e790-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-af33a6-7db8978884-rnjtx\" (UID: \"0c18605a-898a-41ca-ba2d-e7f48597e790\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-af33a6-7db8978884-rnjtx"
Nov 02 15:50:58.745326 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:50:58.745260    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hxmrf\" (UniqueName: \"kubernetes.io/projected/0c18605a-898a-41ca-ba2d-e7f48597e790-kube-api-access-hxmrf\") pod \"payload-processing-e2e-ipp-b-af33a6-7db8978884-rnjtx\" (UID: \"0c18605a-898a-41ca-ba2d-e7f48597e790\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-af33a6-7db8978884-rnjtx"
Nov 02 15:50:58.745480 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:50:58.745343    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69df020e-436a-4ae9-b07b-b138bb2ff1da-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-af33a6-846475bc47-4lq5l\" (UID: \"69df020e-436a-4ae9-b07b-b138bb2ff1da\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-af33a6-846475bc47-4lq5l"
Nov 02 15:50:58.746086 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:50:58.746063    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69df020e-436a-4ae9-b07b-b138bb2ff1da-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-af33a6-846475bc47-4lq5l\" (UID: \"69df020e-436a-4ae9-b07b-b138bb2ff1da\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-af33a6-846475bc47-4lq5l"
Nov 02 15:50:58.769621 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:50:58.769593    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b7znk\" (UniqueName: \"kubernetes.io/projected/69df020e-436a-4ae9-b07b-b138bb2ff1da-kube-api-access-b7znk\") pod \"payload-pre-processing-e2e-ipp-b-af33a6-846475bc47-4lq5l\" (UID: \"69df020e-436a-4ae9-b07b-b138bb2ff1da\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-af33a6-846475bc47-4lq5l"
Nov 02 15:50:58.852664 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:50:58.852622    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hxmrf\" (UniqueName: \"kubernetes.io/projected/0c18605a-898a-41ca-ba2d-e7f48597e790-kube-api-access-hxmrf\") pod \"payload-processing-e2e-ipp-b-af33a6-7db8978884-rnjtx\" (UID: \"0c18605a-898a-41ca-ba2d-e7f48597e790\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-af33a6-7db8978884-rnjtx"
Nov 02 15:50:58.852849 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:50:58.852804    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c18605a-898a-41ca-ba2d-e7f48597e790-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-af33a6-7db8978884-rnjtx\" (UID: \"0c18605a-898a-41ca-ba2d-e7f48597e790\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-af33a6-7db8978884-rnjtx"
Nov 02 15:50:58.854452 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:50:58.853551    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c18605a-898a-41ca-ba2d-e7f48597e790-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-af33a6-7db8978884-rnjtx\" (UID: \"0c18605a-898a-41ca-ba2d-e7f48597e790\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-af33a6-7db8978884-rnjtx"
Nov 02 15:50:58.854452 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:50:58.853937    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-af33a6-846475bc47-4lq5l"
Nov 02 15:50:58.879007 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:50:58.878792    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hxmrf\" (UniqueName: \"kubernetes.io/projected/0c18605a-898a-41ca-ba2d-e7f48597e790-kube-api-access-hxmrf\") pod \"payload-processing-e2e-ipp-b-af33a6-7db8978884-rnjtx\" (UID: \"0c18605a-898a-41ca-ba2d-e7f48597e790\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-af33a6-7db8978884-rnjtx"
Nov 02 15:50:59.017021 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:50:59.016975    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-af33a6-7db8978884-rnjtx"
Nov 02 15:50:59.172877 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:50:59.172800    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-af33a6-846475bc47-4lq5l"]
Nov 02 15:50:59.176547 ip-10-0-135-138 kubenswrapper[5304]: W1102 15:50:59.176515    5304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69df020e_436a_4ae9_b07b_b138bb2ff1da.slice/crio-aeef7ae45756e56b4e71beeaf89f533b019e3ce483c686a514545e44b4e224f7 WatchSource:0}: Error finding container aeef7ae45756e56b4e71beeaf89f533b019e3ce483c686a514545e44b4e224f7: Status 404 returned error can't find the container with id aeef7ae45756e56b4e71beeaf89f533b019e3ce483c686a514545e44b4e224f7
Nov 02 15:50:59.354055 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:50:59.354024    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-af33a6-7db8978884-rnjtx"]
Nov 02 15:50:59.356579 ip-10-0-135-138 kubenswrapper[5304]: W1102 15:50:59.356546    5304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c18605a_898a_41ca_ba2d_e7f48597e790.slice/crio-64ee3c8ea8a43f05b8549a2478444b3df01871ff747f25967f7626c44d0cb74c WatchSource:0}: Error finding container 64ee3c8ea8a43f05b8549a2478444b3df01871ff747f25967f7626c44d0cb74c: Status 404 returned error can't find the container with id 64ee3c8ea8a43f05b8549a2478444b3df01871ff747f25967f7626c44d0cb74c
Nov 02 15:50:59.397330 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:50:59.397285    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-af33a6-7db8978884-rnjtx" event={"ID":"0c18605a-898a-41ca-ba2d-e7f48597e790","Type":"ContainerStarted","Data":"64ee3c8ea8a43f05b8549a2478444b3df01871ff747f25967f7626c44d0cb74c"}
Nov 02 15:50:59.399163 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:50:59.399114    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-af33a6-846475bc47-4lq5l" event={"ID":"69df020e-436a-4ae9-b07b-b138bb2ff1da","Type":"ContainerStarted","Data":"47b4a72b4fec46c50c18aacbadfea0b18f31358203d4ab10b8903cde59045c47"}
Nov 02 15:50:59.399282 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:50:59.399183    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-af33a6-846475bc47-4lq5l" event={"ID":"69df020e-436a-4ae9-b07b-b138bb2ff1da","Type":"ContainerStarted","Data":"aeef7ae45756e56b4e71beeaf89f533b019e3ce483c686a514545e44b4e224f7"}
Nov 02 15:50:59.399282 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:50:59.399216    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-af33a6-846475bc47-4lq5l"
Nov 02 15:50:59.439928 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:50:59.439876    5304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-af33a6-846475bc47-4lq5l" podStartSLOduration=1.439857237 podStartE2EDuration="1.439857237s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:50:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:50:59.438562622 +0000 UTC m=+3259.432387041" watchObservedRunningTime="2026-11-02 15:50:59.439857237 +0000 UTC m=+3259.433681648"
Nov 02 15:51:00.407919 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:51:00.407883    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-af33a6-7db8978884-rnjtx" event={"ID":"0c18605a-898a-41ca-ba2d-e7f48597e790","Type":"ContainerStarted","Data":"07b216156b332eb04fee1014a21768502042bf33cae85026fc027f99ed6381f9"}
Nov 02 15:51:00.408461 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:51:00.407959    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-af33a6-7db8978884-rnjtx"
Nov 02 15:51:00.452425 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:51:00.452365    5304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-af33a6-7db8978884-rnjtx" podStartSLOduration=2.45234605 podStartE2EDuration="2.45234605s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:50:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:51:00.448876937 +0000 UTC m=+3260.442701343" watchObservedRunningTime="2026-11-02 15:51:00.45234605 +0000 UTC m=+3260.446170462"
Nov 02 15:51:03.397898 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:51:03.397862    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-1cc1b2/distinct-w5-1cc1b2-kserve-5b7459bc6d-zl8xr"]
Nov 02 15:51:03.408950 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:51:03.408885    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-1cc1b2/premium-w5-1cc1b2-kserve-58db46f97c-cgdqg"]
Nov 02 15:51:10.410858 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:51:10.410825    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-af33a6-846475bc47-4lq5l"
Nov 02 15:51:11.421055 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:51:11.421024    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-af33a6-7db8978884-rnjtx"
Nov 02 15:51:18.423989 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:51:18.423945    5304 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-1cc1b2/distinct-w5-1cc1b2-kserve-5b7459bc6d-zl8xr" podUID="211ad15d-9234-4f57-b1b9-181dd2d74208" containerName="main" containerID="cri-o://fa543fe9e33bbdbadbef9555f23bbc36535b92baa290827f7897eac73e8dd386" gracePeriod=45
Nov 02 15:51:18.434393 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:51:18.434336    5304 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-1cc1b2/premium-w5-1cc1b2-kserve-58db46f97c-cgdqg" podUID="210f620a-8ae7-4e4c-acc3-46f75ddc6105" containerName="main" containerID="cri-o://ac6c3a0a81e5ed9d73f434bbb1db0c11246882624954e2b5f8c208073408bfd6" gracePeriod=45
Nov 02 15:51:18.598568 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:51:18.598526    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1cc1b2/premium-w5-1cc1b2-kserve-58db46f97c-cgdqg" podUID="210f620a-8ae7-4e4c-acc3-46f75ddc6105" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": dial tcp 10.133.0.45:8000: connect: connection refused"
Nov 02 15:51:18.708463 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:51:18.708327    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-1cc1b2/distinct-w5-1cc1b2-kserve-5b7459bc6d-zl8xr"
Nov 02 15:51:18.731352 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:51:18.731328    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-1cc1b2/premium-w5-1cc1b2-kserve-58db46f97c-cgdqg"
Nov 02 15:51:18.883488 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:51:18.883450    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/211ad15d-9234-4f57-b1b9-181dd2d74208-home\" (UniqueName: \"kubernetes.io/empty-dir/211ad15d-9234-4f57-b1b9-181dd2d74208-home\") pod \"211ad15d-9234-4f57-b1b9-181dd2d74208\" (UID: \"211ad15d-9234-4f57-b1b9-181dd2d74208\") "
Nov 02 15:51:18.883640 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:51:18.883500    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/211ad15d-9234-4f57-b1b9-181dd2d74208-kube-api-access-gdwqm\" (UniqueName: \"kubernetes.io/projected/211ad15d-9234-4f57-b1b9-181dd2d74208-kube-api-access-gdwqm\") pod \"211ad15d-9234-4f57-b1b9-181dd2d74208\" (UID: \"211ad15d-9234-4f57-b1b9-181dd2d74208\") "
Nov 02 15:51:18.883640 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:51:18.883551    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/211ad15d-9234-4f57-b1b9-181dd2d74208-dshm\" (UniqueName: \"kubernetes.io/empty-dir/211ad15d-9234-4f57-b1b9-181dd2d74208-dshm\") pod \"211ad15d-9234-4f57-b1b9-181dd2d74208\" (UID: \"211ad15d-9234-4f57-b1b9-181dd2d74208\") "
Nov 02 15:51:18.883760 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:51:18.883736    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/211ad15d-9234-4f57-b1b9-181dd2d74208-home" pod "211ad15d-9234-4f57-b1b9-181dd2d74208" (UID: "211ad15d-9234-4f57-b1b9-181dd2d74208"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:51:18.883803 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:51:18.883770    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/210f620a-8ae7-4e4c-acc3-46f75ddc6105-kube-api-access-v2cc5\" (UniqueName: \"kubernetes.io/projected/210f620a-8ae7-4e4c-acc3-46f75ddc6105-kube-api-access-v2cc5\") pod \"210f620a-8ae7-4e4c-acc3-46f75ddc6105\" (UID: \"210f620a-8ae7-4e4c-acc3-46f75ddc6105\") "
Nov 02 15:51:18.883900 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:51:18.883883    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/210f620a-8ae7-4e4c-acc3-46f75ddc6105-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/210f620a-8ae7-4e4c-acc3-46f75ddc6105-model-cache\") pod \"210f620a-8ae7-4e4c-acc3-46f75ddc6105\" (UID: \"210f620a-8ae7-4e4c-acc3-46f75ddc6105\") "
Nov 02 15:51:18.883935 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:51:18.883921    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/211ad15d-9234-4f57-b1b9-181dd2d74208-tls-certs\" (UniqueName: \"kubernetes.io/secret/211ad15d-9234-4f57-b1b9-181dd2d74208-tls-certs\") pod \"211ad15d-9234-4f57-b1b9-181dd2d74208\" (UID: \"211ad15d-9234-4f57-b1b9-181dd2d74208\") "
Nov 02 15:51:18.883985 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:51:18.883943    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/210f620a-8ae7-4e4c-acc3-46f75ddc6105-dshm\" (UniqueName: \"kubernetes.io/empty-dir/210f620a-8ae7-4e4c-acc3-46f75ddc6105-dshm\") pod \"210f620a-8ae7-4e4c-acc3-46f75ddc6105\" (UID: \"210f620a-8ae7-4e4c-acc3-46f75ddc6105\") "
Nov 02 15:51:18.884045 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:51:18.883996    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/210f620a-8ae7-4e4c-acc3-46f75ddc6105-home\" (UniqueName: \"kubernetes.io/empty-dir/210f620a-8ae7-4e4c-acc3-46f75ddc6105-home\") pod \"210f620a-8ae7-4e4c-acc3-46f75ddc6105\" (UID: \"210f620a-8ae7-4e4c-acc3-46f75ddc6105\") "
Nov 02 15:51:18.884104 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:51:18.884047    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/211ad15d-9234-4f57-b1b9-181dd2d74208-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/211ad15d-9234-4f57-b1b9-181dd2d74208-model-cache\") pod \"211ad15d-9234-4f57-b1b9-181dd2d74208\" (UID: \"211ad15d-9234-4f57-b1b9-181dd2d74208\") "
Nov 02 15:51:18.884205 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:51:18.884110    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/210f620a-8ae7-4e4c-acc3-46f75ddc6105-tls-certs\" (UniqueName: \"kubernetes.io/secret/210f620a-8ae7-4e4c-acc3-46f75ddc6105-tls-certs\") pod \"210f620a-8ae7-4e4c-acc3-46f75ddc6105\" (UID: \"210f620a-8ae7-4e4c-acc3-46f75ddc6105\") "
Nov 02 15:51:18.884205 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:51:18.884182    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/210f620a-8ae7-4e4c-acc3-46f75ddc6105-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/210f620a-8ae7-4e4c-acc3-46f75ddc6105-tmp-dir\") pod \"210f620a-8ae7-4e4c-acc3-46f75ddc6105\" (UID: \"210f620a-8ae7-4e4c-acc3-46f75ddc6105\") "
Nov 02 15:51:18.884343 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:51:18.884217    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/210f620a-8ae7-4e4c-acc3-46f75ddc6105-model-cache" pod "210f620a-8ae7-4e4c-acc3-46f75ddc6105" (UID: "210f620a-8ae7-4e4c-acc3-46f75ddc6105"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:51:18.884343 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:51:18.884238    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/211ad15d-9234-4f57-b1b9-181dd2d74208-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/211ad15d-9234-4f57-b1b9-181dd2d74208-tmp-dir\") pod \"211ad15d-9234-4f57-b1b9-181dd2d74208\" (UID: \"211ad15d-9234-4f57-b1b9-181dd2d74208\") "
Nov 02 15:51:18.884457 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:51:18.884342    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/210f620a-8ae7-4e4c-acc3-46f75ddc6105-home" pod "210f620a-8ae7-4e4c-acc3-46f75ddc6105" (UID: "210f620a-8ae7-4e4c-acc3-46f75ddc6105"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:51:18.884563 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:51:18.884540    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/211ad15d-9234-4f57-b1b9-181dd2d74208-model-cache" pod "211ad15d-9234-4f57-b1b9-181dd2d74208" (UID: "211ad15d-9234-4f57-b1b9-181dd2d74208"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:51:18.884739 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:51:18.884718    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/211ad15d-9234-4f57-b1b9-181dd2d74208-tmp-dir" pod "211ad15d-9234-4f57-b1b9-181dd2d74208" (UID: "211ad15d-9234-4f57-b1b9-181dd2d74208"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:51:18.884890 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:51:18.884871    5304 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/210f620a-8ae7-4e4c-acc3-46f75ddc6105-model-cache\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:51:18.885017 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:51:18.884895    5304 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/210f620a-8ae7-4e4c-acc3-46f75ddc6105-home\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:51:18.885017 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:51:18.884911    5304 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/211ad15d-9234-4f57-b1b9-181dd2d74208-model-cache\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:51:18.885017 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:51:18.884976    5304 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/211ad15d-9234-4f57-b1b9-181dd2d74208-tmp-dir\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:51:18.885017 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:51:18.884991    5304 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/211ad15d-9234-4f57-b1b9-181dd2d74208-home\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:51:18.885017 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:51:18.884998    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/210f620a-8ae7-4e4c-acc3-46f75ddc6105-tmp-dir" pod "210f620a-8ae7-4e4c-acc3-46f75ddc6105" (UID: "210f620a-8ae7-4e4c-acc3-46f75ddc6105"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:51:18.886439 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:51:18.886411    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/211ad15d-9234-4f57-b1b9-181dd2d74208-dshm" pod "211ad15d-9234-4f57-b1b9-181dd2d74208" (UID: "211ad15d-9234-4f57-b1b9-181dd2d74208"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:51:18.886575 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:51:18.886446    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/211ad15d-9234-4f57-b1b9-181dd2d74208-tls-certs" pod "211ad15d-9234-4f57-b1b9-181dd2d74208" (UID: "211ad15d-9234-4f57-b1b9-181dd2d74208"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:51:18.886575 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:51:18.886453    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/211ad15d-9234-4f57-b1b9-181dd2d74208-kube-api-access-gdwqm" pod "211ad15d-9234-4f57-b1b9-181dd2d74208" (UID: "211ad15d-9234-4f57-b1b9-181dd2d74208"). InnerVolumeSpecName "kube-api-access-gdwqm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:51:18.886575 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:51:18.886500    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/210f620a-8ae7-4e4c-acc3-46f75ddc6105-kube-api-access-v2cc5" pod "210f620a-8ae7-4e4c-acc3-46f75ddc6105" (UID: "210f620a-8ae7-4e4c-acc3-46f75ddc6105"). InnerVolumeSpecName "kube-api-access-v2cc5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:51:18.886575 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:51:18.886515    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/210f620a-8ae7-4e4c-acc3-46f75ddc6105-tls-certs" pod "210f620a-8ae7-4e4c-acc3-46f75ddc6105" (UID: "210f620a-8ae7-4e4c-acc3-46f75ddc6105"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:51:18.887137 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:51:18.887105    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/210f620a-8ae7-4e4c-acc3-46f75ddc6105-dshm" pod "210f620a-8ae7-4e4c-acc3-46f75ddc6105" (UID: "210f620a-8ae7-4e4c-acc3-46f75ddc6105"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:51:18.986272 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:51:18.986189    5304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gdwqm\" (UniqueName: \"kubernetes.io/projected/211ad15d-9234-4f57-b1b9-181dd2d74208-kube-api-access-gdwqm\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:51:18.986272 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:51:18.986218    5304 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/211ad15d-9234-4f57-b1b9-181dd2d74208-dshm\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:51:18.986272 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:51:18.986228    5304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v2cc5\" (UniqueName: \"kubernetes.io/projected/210f620a-8ae7-4e4c-acc3-46f75ddc6105-kube-api-access-v2cc5\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:51:18.986272 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:51:18.986237    5304 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/211ad15d-9234-4f57-b1b9-181dd2d74208-tls-certs\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:51:18.986272 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:51:18.986246    5304 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/210f620a-8ae7-4e4c-acc3-46f75ddc6105-dshm\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:51:18.986272 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:51:18.986254    5304 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/210f620a-8ae7-4e4c-acc3-46f75ddc6105-tls-certs\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:51:18.986272 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:51:18.986261    5304 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/210f620a-8ae7-4e4c-acc3-46f75ddc6105-tmp-dir\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:51:19.543233 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:51:19.543196    5304 generic.go:356] "Generic (PLEG): container finished" podID="210f620a-8ae7-4e4c-acc3-46f75ddc6105" containerID="ac6c3a0a81e5ed9d73f434bbb1db0c11246882624954e2b5f8c208073408bfd6" exitCode=0
Nov 02 15:51:19.543732 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:51:19.543243    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-1cc1b2/premium-w5-1cc1b2-kserve-58db46f97c-cgdqg"
Nov 02 15:51:19.543732 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:51:19.543281    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-1cc1b2/premium-w5-1cc1b2-kserve-58db46f97c-cgdqg" event={"ID":"210f620a-8ae7-4e4c-acc3-46f75ddc6105","Type":"ContainerDied","Data":"ac6c3a0a81e5ed9d73f434bbb1db0c11246882624954e2b5f8c208073408bfd6"}
Nov 02 15:51:19.543732 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:51:19.543332    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-1cc1b2/premium-w5-1cc1b2-kserve-58db46f97c-cgdqg" event={"ID":"210f620a-8ae7-4e4c-acc3-46f75ddc6105","Type":"ContainerDied","Data":"7f0c083bc4a986aa1ca5162a54f161f37e52887b19d31f74b292b81288e0a2dd"}
Nov 02 15:51:19.543732 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:51:19.543354    5304 scope.go:122] "RemoveContainer" containerID="ac6c3a0a81e5ed9d73f434bbb1db0c11246882624954e2b5f8c208073408bfd6"
Nov 02 15:51:19.544799 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:51:19.544766    5304 generic.go:356] "Generic (PLEG): container finished" podID="211ad15d-9234-4f57-b1b9-181dd2d74208" containerID="fa543fe9e33bbdbadbef9555f23bbc36535b92baa290827f7897eac73e8dd386" exitCode=0
Nov 02 15:51:19.545021 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:51:19.544813    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-1cc1b2/distinct-w5-1cc1b2-kserve-5b7459bc6d-zl8xr"
Nov 02 15:51:19.545021 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:51:19.544854    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-1cc1b2/distinct-w5-1cc1b2-kserve-5b7459bc6d-zl8xr" event={"ID":"211ad15d-9234-4f57-b1b9-181dd2d74208","Type":"ContainerDied","Data":"fa543fe9e33bbdbadbef9555f23bbc36535b92baa290827f7897eac73e8dd386"}
Nov 02 15:51:19.545021 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:51:19.544891    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-1cc1b2/distinct-w5-1cc1b2-kserve-5b7459bc6d-zl8xr" event={"ID":"211ad15d-9234-4f57-b1b9-181dd2d74208","Type":"ContainerDied","Data":"e2343f1c408a18a75daf42cea56518aa651372f0a8bbcce11824b7c65b63f3bf"}
Nov 02 15:51:19.555846 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:51:19.555824    5304 scope.go:122] "RemoveContainer" containerID="ac6c3a0a81e5ed9d73f434bbb1db0c11246882624954e2b5f8c208073408bfd6"
Nov 02 15:51:19.556255 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:51:19.556207    5304 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac6c3a0a81e5ed9d73f434bbb1db0c11246882624954e2b5f8c208073408bfd6\": container with ID starting with ac6c3a0a81e5ed9d73f434bbb1db0c11246882624954e2b5f8c208073408bfd6 not found: ID does not exist" containerID="ac6c3a0a81e5ed9d73f434bbb1db0c11246882624954e2b5f8c208073408bfd6"
Nov 02 15:51:19.556364 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:51:19.556270    5304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac6c3a0a81e5ed9d73f434bbb1db0c11246882624954e2b5f8c208073408bfd6"} err="failed to get container status \"ac6c3a0a81e5ed9d73f434bbb1db0c11246882624954e2b5f8c208073408bfd6\": rpc error: code = NotFound desc = could not find container \"ac6c3a0a81e5ed9d73f434bbb1db0c11246882624954e2b5f8c208073408bfd6\": container with ID starting with ac6c3a0a81e5ed9d73f434bbb1db0c11246882624954e2b5f8c208073408bfd6 not found: ID does not exist"
Nov 02 15:51:19.556364 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:51:19.556296    5304 scope.go:122] "RemoveContainer" containerID="fa543fe9e33bbdbadbef9555f23bbc36535b92baa290827f7897eac73e8dd386"
Nov 02 15:51:19.565530 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:51:19.565513    5304 scope.go:122] "RemoveContainer" containerID="fa543fe9e33bbdbadbef9555f23bbc36535b92baa290827f7897eac73e8dd386"
Nov 02 15:51:19.565805 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:51:19.565786    5304 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa543fe9e33bbdbadbef9555f23bbc36535b92baa290827f7897eac73e8dd386\": container with ID starting with fa543fe9e33bbdbadbef9555f23bbc36535b92baa290827f7897eac73e8dd386 not found: ID does not exist" containerID="fa543fe9e33bbdbadbef9555f23bbc36535b92baa290827f7897eac73e8dd386"
Nov 02 15:51:19.565855 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:51:19.565813    5304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa543fe9e33bbdbadbef9555f23bbc36535b92baa290827f7897eac73e8dd386"} err="failed to get container status \"fa543fe9e33bbdbadbef9555f23bbc36535b92baa290827f7897eac73e8dd386\": rpc error: code = NotFound desc = could not find container \"fa543fe9e33bbdbadbef9555f23bbc36535b92baa290827f7897eac73e8dd386\": container with ID starting with fa543fe9e33bbdbadbef9555f23bbc36535b92baa290827f7897eac73e8dd386 not found: ID does not exist"
Nov 02 15:51:19.597142 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:51:19.597105    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-1cc1b2/distinct-w5-1cc1b2-kserve-5b7459bc6d-zl8xr"]
Nov 02 15:51:19.608745 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:51:19.608717    5304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-1cc1b2/distinct-w5-1cc1b2-kserve-5b7459bc6d-zl8xr"]
Nov 02 15:51:19.634626 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:51:19.634598    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-1cc1b2/premium-w5-1cc1b2-kserve-58db46f97c-cgdqg"]
Nov 02 15:51:19.640885 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:51:19.640860    5304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-1cc1b2/premium-w5-1cc1b2-kserve-58db46f97c-cgdqg"]
Nov 02 15:51:20.119019 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:51:20.118978    5304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="210f620a-8ae7-4e4c-acc3-46f75ddc6105" path="/var/lib/kubelet/pods/210f620a-8ae7-4e4c-acc3-46f75ddc6105/volumes"
Nov 02 15:51:20.119548 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:51:20.119525    5304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="211ad15d-9234-4f57-b1b9-181dd2d74208" path="/var/lib/kubelet/pods/211ad15d-9234-4f57-b1b9-181dd2d74208/volumes"
Nov 02 15:51:23.486361 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:51:23.486328    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-5721b1-openshift-default-56c5cbb5cc-987lj"]
Nov 02 15:51:23.486779 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:51:23.486585    5304 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-5721b1-openshift-default-56c5cbb5cc-987lj" podUID="1285186b-0b54-48b3-84c6-b640966bb312" containerName="istio-proxy" containerID="cri-o://69030f9b3280d9c4cc9be8d8541281ac5da89159dba6f823a5d22e19d15d1cec" gracePeriod=30
Nov 02 15:51:24.800238 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:51:24.800201    5304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k6fjk" containerName="extractor"
Nov 02 15:51:28.635509 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:51:28.635467    5304 generic.go:356] "Generic (PLEG): container finished" podID="1285186b-0b54-48b3-84c6-b640966bb312" containerID="69030f9b3280d9c4cc9be8d8541281ac5da89159dba6f823a5d22e19d15d1cec" exitCode=0
Nov 02 15:51:28.635986 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:51:28.635549    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-5721b1-openshift-default-56c5cbb5cc-987lj" event={"ID":"1285186b-0b54-48b3-84c6-b640966bb312","Type":"ContainerDied","Data":"69030f9b3280d9c4cc9be8d8541281ac5da89159dba6f823a5d22e19d15d1cec"}
Nov 02 15:51:28.635986 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:51:28.635630    5304 scope.go:122] "RemoveContainer" containerID="4eb2e692475edc363f1e913a7bc8179f0ccb216421f28c4a087c0e4a9bcd6285"
Nov 02 15:51:28.775760 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:51:28.775699    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-5721b1-openshift-default-56c5cbb5cc-987lj"
Nov 02 15:51:28.830011 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:51:28.829973    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1285186b-0b54-48b3-84c6-b640966bb312-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1285186b-0b54-48b3-84c6-b640966bb312-workload-certs\") pod \"1285186b-0b54-48b3-84c6-b640966bb312\" (UID: \"1285186b-0b54-48b3-84c6-b640966bb312\") "
Nov 02 15:51:28.830252 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:51:28.830069    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1285186b-0b54-48b3-84c6-b640966bb312-kube-api-access-b2fbt\" (UniqueName: \"kubernetes.io/projected/1285186b-0b54-48b3-84c6-b640966bb312-kube-api-access-b2fbt\") pod \"1285186b-0b54-48b3-84c6-b640966bb312\" (UID: \"1285186b-0b54-48b3-84c6-b640966bb312\") "
Nov 02 15:51:28.830252 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:51:28.830098    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1285186b-0b54-48b3-84c6-b640966bb312-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1285186b-0b54-48b3-84c6-b640966bb312-workload-socket\") pod \"1285186b-0b54-48b3-84c6-b640966bb312\" (UID: \"1285186b-0b54-48b3-84c6-b640966bb312\") "
Nov 02 15:51:28.830252 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:51:28.830178    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1285186b-0b54-48b3-84c6-b640966bb312-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1285186b-0b54-48b3-84c6-b640966bb312-istio-envoy\") pod \"1285186b-0b54-48b3-84c6-b640966bb312\" (UID: \"1285186b-0b54-48b3-84c6-b640966bb312\") "
Nov 02 15:51:28.830437 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:51:28.830350    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1285186b-0b54-48b3-84c6-b640966bb312-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1285186b-0b54-48b3-84c6-b640966bb312-credential-socket\") pod \"1285186b-0b54-48b3-84c6-b640966bb312\" (UID: \"1285186b-0b54-48b3-84c6-b640966bb312\") "
Nov 02 15:51:28.830437 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:51:28.830364    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1285186b-0b54-48b3-84c6-b640966bb312-workload-certs" pod "1285186b-0b54-48b3-84c6-b640966bb312" (UID: "1285186b-0b54-48b3-84c6-b640966bb312"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:51:28.830437 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:51:28.830381    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1285186b-0b54-48b3-84c6-b640966bb312-istio-token\" (UniqueName: \"kubernetes.io/projected/1285186b-0b54-48b3-84c6-b640966bb312-istio-token\") pod \"1285186b-0b54-48b3-84c6-b640966bb312\" (UID: \"1285186b-0b54-48b3-84c6-b640966bb312\") "
Nov 02 15:51:28.830610 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:51:28.830464    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1285186b-0b54-48b3-84c6-b640966bb312-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1285186b-0b54-48b3-84c6-b640966bb312-istio-data\") pod \"1285186b-0b54-48b3-84c6-b640966bb312\" (UID: \"1285186b-0b54-48b3-84c6-b640966bb312\") "
Nov 02 15:51:28.830610 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:51:28.830510    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1285186b-0b54-48b3-84c6-b640966bb312-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1285186b-0b54-48b3-84c6-b640966bb312-istio-podinfo\") pod \"1285186b-0b54-48b3-84c6-b640966bb312\" (UID: \"1285186b-0b54-48b3-84c6-b640966bb312\") "
Nov 02 15:51:28.830610 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:51:28.830521    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1285186b-0b54-48b3-84c6-b640966bb312-workload-socket" pod "1285186b-0b54-48b3-84c6-b640966bb312" (UID: "1285186b-0b54-48b3-84c6-b640966bb312"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:51:28.830610 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:51:28.830541    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1285186b-0b54-48b3-84c6-b640966bb312-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1285186b-0b54-48b3-84c6-b640966bb312-istiod-ca-cert\") pod \"1285186b-0b54-48b3-84c6-b640966bb312\" (UID: \"1285186b-0b54-48b3-84c6-b640966bb312\") "
Nov 02 15:51:28.830610 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:51:28.830595    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1285186b-0b54-48b3-84c6-b640966bb312-credential-socket" pod "1285186b-0b54-48b3-84c6-b640966bb312" (UID: "1285186b-0b54-48b3-84c6-b640966bb312"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:51:28.830875 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:51:28.830819    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1285186b-0b54-48b3-84c6-b640966bb312-istiod-ca-cert" pod "1285186b-0b54-48b3-84c6-b640966bb312" (UID: "1285186b-0b54-48b3-84c6-b640966bb312"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:51:28.831103 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:51:28.831077    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1285186b-0b54-48b3-84c6-b640966bb312-istio-data" pod "1285186b-0b54-48b3-84c6-b640966bb312" (UID: "1285186b-0b54-48b3-84c6-b640966bb312"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:51:28.831300 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:51:28.831278    5304 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1285186b-0b54-48b3-84c6-b640966bb312-credential-socket\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:51:28.831383 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:51:28.831301    5304 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1285186b-0b54-48b3-84c6-b640966bb312-istio-data\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:51:28.831383 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:51:28.831311    5304 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1285186b-0b54-48b3-84c6-b640966bb312-istiod-ca-cert\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:51:28.831383 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:51:28.831320    5304 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1285186b-0b54-48b3-84c6-b640966bb312-workload-certs\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:51:28.831383 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:51:28.831328    5304 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1285186b-0b54-48b3-84c6-b640966bb312-workload-socket\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:51:28.832712 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:51:28.832683    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1285186b-0b54-48b3-84c6-b640966bb312-istio-podinfo" pod "1285186b-0b54-48b3-84c6-b640966bb312" (UID: "1285186b-0b54-48b3-84c6-b640966bb312"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 15:51:28.832712 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:51:28.832701    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1285186b-0b54-48b3-84c6-b640966bb312-kube-api-access-b2fbt" pod "1285186b-0b54-48b3-84c6-b640966bb312" (UID: "1285186b-0b54-48b3-84c6-b640966bb312"). InnerVolumeSpecName "kube-api-access-b2fbt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:51:28.832978 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:51:28.832954    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1285186b-0b54-48b3-84c6-b640966bb312-istio-envoy" pod "1285186b-0b54-48b3-84c6-b640966bb312" (UID: "1285186b-0b54-48b3-84c6-b640966bb312"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:51:28.832978 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:51:28.832968    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1285186b-0b54-48b3-84c6-b640966bb312-istio-token" pod "1285186b-0b54-48b3-84c6-b640966bb312" (UID: "1285186b-0b54-48b3-84c6-b640966bb312"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:51:28.932788 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:51:28.932753    5304 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1285186b-0b54-48b3-84c6-b640966bb312-istio-token\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:51:28.932788 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:51:28.932783    5304 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1285186b-0b54-48b3-84c6-b640966bb312-istio-podinfo\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:51:28.932788 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:51:28.932793    5304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b2fbt\" (UniqueName: \"kubernetes.io/projected/1285186b-0b54-48b3-84c6-b640966bb312-kube-api-access-b2fbt\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:51:28.933039 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:51:28.932803    5304 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1285186b-0b54-48b3-84c6-b640966bb312-istio-envoy\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:51:29.644539 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:51:29.644500    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-5721b1-openshift-default-56c5cbb5cc-987lj"
Nov 02 15:51:29.644979 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:51:29.644541    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-5721b1-openshift-default-56c5cbb5cc-987lj" event={"ID":"1285186b-0b54-48b3-84c6-b640966bb312","Type":"ContainerDied","Data":"9b4e1b02012bf83802c44c13f4572ee55eb7cb0e0bdcaf043c9f10b5e0af75d3"}
Nov 02 15:51:29.644979 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:51:29.644603    5304 scope.go:122] "RemoveContainer" containerID="69030f9b3280d9c4cc9be8d8541281ac5da89159dba6f823a5d22e19d15d1cec"
Nov 02 15:51:29.726166 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:51:29.720625    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-5721b1-openshift-default-56c5cbb5cc-987lj"]
Nov 02 15:51:29.732685 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:51:29.731885    5304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-5721b1-openshift-default-56c5cbb5cc-987lj"]
Nov 02 15:51:30.118548 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:51:30.118514    5304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1285186b-0b54-48b3-84c6-b640966bb312" path="/var/lib/kubelet/pods/1285186b-0b54-48b3-84c6-b640966bb312/volumes"
Nov 02 15:51:42.995450 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:51:42.995409    5304 scope.go:122] "RemoveContainer" containerID="fccc01c9ac53fde16b503bec6219393089d9a23485208ba252d89e3a508b275d"
Nov 02 15:51:43.005141 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:51:43.005101    5304 scope.go:122] "RemoveContainer" containerID="82cb496a400aa30d70a7faa9a1af71be6a94237fcc1b90726ad7cc9fe7386048"
Nov 02 15:51:43.014325 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:51:43.014297    5304 scope.go:122] "RemoveContainer" containerID="d9acb6dbd5f54b30c15ce3b101055c388a0b8a35443bf5ac415ca57978ae962b"
Nov 02 15:51:43.023673 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:51:43.023643    5304 scope.go:122] "RemoveContainer" containerID="96ea1c70384ba8e82a5d8e2d54cea95e3177263337f5f57689a238f43e01a60a"
Nov 02 15:51:43.032177 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:51:43.032159    5304 scope.go:122] "RemoveContainer" containerID="ab2550b18afc069e02c937d4a37eac19199136fef39bc5e683795aceac45c368"
Nov 02 15:51:47.328702 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:51:47.328661    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-d79991-6d4b6d8c48-ptv5r"]
Nov 02 15:51:47.329200 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:51:47.328871    5304 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-d79991-6d4b6d8c48-ptv5r" podUID="fe452842-bf9b-4129-a1dc-4a5b98d2672f" containerName="payload-processing" containerID="cri-o://01aa731f1b95615f39e11bf514b6c741f59446ce23c18bb0334572291fab4459" gracePeriod=30
Nov 02 15:51:47.621745 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:51:47.621711    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-d79991-6d4b6d8c48-ptv5r"
Nov 02 15:51:47.733085 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:51:47.733048    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fe452842-bf9b-4129-a1dc-4a5b98d2672f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe452842-bf9b-4129-a1dc-4a5b98d2672f-plugins-config-volume\") pod \"fe452842-bf9b-4129-a1dc-4a5b98d2672f\" (UID: \"fe452842-bf9b-4129-a1dc-4a5b98d2672f\") "
Nov 02 15:51:47.733301 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:51:47.733102    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe452842-bf9b-4129-a1dc-4a5b98d2672f-kube-api-access-mjtcm\" (UniqueName: \"kubernetes.io/projected/fe452842-bf9b-4129-a1dc-4a5b98d2672f-kube-api-access-mjtcm\") pod \"fe452842-bf9b-4129-a1dc-4a5b98d2672f\" (UID: \"fe452842-bf9b-4129-a1dc-4a5b98d2672f\") "
Nov 02 15:51:47.733459 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:51:47.733435    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe452842-bf9b-4129-a1dc-4a5b98d2672f-plugins-config-volume" pod "fe452842-bf9b-4129-a1dc-4a5b98d2672f" (UID: "fe452842-bf9b-4129-a1dc-4a5b98d2672f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:51:47.733769 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:51:47.733751    5304 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe452842-bf9b-4129-a1dc-4a5b98d2672f-plugins-config-volume\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:51:47.735347 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:51:47.735325    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe452842-bf9b-4129-a1dc-4a5b98d2672f-kube-api-access-mjtcm" pod "fe452842-bf9b-4129-a1dc-4a5b98d2672f" (UID: "fe452842-bf9b-4129-a1dc-4a5b98d2672f"). InnerVolumeSpecName "kube-api-access-mjtcm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:51:47.771268 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:51:47.771236    5304 generic.go:356] "Generic (PLEG): container finished" podID="fe452842-bf9b-4129-a1dc-4a5b98d2672f" containerID="01aa731f1b95615f39e11bf514b6c741f59446ce23c18bb0334572291fab4459" exitCode=0
Nov 02 15:51:47.771454 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:51:47.771306    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-d79991-6d4b6d8c48-ptv5r"
Nov 02 15:51:47.771454 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:51:47.771317    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-d79991-6d4b6d8c48-ptv5r" event={"ID":"fe452842-bf9b-4129-a1dc-4a5b98d2672f","Type":"ContainerDied","Data":"01aa731f1b95615f39e11bf514b6c741f59446ce23c18bb0334572291fab4459"}
Nov 02 15:51:47.771454 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:51:47.771393    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-d79991-6d4b6d8c48-ptv5r" event={"ID":"fe452842-bf9b-4129-a1dc-4a5b98d2672f","Type":"ContainerDied","Data":"120da66418d003200617c06ae7eea5dca3611719993f1b35433a24b24cddc00c"}
Nov 02 15:51:47.771454 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:51:47.771422    5304 scope.go:122] "RemoveContainer" containerID="01aa731f1b95615f39e11bf514b6c741f59446ce23c18bb0334572291fab4459"
Nov 02 15:51:47.789862 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:51:47.789828    5304 scope.go:122] "RemoveContainer" containerID="01aa731f1b95615f39e11bf514b6c741f59446ce23c18bb0334572291fab4459"
Nov 02 15:51:47.790491 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:51:47.790458    5304 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"01aa731f1b95615f39e11bf514b6c741f59446ce23c18bb0334572291fab4459\": container with ID starting with 01aa731f1b95615f39e11bf514b6c741f59446ce23c18bb0334572291fab4459 not found: ID does not exist" containerID="01aa731f1b95615f39e11bf514b6c741f59446ce23c18bb0334572291fab4459"
Nov 02 15:51:47.790635 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:51:47.790506    5304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01aa731f1b95615f39e11bf514b6c741f59446ce23c18bb0334572291fab4459"} err="failed to get container status \"01aa731f1b95615f39e11bf514b6c741f59446ce23c18bb0334572291fab4459\": rpc error: code = NotFound desc = could not find container \"01aa731f1b95615f39e11bf514b6c741f59446ce23c18bb0334572291fab4459\": container with ID starting with 01aa731f1b95615f39e11bf514b6c741f59446ce23c18bb0334572291fab4459 not found: ID does not exist"
Nov 02 15:51:47.834857 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:51:47.834702    5304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mjtcm\" (UniqueName: \"kubernetes.io/projected/fe452842-bf9b-4129-a1dc-4a5b98d2672f-kube-api-access-mjtcm\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:51:47.836998 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:51:47.836960    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-d79991-6d4b6d8c48-ptv5r"]
Nov 02 15:51:47.841720 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:51:47.841685    5304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-d79991-6d4b6d8c48-ptv5r"]
Nov 02 15:51:48.146028 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:51:48.145935    5304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fe452842-bf9b-4129-a1dc-4a5b98d2672f" path="/var/lib/kubelet/pods/fe452842-bf9b-4129-a1dc-4a5b98d2672f/volumes"
Nov 02 15:51:54.800868 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:51:54.800833    5304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k6fjk" containerName="extractor"
Nov 02 15:51:55.875675 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:51:55.875638    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-d79991-openshift-default-74c857f7cc-p2snd"]
Nov 02 15:51:55.876094 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:51:55.875937    5304 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-d79991-openshift-default-74c857f7cc-p2snd" podUID="1020808c-e760-46e7-99e8-66932fca1956" containerName="istio-proxy" containerID="cri-o://4f574ad5724d4e728c194a366add175a69e11f44da92c080ed4c3dcaad1512a1" gracePeriod=30
Nov 02 15:52:01.142874 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:52:01.142811    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-d79991-openshift-default-74c857f7cc-p2snd"
Nov 02 15:52:01.245603 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:52:01.245565    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1020808c-e760-46e7-99e8-66932fca1956-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1020808c-e760-46e7-99e8-66932fca1956-istiod-ca-cert\") pod \"1020808c-e760-46e7-99e8-66932fca1956\" (UID: \"1020808c-e760-46e7-99e8-66932fca1956\") "
Nov 02 15:52:01.245813 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:52:01.245626    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1020808c-e760-46e7-99e8-66932fca1956-istio-token\" (UniqueName: \"kubernetes.io/projected/1020808c-e760-46e7-99e8-66932fca1956-istio-token\") pod \"1020808c-e760-46e7-99e8-66932fca1956\" (UID: \"1020808c-e760-46e7-99e8-66932fca1956\") "
Nov 02 15:52:01.245813 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:52:01.245679    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1020808c-e760-46e7-99e8-66932fca1956-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1020808c-e760-46e7-99e8-66932fca1956-istio-envoy\") pod \"1020808c-e760-46e7-99e8-66932fca1956\" (UID: \"1020808c-e760-46e7-99e8-66932fca1956\") "
Nov 02 15:52:01.245813 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:52:01.245724    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1020808c-e760-46e7-99e8-66932fca1956-kube-api-access-t67w7\" (UniqueName: \"kubernetes.io/projected/1020808c-e760-46e7-99e8-66932fca1956-kube-api-access-t67w7\") pod \"1020808c-e760-46e7-99e8-66932fca1956\" (UID: \"1020808c-e760-46e7-99e8-66932fca1956\") "
Nov 02 15:52:01.246019 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:52:01.245851    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1020808c-e760-46e7-99e8-66932fca1956-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1020808c-e760-46e7-99e8-66932fca1956-credential-socket\") pod \"1020808c-e760-46e7-99e8-66932fca1956\" (UID: \"1020808c-e760-46e7-99e8-66932fca1956\") "
Nov 02 15:52:01.246019 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:52:01.245891    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1020808c-e760-46e7-99e8-66932fca1956-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1020808c-e760-46e7-99e8-66932fca1956-workload-certs\") pod \"1020808c-e760-46e7-99e8-66932fca1956\" (UID: \"1020808c-e760-46e7-99e8-66932fca1956\") "
Nov 02 15:52:01.246182 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:52:01.246152    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1020808c-e760-46e7-99e8-66932fca1956-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1020808c-e760-46e7-99e8-66932fca1956-workload-socket\") pod \"1020808c-e760-46e7-99e8-66932fca1956\" (UID: \"1020808c-e760-46e7-99e8-66932fca1956\") "
Nov 02 15:52:01.246538 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:52:01.246501    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1020808c-e760-46e7-99e8-66932fca1956-istiod-ca-cert" pod "1020808c-e760-46e7-99e8-66932fca1956" (UID: "1020808c-e760-46e7-99e8-66932fca1956"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:52:01.246906 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:52:01.246870    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1020808c-e760-46e7-99e8-66932fca1956-workload-socket" pod "1020808c-e760-46e7-99e8-66932fca1956" (UID: "1020808c-e760-46e7-99e8-66932fca1956"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:52:01.247004 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:52:01.246920    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1020808c-e760-46e7-99e8-66932fca1956-istio-data" pod "1020808c-e760-46e7-99e8-66932fca1956" (UID: "1020808c-e760-46e7-99e8-66932fca1956"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:52:01.247004 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:52:01.246832    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1020808c-e760-46e7-99e8-66932fca1956-credential-socket" pod "1020808c-e760-46e7-99e8-66932fca1956" (UID: "1020808c-e760-46e7-99e8-66932fca1956"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:52:01.247004 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:52:01.246555    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1020808c-e760-46e7-99e8-66932fca1956-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1020808c-e760-46e7-99e8-66932fca1956-istio-data\") pod \"1020808c-e760-46e7-99e8-66932fca1956\" (UID: \"1020808c-e760-46e7-99e8-66932fca1956\") "
Nov 02 15:52:01.247004 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:52:01.246990    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1020808c-e760-46e7-99e8-66932fca1956-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1020808c-e760-46e7-99e8-66932fca1956-istio-podinfo\") pod \"1020808c-e760-46e7-99e8-66932fca1956\" (UID: \"1020808c-e760-46e7-99e8-66932fca1956\") "
Nov 02 15:52:01.247285 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:52:01.247239    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1020808c-e760-46e7-99e8-66932fca1956-workload-certs" pod "1020808c-e760-46e7-99e8-66932fca1956" (UID: "1020808c-e760-46e7-99e8-66932fca1956"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:52:01.247551 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:52:01.247531    5304 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1020808c-e760-46e7-99e8-66932fca1956-credential-socket\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:52:01.247642 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:52:01.247556    5304 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1020808c-e760-46e7-99e8-66932fca1956-workload-certs\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:52:01.247642 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:52:01.247570    5304 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1020808c-e760-46e7-99e8-66932fca1956-workload-socket\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:52:01.247642 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:52:01.247583    5304 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1020808c-e760-46e7-99e8-66932fca1956-istio-data\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:52:01.247642 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:52:01.247596    5304 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1020808c-e760-46e7-99e8-66932fca1956-istiod-ca-cert\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:52:01.252590 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:52:01.252554    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1020808c-e760-46e7-99e8-66932fca1956-istio-envoy" pod "1020808c-e760-46e7-99e8-66932fca1956" (UID: "1020808c-e760-46e7-99e8-66932fca1956"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:52:01.252713 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:52:01.252600    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1020808c-e760-46e7-99e8-66932fca1956-istio-podinfo" pod "1020808c-e760-46e7-99e8-66932fca1956" (UID: "1020808c-e760-46e7-99e8-66932fca1956"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 15:52:01.252713 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:52:01.252629    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1020808c-e760-46e7-99e8-66932fca1956-istio-token" pod "1020808c-e760-46e7-99e8-66932fca1956" (UID: "1020808c-e760-46e7-99e8-66932fca1956"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:52:01.252944 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:52:01.252926    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1020808c-e760-46e7-99e8-66932fca1956-kube-api-access-t67w7" pod "1020808c-e760-46e7-99e8-66932fca1956" (UID: "1020808c-e760-46e7-99e8-66932fca1956"). InnerVolumeSpecName "kube-api-access-t67w7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:52:01.348744 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:52:01.348708    5304 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1020808c-e760-46e7-99e8-66932fca1956-istio-podinfo\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:52:01.348744 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:52:01.348738    5304 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1020808c-e760-46e7-99e8-66932fca1956-istio-token\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:52:01.348744 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:52:01.348747    5304 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1020808c-e760-46e7-99e8-66932fca1956-istio-envoy\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:52:01.348980 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:52:01.348757    5304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t67w7\" (UniqueName: \"kubernetes.io/projected/1020808c-e760-46e7-99e8-66932fca1956-kube-api-access-t67w7\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:52:01.917518 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:52:01.917475    5304 generic.go:356] "Generic (PLEG): container finished" podID="1020808c-e760-46e7-99e8-66932fca1956" containerID="4f574ad5724d4e728c194a366add175a69e11f44da92c080ed4c3dcaad1512a1" exitCode=0
Nov 02 15:52:01.917734 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:52:01.917535    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-d79991-openshift-default-74c857f7cc-p2snd"
Nov 02 15:52:01.917734 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:52:01.917566    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-d79991-openshift-default-74c857f7cc-p2snd" event={"ID":"1020808c-e760-46e7-99e8-66932fca1956","Type":"ContainerDied","Data":"4f574ad5724d4e728c194a366add175a69e11f44da92c080ed4c3dcaad1512a1"}
Nov 02 15:52:01.917734 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:52:01.917674    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-d79991-openshift-default-74c857f7cc-p2snd" event={"ID":"1020808c-e760-46e7-99e8-66932fca1956","Type":"ContainerDied","Data":"97d29ccda53d96ac36fc9fe56ca9c00fea498d0f8793064b8e700feb2f52268e"}
Nov 02 15:52:01.917734 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:52:01.917706    5304 scope.go:122] "RemoveContainer" containerID="4f574ad5724d4e728c194a366add175a69e11f44da92c080ed4c3dcaad1512a1"
Nov 02 15:52:01.929769 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:52:01.929742    5304 scope.go:122] "RemoveContainer" containerID="554b43614667dd9cb4b1f94d5281367be991545f9ad45480363d8879d52c5001"
Nov 02 15:52:01.940160 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:52:01.940138    5304 scope.go:122] "RemoveContainer" containerID="4f574ad5724d4e728c194a366add175a69e11f44da92c080ed4c3dcaad1512a1"
Nov 02 15:52:01.940477 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:52:01.940460    5304 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f574ad5724d4e728c194a366add175a69e11f44da92c080ed4c3dcaad1512a1\": container with ID starting with 4f574ad5724d4e728c194a366add175a69e11f44da92c080ed4c3dcaad1512a1 not found: ID does not exist" containerID="4f574ad5724d4e728c194a366add175a69e11f44da92c080ed4c3dcaad1512a1"
Nov 02 15:52:01.940551 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:52:01.940486    5304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f574ad5724d4e728c194a366add175a69e11f44da92c080ed4c3dcaad1512a1"} err="failed to get container status \"4f574ad5724d4e728c194a366add175a69e11f44da92c080ed4c3dcaad1512a1\": rpc error: code = NotFound desc = could not find container \"4f574ad5724d4e728c194a366add175a69e11f44da92c080ed4c3dcaad1512a1\": container with ID starting with 4f574ad5724d4e728c194a366add175a69e11f44da92c080ed4c3dcaad1512a1 not found: ID does not exist"
Nov 02 15:52:01.940551 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:52:01.940504    5304 scope.go:122] "RemoveContainer" containerID="554b43614667dd9cb4b1f94d5281367be991545f9ad45480363d8879d52c5001"
Nov 02 15:52:01.940762 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:52:01.940747    5304 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"554b43614667dd9cb4b1f94d5281367be991545f9ad45480363d8879d52c5001\": container with ID starting with 554b43614667dd9cb4b1f94d5281367be991545f9ad45480363d8879d52c5001 not found: ID does not exist" containerID="554b43614667dd9cb4b1f94d5281367be991545f9ad45480363d8879d52c5001"
Nov 02 15:52:01.940804 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:52:01.940765    5304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"554b43614667dd9cb4b1f94d5281367be991545f9ad45480363d8879d52c5001"} err="failed to get container status \"554b43614667dd9cb4b1f94d5281367be991545f9ad45480363d8879d52c5001\": rpc error: code = NotFound desc = could not find container \"554b43614667dd9cb4b1f94d5281367be991545f9ad45480363d8879d52c5001\": container with ID starting with 554b43614667dd9cb4b1f94d5281367be991545f9ad45480363d8879d52c5001 not found: ID does not exist"
Nov 02 15:52:01.982510 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:52:01.982470    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-d79991-openshift-default-74c857f7cc-p2snd"]
Nov 02 15:52:01.994531 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:52:01.994479    5304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-d79991-openshift-default-74c857f7cc-p2snd"]
Nov 02 15:52:02.118998 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:52:02.118964    5304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1020808c-e760-46e7-99e8-66932fca1956" path="/var/lib/kubelet/pods/1020808c-e760-46e7-99e8-66932fca1956/volumes"
Nov 02 15:52:20.335735 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:52:20.335676    5304 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-f0039c-ff4996964-9pzcn_43c5260b-b089-4c06-ac76-e6e170cac754/payload-processing/0.log"
Nov 02 15:52:20.559682 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:52:20.559651    5304 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-wxjl7_2f905a4e-b982-4d66-a7bc-171cda1c78dd/payload-processing/0.log"
Nov 02 15:52:20.766494 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:52:20.766415    5304 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-wxjl7_2f905a4e-b982-4d66-a7bc-171cda1c78dd/payload-processing/0.log"
Nov 02 15:52:24.798794 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:52:24.798755    5304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k6fjk" containerName="extractor"
Nov 02 15:52:30.508402 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:52:30.508371    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-453864-openshift-default-77d4cd74c7-bp7wp"]
Nov 02 15:52:30.508862 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:52:30.508596    5304 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w0-453864-openshift-default-77d4cd74c7-bp7wp" podUID="73b5a23d-1e75-4beb-920e-8953826899af" containerName="istio-proxy" containerID="cri-o://b2f58ba4fa7f872702bd7b3dffcdec3fee1050b47c7dcc372632cd4561c2f665" gracePeriod=30
Nov 02 15:52:33.625746 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:52:33.625711    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-af33a6-7db8978884-rnjtx"]
Nov 02 15:52:33.626238 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:52:33.625920    5304 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-af33a6-7db8978884-rnjtx" podUID="0c18605a-898a-41ca-ba2d-e7f48597e790" containerName="payload-processing" containerID="cri-o://07b216156b332eb04fee1014a21768502042bf33cae85026fc027f99ed6381f9" gracePeriod=30
Nov 02 15:52:33.659448 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:52:33.659402    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-af33a6-846475bc47-4lq5l"]
Nov 02 15:52:33.659679 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:52:33.659651    5304 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-af33a6-846475bc47-4lq5l" podUID="69df020e-436a-4ae9-b07b-b138bb2ff1da" containerName="payload-pre-processing" containerID="cri-o://47b4a72b4fec46c50c18aacbadfea0b18f31358203d4ab10b8903cde59045c47" gracePeriod=30
Nov 02 15:52:33.989252 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:52:33.987604    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-af33a6-7db8978884-rnjtx"
Nov 02 15:52:33.995954 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:52:33.995847    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-af33a6-846475bc47-4lq5l"
Nov 02 15:52:34.112648 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:52:34.112610    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69df020e-436a-4ae9-b07b-b138bb2ff1da-kube-api-access-b7znk\" (UniqueName: \"kubernetes.io/projected/69df020e-436a-4ae9-b07b-b138bb2ff1da-kube-api-access-b7znk\") pod \"69df020e-436a-4ae9-b07b-b138bb2ff1da\" (UID: \"69df020e-436a-4ae9-b07b-b138bb2ff1da\") "
Nov 02 15:52:34.113059 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:52:34.113034    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c18605a-898a-41ca-ba2d-e7f48597e790-kube-api-access-hxmrf\" (UniqueName: \"kubernetes.io/projected/0c18605a-898a-41ca-ba2d-e7f48597e790-kube-api-access-hxmrf\") pod \"0c18605a-898a-41ca-ba2d-e7f48597e790\" (UID: \"0c18605a-898a-41ca-ba2d-e7f48597e790\") "
Nov 02 15:52:34.113246 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:52:34.113221    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/69df020e-436a-4ae9-b07b-b138bb2ff1da-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69df020e-436a-4ae9-b07b-b138bb2ff1da-plugins-config-volume\") pod \"69df020e-436a-4ae9-b07b-b138bb2ff1da\" (UID: \"69df020e-436a-4ae9-b07b-b138bb2ff1da\") "
Nov 02 15:52:34.113450 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:52:34.113334    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0c18605a-898a-41ca-ba2d-e7f48597e790-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c18605a-898a-41ca-ba2d-e7f48597e790-plugins-config-volume\") pod \"0c18605a-898a-41ca-ba2d-e7f48597e790\" (UID: \"0c18605a-898a-41ca-ba2d-e7f48597e790\") "
Nov 02 15:52:34.113976 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:52:34.113950    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69df020e-436a-4ae9-b07b-b138bb2ff1da-plugins-config-volume" pod "69df020e-436a-4ae9-b07b-b138bb2ff1da" (UID: "69df020e-436a-4ae9-b07b-b138bb2ff1da"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:52:34.114116 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:52:34.114093    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c18605a-898a-41ca-ba2d-e7f48597e790-plugins-config-volume" pod "0c18605a-898a-41ca-ba2d-e7f48597e790" (UID: "0c18605a-898a-41ca-ba2d-e7f48597e790"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:52:34.116431 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:52:34.116403    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69df020e-436a-4ae9-b07b-b138bb2ff1da-kube-api-access-b7znk" pod "69df020e-436a-4ae9-b07b-b138bb2ff1da" (UID: "69df020e-436a-4ae9-b07b-b138bb2ff1da"). InnerVolumeSpecName "kube-api-access-b7znk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:52:34.116718 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:52:34.116689    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c18605a-898a-41ca-ba2d-e7f48597e790-kube-api-access-hxmrf" pod "0c18605a-898a-41ca-ba2d-e7f48597e790" (UID: "0c18605a-898a-41ca-ba2d-e7f48597e790"). InnerVolumeSpecName "kube-api-access-hxmrf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:52:34.168903 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:52:34.168870    5304 generic.go:356] "Generic (PLEG): container finished" podID="69df020e-436a-4ae9-b07b-b138bb2ff1da" containerID="47b4a72b4fec46c50c18aacbadfea0b18f31358203d4ab10b8903cde59045c47" exitCode=0
Nov 02 15:52:34.169105 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:52:34.168917    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-af33a6-846475bc47-4lq5l"
Nov 02 15:52:34.169105 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:52:34.168958    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-af33a6-846475bc47-4lq5l" event={"ID":"69df020e-436a-4ae9-b07b-b138bb2ff1da","Type":"ContainerDied","Data":"47b4a72b4fec46c50c18aacbadfea0b18f31358203d4ab10b8903cde59045c47"}
Nov 02 15:52:34.169105 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:52:34.169013    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-af33a6-846475bc47-4lq5l" event={"ID":"69df020e-436a-4ae9-b07b-b138bb2ff1da","Type":"ContainerDied","Data":"aeef7ae45756e56b4e71beeaf89f533b019e3ce483c686a514545e44b4e224f7"}
Nov 02 15:52:34.169105 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:52:34.169042    5304 scope.go:122] "RemoveContainer" containerID="47b4a72b4fec46c50c18aacbadfea0b18f31358203d4ab10b8903cde59045c47"
Nov 02 15:52:34.170419 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:52:34.170396    5304 generic.go:356] "Generic (PLEG): container finished" podID="0c18605a-898a-41ca-ba2d-e7f48597e790" containerID="07b216156b332eb04fee1014a21768502042bf33cae85026fc027f99ed6381f9" exitCode=0
Nov 02 15:52:34.170524 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:52:34.170450    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-af33a6-7db8978884-rnjtx"
Nov 02 15:52:34.170524 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:52:34.170487    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-af33a6-7db8978884-rnjtx" event={"ID":"0c18605a-898a-41ca-ba2d-e7f48597e790","Type":"ContainerDied","Data":"07b216156b332eb04fee1014a21768502042bf33cae85026fc027f99ed6381f9"}
Nov 02 15:52:34.170623 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:52:34.170535    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-af33a6-7db8978884-rnjtx" event={"ID":"0c18605a-898a-41ca-ba2d-e7f48597e790","Type":"ContainerDied","Data":"64ee3c8ea8a43f05b8549a2478444b3df01871ff747f25967f7626c44d0cb74c"}
Nov 02 15:52:34.179505 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:52:34.179484    5304 scope.go:122] "RemoveContainer" containerID="47b4a72b4fec46c50c18aacbadfea0b18f31358203d4ab10b8903cde59045c47"
Nov 02 15:52:34.179787 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:52:34.179769    5304 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"47b4a72b4fec46c50c18aacbadfea0b18f31358203d4ab10b8903cde59045c47\": container with ID starting with 47b4a72b4fec46c50c18aacbadfea0b18f31358203d4ab10b8903cde59045c47 not found: ID does not exist" containerID="47b4a72b4fec46c50c18aacbadfea0b18f31358203d4ab10b8903cde59045c47"
Nov 02 15:52:34.179839 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:52:34.179801    5304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47b4a72b4fec46c50c18aacbadfea0b18f31358203d4ab10b8903cde59045c47"} err="failed to get container status \"47b4a72b4fec46c50c18aacbadfea0b18f31358203d4ab10b8903cde59045c47\": rpc error: code = NotFound desc = could not find container \"47b4a72b4fec46c50c18aacbadfea0b18f31358203d4ab10b8903cde59045c47\": container with ID starting with 47b4a72b4fec46c50c18aacbadfea0b18f31358203d4ab10b8903cde59045c47 not found: ID does not exist"
Nov 02 15:52:34.179839 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:52:34.179819    5304 scope.go:122] "RemoveContainer" containerID="07b216156b332eb04fee1014a21768502042bf33cae85026fc027f99ed6381f9"
Nov 02 15:52:34.189185 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:52:34.189110    5304 scope.go:122] "RemoveContainer" containerID="07b216156b332eb04fee1014a21768502042bf33cae85026fc027f99ed6381f9"
Nov 02 15:52:34.189434 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:52:34.189415    5304 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"07b216156b332eb04fee1014a21768502042bf33cae85026fc027f99ed6381f9\": container with ID starting with 07b216156b332eb04fee1014a21768502042bf33cae85026fc027f99ed6381f9 not found: ID does not exist" containerID="07b216156b332eb04fee1014a21768502042bf33cae85026fc027f99ed6381f9"
Nov 02 15:52:34.189496 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:52:34.189440    5304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07b216156b332eb04fee1014a21768502042bf33cae85026fc027f99ed6381f9"} err="failed to get container status \"07b216156b332eb04fee1014a21768502042bf33cae85026fc027f99ed6381f9\": rpc error: code = NotFound desc = could not find container \"07b216156b332eb04fee1014a21768502042bf33cae85026fc027f99ed6381f9\": container with ID starting with 07b216156b332eb04fee1014a21768502042bf33cae85026fc027f99ed6381f9 not found: ID does not exist"
Nov 02 15:52:34.215564 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:52:34.215529    5304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b7znk\" (UniqueName: \"kubernetes.io/projected/69df020e-436a-4ae9-b07b-b138bb2ff1da-kube-api-access-b7znk\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:52:34.215693 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:52:34.215568    5304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hxmrf\" (UniqueName: \"kubernetes.io/projected/0c18605a-898a-41ca-ba2d-e7f48597e790-kube-api-access-hxmrf\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:52:34.215693 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:52:34.215585    5304 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69df020e-436a-4ae9-b07b-b138bb2ff1da-plugins-config-volume\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:52:34.215693 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:52:34.215599    5304 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c18605a-898a-41ca-ba2d-e7f48597e790-plugins-config-volume\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:52:34.227843 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:52:34.227815    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-af33a6-7db8978884-rnjtx"]
Nov 02 15:52:34.232368 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:52:34.232343    5304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-af33a6-7db8978884-rnjtx"]
Nov 02 15:52:34.266512 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:52:34.266482    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-af33a6-846475bc47-4lq5l"]
Nov 02 15:52:34.271927 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:52:34.271900    5304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-af33a6-846475bc47-4lq5l"]
Nov 02 15:52:35.792312 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:52:35.792251    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-453864-openshift-default-77d4cd74c7-bp7wp"
Nov 02 15:52:35.936293 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:52:35.936259    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73b5a23d-1e75-4beb-920e-8953826899af-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73b5a23d-1e75-4beb-920e-8953826899af-istio-data\") pod \"73b5a23d-1e75-4beb-920e-8953826899af\" (UID: \"73b5a23d-1e75-4beb-920e-8953826899af\") "
Nov 02 15:52:35.936293 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:52:35.936298    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/73b5a23d-1e75-4beb-920e-8953826899af-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73b5a23d-1e75-4beb-920e-8953826899af-istio-podinfo\") pod \"73b5a23d-1e75-4beb-920e-8953826899af\" (UID: \"73b5a23d-1e75-4beb-920e-8953826899af\") "
Nov 02 15:52:35.936552 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:52:35.936337    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73b5a23d-1e75-4beb-920e-8953826899af-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73b5a23d-1e75-4beb-920e-8953826899af-workload-certs\") pod \"73b5a23d-1e75-4beb-920e-8953826899af\" (UID: \"73b5a23d-1e75-4beb-920e-8953826899af\") "
Nov 02 15:52:35.936552 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:52:35.936404    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73b5a23d-1e75-4beb-920e-8953826899af-kube-api-access-jspkf\" (UniqueName: \"kubernetes.io/projected/73b5a23d-1e75-4beb-920e-8953826899af-kube-api-access-jspkf\") pod \"73b5a23d-1e75-4beb-920e-8953826899af\" (UID: \"73b5a23d-1e75-4beb-920e-8953826899af\") "
Nov 02 15:52:35.936552 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:52:35.936433    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/73b5a23d-1e75-4beb-920e-8953826899af-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73b5a23d-1e75-4beb-920e-8953826899af-istiod-ca-cert\") pod \"73b5a23d-1e75-4beb-920e-8953826899af\" (UID: \"73b5a23d-1e75-4beb-920e-8953826899af\") "
Nov 02 15:52:35.936552 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:52:35.936525    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73b5a23d-1e75-4beb-920e-8953826899af-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73b5a23d-1e75-4beb-920e-8953826899af-workload-socket\") pod \"73b5a23d-1e75-4beb-920e-8953826899af\" (UID: \"73b5a23d-1e75-4beb-920e-8953826899af\") "
Nov 02 15:52:35.936778 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:52:35.936555    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73b5a23d-1e75-4beb-920e-8953826899af-istio-data" pod "73b5a23d-1e75-4beb-920e-8953826899af" (UID: "73b5a23d-1e75-4beb-920e-8953826899af"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:52:35.936778 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:52:35.936574    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73b5a23d-1e75-4beb-920e-8953826899af-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73b5a23d-1e75-4beb-920e-8953826899af-istio-envoy\") pod \"73b5a23d-1e75-4beb-920e-8953826899af\" (UID: \"73b5a23d-1e75-4beb-920e-8953826899af\") "
Nov 02 15:52:35.936778 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:52:35.936674    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73b5a23d-1e75-4beb-920e-8953826899af-istio-token\" (UniqueName: \"kubernetes.io/projected/73b5a23d-1e75-4beb-920e-8953826899af-istio-token\") pod \"73b5a23d-1e75-4beb-920e-8953826899af\" (UID: \"73b5a23d-1e75-4beb-920e-8953826899af\") "
Nov 02 15:52:35.936778 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:52:35.936700    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73b5a23d-1e75-4beb-920e-8953826899af-workload-certs" pod "73b5a23d-1e75-4beb-920e-8953826899af" (UID: "73b5a23d-1e75-4beb-920e-8953826899af"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:52:35.937004 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:52:35.936779    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73b5a23d-1e75-4beb-920e-8953826899af-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73b5a23d-1e75-4beb-920e-8953826899af-credential-socket\") pod \"73b5a23d-1e75-4beb-920e-8953826899af\" (UID: \"73b5a23d-1e75-4beb-920e-8953826899af\") "
Nov 02 15:52:35.937004 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:52:35.936801    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73b5a23d-1e75-4beb-920e-8953826899af-workload-socket" pod "73b5a23d-1e75-4beb-920e-8953826899af" (UID: "73b5a23d-1e75-4beb-920e-8953826899af"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:52:35.937004 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:52:35.936915    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73b5a23d-1e75-4beb-920e-8953826899af-istiod-ca-cert" pod "73b5a23d-1e75-4beb-920e-8953826899af" (UID: "73b5a23d-1e75-4beb-920e-8953826899af"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:52:35.937214 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:52:35.937036    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73b5a23d-1e75-4beb-920e-8953826899af-credential-socket" pod "73b5a23d-1e75-4beb-920e-8953826899af" (UID: "73b5a23d-1e75-4beb-920e-8953826899af"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:52:35.937405 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:52:35.937378    5304 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73b5a23d-1e75-4beb-920e-8953826899af-credential-socket\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:52:35.937553 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:52:35.937408    5304 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73b5a23d-1e75-4beb-920e-8953826899af-istio-data\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:52:35.937553 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:52:35.937423    5304 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73b5a23d-1e75-4beb-920e-8953826899af-workload-certs\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:52:35.937553 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:52:35.937436    5304 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73b5a23d-1e75-4beb-920e-8953826899af-istiod-ca-cert\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:52:35.937553 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:52:35.937450    5304 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73b5a23d-1e75-4beb-920e-8953826899af-workload-socket\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:52:35.938894 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:52:35.938856    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/73b5a23d-1e75-4beb-920e-8953826899af-istio-podinfo" pod "73b5a23d-1e75-4beb-920e-8953826899af" (UID: "73b5a23d-1e75-4beb-920e-8953826899af"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 15:52:35.939000 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:52:35.938893    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73b5a23d-1e75-4beb-920e-8953826899af-istio-token" pod "73b5a23d-1e75-4beb-920e-8953826899af" (UID: "73b5a23d-1e75-4beb-920e-8953826899af"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:52:35.939000 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:52:35.938909    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73b5a23d-1e75-4beb-920e-8953826899af-istio-envoy" pod "73b5a23d-1e75-4beb-920e-8953826899af" (UID: "73b5a23d-1e75-4beb-920e-8953826899af"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:52:35.939000 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:52:35.938903    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73b5a23d-1e75-4beb-920e-8953826899af-kube-api-access-jspkf" pod "73b5a23d-1e75-4beb-920e-8953826899af" (UID: "73b5a23d-1e75-4beb-920e-8953826899af"). InnerVolumeSpecName "kube-api-access-jspkf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:52:35.984676 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:52:35.984645    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-6af50f/distinct-w0-6af50f-kserve-c6d994647-hbj9z"]
Nov 02 15:52:36.038499 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:52:36.038455    5304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jspkf\" (UniqueName: \"kubernetes.io/projected/73b5a23d-1e75-4beb-920e-8953826899af-kube-api-access-jspkf\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:52:36.038499 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:52:36.038489    5304 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73b5a23d-1e75-4beb-920e-8953826899af-istio-envoy\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:52:36.038499 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:52:36.038503    5304 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/73b5a23d-1e75-4beb-920e-8953826899af-istio-token\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:52:36.038748 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:52:36.038514    5304 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73b5a23d-1e75-4beb-920e-8953826899af-istio-podinfo\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:52:36.119705 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:52:36.119676    5304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0c18605a-898a-41ca-ba2d-e7f48597e790" path="/var/lib/kubelet/pods/0c18605a-898a-41ca-ba2d-e7f48597e790/volumes"
Nov 02 15:52:36.120084 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:52:36.120070    5304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="69df020e-436a-4ae9-b07b-b138bb2ff1da" path="/var/lib/kubelet/pods/69df020e-436a-4ae9-b07b-b138bb2ff1da/volumes"
Nov 02 15:52:36.185435 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:52:36.185392    5304 generic.go:356] "Generic (PLEG): container finished" podID="73b5a23d-1e75-4beb-920e-8953826899af" containerID="b2f58ba4fa7f872702bd7b3dffcdec3fee1050b47c7dcc372632cd4561c2f665" exitCode=0
Nov 02 15:52:36.185635 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:52:36.185442    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-453864-openshift-default-77d4cd74c7-bp7wp"
Nov 02 15:52:36.185635 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:52:36.185431    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-453864-openshift-default-77d4cd74c7-bp7wp" event={"ID":"73b5a23d-1e75-4beb-920e-8953826899af","Type":"ContainerDied","Data":"b2f58ba4fa7f872702bd7b3dffcdec3fee1050b47c7dcc372632cd4561c2f665"}
Nov 02 15:52:36.185635 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:52:36.185582    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-453864-openshift-default-77d4cd74c7-bp7wp" event={"ID":"73b5a23d-1e75-4beb-920e-8953826899af","Type":"ContainerDied","Data":"335685da465ea3eb2e51af2a0f31b60452b7db4a047d19f174fb4cd8e9e9bbe4"}
Nov 02 15:52:36.185635 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:52:36.185622    5304 scope.go:122] "RemoveContainer" containerID="b2f58ba4fa7f872702bd7b3dffcdec3fee1050b47c7dcc372632cd4561c2f665"
Nov 02 15:52:36.196333 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:52:36.196316    5304 scope.go:122] "RemoveContainer" containerID="8dc3c1116b7a792b9475dfa4ec638acd8ca79da4050680fd317a05e51bd4dd81"
Nov 02 15:52:36.208965 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:52:36.208945    5304 scope.go:122] "RemoveContainer" containerID="b2f58ba4fa7f872702bd7b3dffcdec3fee1050b47c7dcc372632cd4561c2f665"
Nov 02 15:52:36.209249 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:52:36.209228    5304 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b2f58ba4fa7f872702bd7b3dffcdec3fee1050b47c7dcc372632cd4561c2f665\": container with ID starting with b2f58ba4fa7f872702bd7b3dffcdec3fee1050b47c7dcc372632cd4561c2f665 not found: ID does not exist" containerID="b2f58ba4fa7f872702bd7b3dffcdec3fee1050b47c7dcc372632cd4561c2f665"
Nov 02 15:52:36.209312 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:52:36.209259    5304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2f58ba4fa7f872702bd7b3dffcdec3fee1050b47c7dcc372632cd4561c2f665"} err="failed to get container status \"b2f58ba4fa7f872702bd7b3dffcdec3fee1050b47c7dcc372632cd4561c2f665\": rpc error: code = NotFound desc = could not find container \"b2f58ba4fa7f872702bd7b3dffcdec3fee1050b47c7dcc372632cd4561c2f665\": container with ID starting with b2f58ba4fa7f872702bd7b3dffcdec3fee1050b47c7dcc372632cd4561c2f665 not found: ID does not exist"
Nov 02 15:52:36.209312 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:52:36.209277    5304 scope.go:122] "RemoveContainer" containerID="8dc3c1116b7a792b9475dfa4ec638acd8ca79da4050680fd317a05e51bd4dd81"
Nov 02 15:52:36.209548 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:52:36.209530    5304 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8dc3c1116b7a792b9475dfa4ec638acd8ca79da4050680fd317a05e51bd4dd81\": container with ID starting with 8dc3c1116b7a792b9475dfa4ec638acd8ca79da4050680fd317a05e51bd4dd81 not found: ID does not exist" containerID="8dc3c1116b7a792b9475dfa4ec638acd8ca79da4050680fd317a05e51bd4dd81"
Nov 02 15:52:36.209589 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:52:36.209554    5304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8dc3c1116b7a792b9475dfa4ec638acd8ca79da4050680fd317a05e51bd4dd81"} err="failed to get container status \"8dc3c1116b7a792b9475dfa4ec638acd8ca79da4050680fd317a05e51bd4dd81\": rpc error: code = NotFound desc = could not find container \"8dc3c1116b7a792b9475dfa4ec638acd8ca79da4050680fd317a05e51bd4dd81\": container with ID starting with 8dc3c1116b7a792b9475dfa4ec638acd8ca79da4050680fd317a05e51bd4dd81 not found: ID does not exist"
Nov 02 15:52:36.239438 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:52:36.239406    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-453864-openshift-default-77d4cd74c7-bp7wp"]
Nov 02 15:52:36.244854 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:52:36.244829    5304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-453864-openshift-default-77d4cd74c7-bp7wp"]
Nov 02 15:52:38.118729 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:52:38.118690    5304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="73b5a23d-1e75-4beb-920e-8953826899af" path="/var/lib/kubelet/pods/73b5a23d-1e75-4beb-920e-8953826899af/volumes"
Nov 02 15:52:41.051489 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:52:41.051449    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-e40940-5dc9c99d5d-dwtqt"]
Nov 02 15:52:41.051967 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:52:41.051685    5304 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-e40940-5dc9c99d5d-dwtqt" podUID="99313de5-a08c-4544-86ec-54aeedfe4ec2" containerName="payload-processing" containerID="cri-o://0c1481f6c3b5b393817cbb1bb02ff827c5918c40deaeb074ace187c0b17bf0da" gracePeriod=30
Nov 02 15:52:41.075830 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:52:41.075799    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-e40940-7c4db584c5-jjf2w"]
Nov 02 15:52:41.076094 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:52:41.076049    5304 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e40940-7c4db584c5-jjf2w" podUID="e04e8ce8-dec7-47b8-9eba-a8f87a539790" containerName="payload-pre-processing" containerID="cri-o://88f474c9c2c74216db260321e1bab527903d86129dd2a5298ee3c518b266fab9" gracePeriod=30
Nov 02 15:52:41.223960 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:52:41.222843    5304 generic.go:356] "Generic (PLEG): container finished" podID="e04e8ce8-dec7-47b8-9eba-a8f87a539790" containerID="88f474c9c2c74216db260321e1bab527903d86129dd2a5298ee3c518b266fab9" exitCode=0
Nov 02 15:52:41.223960 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:52:41.223054    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e40940-7c4db584c5-jjf2w" event={"ID":"e04e8ce8-dec7-47b8-9eba-a8f87a539790","Type":"ContainerDied","Data":"88f474c9c2c74216db260321e1bab527903d86129dd2a5298ee3c518b266fab9"}
Nov 02 15:52:41.225499 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:52:41.225472    5304 generic.go:356] "Generic (PLEG): container finished" podID="99313de5-a08c-4544-86ec-54aeedfe4ec2" containerID="0c1481f6c3b5b393817cbb1bb02ff827c5918c40deaeb074ace187c0b17bf0da" exitCode=0
Nov 02 15:52:41.225639 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:52:41.225535    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-e40940-5dc9c99d5d-dwtqt" event={"ID":"99313de5-a08c-4544-86ec-54aeedfe4ec2","Type":"ContainerDied","Data":"0c1481f6c3b5b393817cbb1bb02ff827c5918c40deaeb074ace187c0b17bf0da"}
Nov 02 15:52:41.461516 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:52:41.461376    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e40940-7c4db584c5-jjf2w"
Nov 02 15:52:41.464245 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:52:41.464167    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-e40940-5dc9c99d5d-dwtqt"
Nov 02 15:52:41.541624 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:52:41.541591    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/99313de5-a08c-4544-86ec-54aeedfe4ec2-kube-api-access-wmw6l\" (UniqueName: \"kubernetes.io/projected/99313de5-a08c-4544-86ec-54aeedfe4ec2-kube-api-access-wmw6l\") pod \"99313de5-a08c-4544-86ec-54aeedfe4ec2\" (UID: \"99313de5-a08c-4544-86ec-54aeedfe4ec2\") "
Nov 02 15:52:41.541816 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:52:41.541673    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/99313de5-a08c-4544-86ec-54aeedfe4ec2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99313de5-a08c-4544-86ec-54aeedfe4ec2-plugins-config-volume\") pod \"99313de5-a08c-4544-86ec-54aeedfe4ec2\" (UID: \"99313de5-a08c-4544-86ec-54aeedfe4ec2\") "
Nov 02 15:52:41.541816 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:52:41.541731    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e04e8ce8-dec7-47b8-9eba-a8f87a539790-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e04e8ce8-dec7-47b8-9eba-a8f87a539790-plugins-config-volume\") pod \"e04e8ce8-dec7-47b8-9eba-a8f87a539790\" (UID: \"e04e8ce8-dec7-47b8-9eba-a8f87a539790\") "
Nov 02 15:52:41.541957 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:52:41.541858    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e04e8ce8-dec7-47b8-9eba-a8f87a539790-kube-api-access-5pz76\" (UniqueName: \"kubernetes.io/projected/e04e8ce8-dec7-47b8-9eba-a8f87a539790-kube-api-access-5pz76\") pod \"e04e8ce8-dec7-47b8-9eba-a8f87a539790\" (UID: \"e04e8ce8-dec7-47b8-9eba-a8f87a539790\") "
Nov 02 15:52:41.542177 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:52:41.542141    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99313de5-a08c-4544-86ec-54aeedfe4ec2-plugins-config-volume" pod "99313de5-a08c-4544-86ec-54aeedfe4ec2" (UID: "99313de5-a08c-4544-86ec-54aeedfe4ec2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:52:41.542444 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:52:41.542344    5304 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99313de5-a08c-4544-86ec-54aeedfe4ec2-plugins-config-volume\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:52:41.542580 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:52:41.542557    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e04e8ce8-dec7-47b8-9eba-a8f87a539790-plugins-config-volume" pod "e04e8ce8-dec7-47b8-9eba-a8f87a539790" (UID: "e04e8ce8-dec7-47b8-9eba-a8f87a539790"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:52:41.544414 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:52:41.544380    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99313de5-a08c-4544-86ec-54aeedfe4ec2-kube-api-access-wmw6l" pod "99313de5-a08c-4544-86ec-54aeedfe4ec2" (UID: "99313de5-a08c-4544-86ec-54aeedfe4ec2"). InnerVolumeSpecName "kube-api-access-wmw6l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:52:41.544520 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:52:41.544409    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e04e8ce8-dec7-47b8-9eba-a8f87a539790-kube-api-access-5pz76" pod "e04e8ce8-dec7-47b8-9eba-a8f87a539790" (UID: "e04e8ce8-dec7-47b8-9eba-a8f87a539790"). InnerVolumeSpecName "kube-api-access-5pz76". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:52:41.643102 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:52:41.643067    5304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5pz76\" (UniqueName: \"kubernetes.io/projected/e04e8ce8-dec7-47b8-9eba-a8f87a539790-kube-api-access-5pz76\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:52:41.643102 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:52:41.643095    5304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wmw6l\" (UniqueName: \"kubernetes.io/projected/99313de5-a08c-4544-86ec-54aeedfe4ec2-kube-api-access-wmw6l\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:52:41.643102 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:52:41.643106    5304 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e04e8ce8-dec7-47b8-9eba-a8f87a539790-plugins-config-volume\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:52:42.234404 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:52:42.234372    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-e40940-5dc9c99d5d-dwtqt"
Nov 02 15:52:42.234404 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:52:42.234400    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-e40940-5dc9c99d5d-dwtqt" event={"ID":"99313de5-a08c-4544-86ec-54aeedfe4ec2","Type":"ContainerDied","Data":"a6705673c26520a7ce70104144fd45dbcc62d23c88f51ff4722d300142f1df06"}
Nov 02 15:52:42.235157 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:52:42.234462    5304 scope.go:122] "RemoveContainer" containerID="0c1481f6c3b5b393817cbb1bb02ff827c5918c40deaeb074ace187c0b17bf0da"
Nov 02 15:52:42.236243 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:52:42.236008    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e40940-7c4db584c5-jjf2w" event={"ID":"e04e8ce8-dec7-47b8-9eba-a8f87a539790","Type":"ContainerDied","Data":"0bd8b5ed7ce54fa6e95e75444012cebf7ea7f5b8d6cbece15a496ca3a5b16d62"}
Nov 02 15:52:42.236243 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:52:42.236049    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e40940-7c4db584c5-jjf2w"
Nov 02 15:52:42.245857 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:52:42.245833    5304 scope.go:122] "RemoveContainer" containerID="88f474c9c2c74216db260321e1bab527903d86129dd2a5298ee3c518b266fab9"
Nov 02 15:52:42.286852 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:52:42.286822    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-e40940-5dc9c99d5d-dwtqt"]
Nov 02 15:52:42.298438 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:52:42.298411    5304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-e40940-5dc9c99d5d-dwtqt"]
Nov 02 15:52:42.329919 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:52:42.328841    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-e40940-7c4db584c5-jjf2w"]
Nov 02 15:52:42.339054 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:52:42.339025    5304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-e40940-7c4db584c5-jjf2w"]
Nov 02 15:52:42.995654 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:52:42.995623    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-af33a6-openshift-default-759865cbb9-rzw5f"]
Nov 02 15:52:42.995863 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:52:42.995841    5304 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-af33a6-openshift-default-759865cbb9-rzw5f" podUID="bdab310f-f084-4e60-9b19-3fd5e0b6a953" containerName="istio-proxy" containerID="cri-o://9d0983157d56b071b0f8f59c7711c0c35e654feb059fa2e2b5150c445b1b25b5" gracePeriod=30
Nov 02 15:52:43.155628 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:52:43.155589    5304 scope.go:122] "RemoveContainer" containerID="c0cd3ff1c2c73e720b00f22565cea7da049e32138f39195f62aa04488adde269"
Nov 02 15:52:44.119095 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:52:44.119054    5304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="99313de5-a08c-4544-86ec-54aeedfe4ec2" path="/var/lib/kubelet/pods/99313de5-a08c-4544-86ec-54aeedfe4ec2/volumes"
Nov 02 15:52:44.119578 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:52:44.119561    5304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e04e8ce8-dec7-47b8-9eba-a8f87a539790" path="/var/lib/kubelet/pods/e04e8ce8-dec7-47b8-9eba-a8f87a539790/volumes"
Nov 02 15:52:46.126837 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:52:46.126803    5304 patch_prober.go:29] interesting pod/e2e-ipp-b-af33a6-openshift-default-759865cbb9-rzw5f container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 15:52:46.127242 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:52:46.126876    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-af33a6-openshift-default-759865cbb9-rzw5f" podUID="bdab310f-f084-4e60-9b19-3fd5e0b6a953" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 15:52:48.279896 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:52:48.279859    5304 generic.go:356] "Generic (PLEG): container finished" podID="bdab310f-f084-4e60-9b19-3fd5e0b6a953" containerID="9d0983157d56b071b0f8f59c7711c0c35e654feb059fa2e2b5150c445b1b25b5" exitCode=0
Nov 02 15:52:48.280296 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:52:48.279932    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-af33a6-openshift-default-759865cbb9-rzw5f" event={"ID":"bdab310f-f084-4e60-9b19-3fd5e0b6a953","Type":"ContainerDied","Data":"9d0983157d56b071b0f8f59c7711c0c35e654feb059fa2e2b5150c445b1b25b5"}
Nov 02 15:52:48.280296 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:52:48.279995    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-af33a6-openshift-default-759865cbb9-rzw5f" event={"ID":"bdab310f-f084-4e60-9b19-3fd5e0b6a953","Type":"ContainerDied","Data":"f8a04414ae93734bf8a4f73850c089df32d00998d80b631b6b74fdafae59a025"}
Nov 02 15:52:48.280296 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:52:48.280056    5304 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f8a04414ae93734bf8a4f73850c089df32d00998d80b631b6b74fdafae59a025"
Nov 02 15:52:48.282111 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:52:48.282095    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-af33a6-openshift-default-759865cbb9-rzw5f"
Nov 02 15:52:48.329555 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:52:48.329518    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bdab310f-f084-4e60-9b19-3fd5e0b6a953-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bdab310f-f084-4e60-9b19-3fd5e0b6a953-istio-data\") pod \"bdab310f-f084-4e60-9b19-3fd5e0b6a953\" (UID: \"bdab310f-f084-4e60-9b19-3fd5e0b6a953\") "
Nov 02 15:52:48.329713 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:52:48.329604    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bdab310f-f084-4e60-9b19-3fd5e0b6a953-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bdab310f-f084-4e60-9b19-3fd5e0b6a953-istio-envoy\") pod \"bdab310f-f084-4e60-9b19-3fd5e0b6a953\" (UID: \"bdab310f-f084-4e60-9b19-3fd5e0b6a953\") "
Nov 02 15:52:48.329713 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:52:48.329646    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bdab310f-f084-4e60-9b19-3fd5e0b6a953-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdab310f-f084-4e60-9b19-3fd5e0b6a953-workload-socket\") pod \"bdab310f-f084-4e60-9b19-3fd5e0b6a953\" (UID: \"bdab310f-f084-4e60-9b19-3fd5e0b6a953\") "
Nov 02 15:52:48.329713 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:52:48.329670    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bdab310f-f084-4e60-9b19-3fd5e0b6a953-kube-api-access-sw5hb\" (UniqueName: \"kubernetes.io/projected/bdab310f-f084-4e60-9b19-3fd5e0b6a953-kube-api-access-sw5hb\") pod \"bdab310f-f084-4e60-9b19-3fd5e0b6a953\" (UID: \"bdab310f-f084-4e60-9b19-3fd5e0b6a953\") "
Nov 02 15:52:48.329713 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:52:48.329707    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bdab310f-f084-4e60-9b19-3fd5e0b6a953-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bdab310f-f084-4e60-9b19-3fd5e0b6a953-istiod-ca-cert\") pod \"bdab310f-f084-4e60-9b19-3fd5e0b6a953\" (UID: \"bdab310f-f084-4e60-9b19-3fd5e0b6a953\") "
Nov 02 15:52:48.329957 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:52:48.329753    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bdab310f-f084-4e60-9b19-3fd5e0b6a953-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdab310f-f084-4e60-9b19-3fd5e0b6a953-credential-socket\") pod \"bdab310f-f084-4e60-9b19-3fd5e0b6a953\" (UID: \"bdab310f-f084-4e60-9b19-3fd5e0b6a953\") "
Nov 02 15:52:48.329957 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:52:48.329796    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdab310f-f084-4e60-9b19-3fd5e0b6a953-istio-data" pod "bdab310f-f084-4e60-9b19-3fd5e0b6a953" (UID: "bdab310f-f084-4e60-9b19-3fd5e0b6a953"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:52:48.329957 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:52:48.329802    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bdab310f-f084-4e60-9b19-3fd5e0b6a953-istio-token\" (UniqueName: \"kubernetes.io/projected/bdab310f-f084-4e60-9b19-3fd5e0b6a953-istio-token\") pod \"bdab310f-f084-4e60-9b19-3fd5e0b6a953\" (UID: \"bdab310f-f084-4e60-9b19-3fd5e0b6a953\") "
Nov 02 15:52:48.329957 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:52:48.329936    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bdab310f-f084-4e60-9b19-3fd5e0b6a953-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bdab310f-f084-4e60-9b19-3fd5e0b6a953-istio-podinfo\") pod \"bdab310f-f084-4e60-9b19-3fd5e0b6a953\" (UID: \"bdab310f-f084-4e60-9b19-3fd5e0b6a953\") "
Nov 02 15:52:48.330313 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:52:48.329970    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdab310f-f084-4e60-9b19-3fd5e0b6a953-workload-socket" pod "bdab310f-f084-4e60-9b19-3fd5e0b6a953" (UID: "bdab310f-f084-4e60-9b19-3fd5e0b6a953"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:52:48.330313 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:52:48.330067    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bdab310f-f084-4e60-9b19-3fd5e0b6a953-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bdab310f-f084-4e60-9b19-3fd5e0b6a953-workload-certs\") pod \"bdab310f-f084-4e60-9b19-3fd5e0b6a953\" (UID: \"bdab310f-f084-4e60-9b19-3fd5e0b6a953\") "
Nov 02 15:52:48.330313 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:52:48.330107    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdab310f-f084-4e60-9b19-3fd5e0b6a953-credential-socket" pod "bdab310f-f084-4e60-9b19-3fd5e0b6a953" (UID: "bdab310f-f084-4e60-9b19-3fd5e0b6a953"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:52:48.330313 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:52:48.330208    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bdab310f-f084-4e60-9b19-3fd5e0b6a953-istiod-ca-cert" pod "bdab310f-f084-4e60-9b19-3fd5e0b6a953" (UID: "bdab310f-f084-4e60-9b19-3fd5e0b6a953"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:52:48.330539 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:52:48.330390    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdab310f-f084-4e60-9b19-3fd5e0b6a953-workload-certs" pod "bdab310f-f084-4e60-9b19-3fd5e0b6a953" (UID: "bdab310f-f084-4e60-9b19-3fd5e0b6a953"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:52:48.330603 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:52:48.330561    5304 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bdab310f-f084-4e60-9b19-3fd5e0b6a953-workload-certs\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:52:48.330603 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:52:48.330582    5304 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bdab310f-f084-4e60-9b19-3fd5e0b6a953-istio-data\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:52:48.330603 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:52:48.330597    5304 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdab310f-f084-4e60-9b19-3fd5e0b6a953-workload-socket\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:52:48.330766 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:52:48.330611    5304 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bdab310f-f084-4e60-9b19-3fd5e0b6a953-istiod-ca-cert\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:52:48.330766 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:52:48.330625    5304 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdab310f-f084-4e60-9b19-3fd5e0b6a953-credential-socket\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:52:48.332204 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:52:48.332166    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdab310f-f084-4e60-9b19-3fd5e0b6a953-istio-envoy" pod "bdab310f-f084-4e60-9b19-3fd5e0b6a953" (UID: "bdab310f-f084-4e60-9b19-3fd5e0b6a953"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:52:48.332311 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:52:48.332237    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bdab310f-f084-4e60-9b19-3fd5e0b6a953-istio-podinfo" pod "bdab310f-f084-4e60-9b19-3fd5e0b6a953" (UID: "bdab310f-f084-4e60-9b19-3fd5e0b6a953"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 15:52:48.332382 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:52:48.332362    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bdab310f-f084-4e60-9b19-3fd5e0b6a953-kube-api-access-sw5hb" pod "bdab310f-f084-4e60-9b19-3fd5e0b6a953" (UID: "bdab310f-f084-4e60-9b19-3fd5e0b6a953"). InnerVolumeSpecName "kube-api-access-sw5hb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:52:48.332446 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:52:48.332416    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bdab310f-f084-4e60-9b19-3fd5e0b6a953-istio-token" pod "bdab310f-f084-4e60-9b19-3fd5e0b6a953" (UID: "bdab310f-f084-4e60-9b19-3fd5e0b6a953"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:52:48.432025 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:52:48.431993    5304 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bdab310f-f084-4e60-9b19-3fd5e0b6a953-istio-envoy\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:52:48.432025 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:52:48.432022    5304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sw5hb\" (UniqueName: \"kubernetes.io/projected/bdab310f-f084-4e60-9b19-3fd5e0b6a953-kube-api-access-sw5hb\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:52:48.432025 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:52:48.432032    5304 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bdab310f-f084-4e60-9b19-3fd5e0b6a953-istio-token\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:52:48.432305 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:52:48.432041    5304 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bdab310f-f084-4e60-9b19-3fd5e0b6a953-istio-podinfo\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:52:49.287153 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:52:49.287103    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-af33a6-openshift-default-759865cbb9-rzw5f"
Nov 02 15:52:49.352364 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:52:49.352332    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-af33a6-openshift-default-759865cbb9-rzw5f"]
Nov 02 15:52:49.363569 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:52:49.363535    5304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-af33a6-openshift-default-759865cbb9-rzw5f"]
Nov 02 15:52:49.534865 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:52:49.534831    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-e40940-openshift-default-6fb44b465f-75t4b"]
Nov 02 15:52:49.535156 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:52:49.535108    5304 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-e40940-openshift-default-6fb44b465f-75t4b" podUID="d7ab8a29-4410-4fde-8975-49c13b3ff1fb" containerName="istio-proxy" containerID="cri-o://334e92c952bf360c8eca5a791970798152aaf9ca276333fe8a507a4a7e643043" gracePeriod=30
Nov 02 15:52:50.119096 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:52:50.119061    5304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bdab310f-f084-4e60-9b19-3fd5e0b6a953" path="/var/lib/kubelet/pods/bdab310f-f084-4e60-9b19-3fd5e0b6a953/volumes"
Nov 02 15:52:51.005460 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:52:51.005419    5304 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-6af50f/distinct-w0-6af50f-kserve-c6d994647-hbj9z" podUID="3dca5bf3-8e08-47a0-936e-205670d59ede" containerName="main" containerID="cri-o://5b732982a0835eb5b45687c64c425c6d675c98581f1ec840e03e88464373b88e" gracePeriod=45
Nov 02 15:52:51.278999 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:52:51.278973    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-6af50f/distinct-w0-6af50f-kserve-c6d994647-hbj9z"
Nov 02 15:52:51.299944 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:52:51.299906    5304 generic.go:356] "Generic (PLEG): container finished" podID="3dca5bf3-8e08-47a0-936e-205670d59ede" containerID="5b732982a0835eb5b45687c64c425c6d675c98581f1ec840e03e88464373b88e" exitCode=0
Nov 02 15:52:51.300166 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:52:51.299951    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-6af50f/distinct-w0-6af50f-kserve-c6d994647-hbj9z"
Nov 02 15:52:51.300166 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:52:51.299985    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-6af50f/distinct-w0-6af50f-kserve-c6d994647-hbj9z" event={"ID":"3dca5bf3-8e08-47a0-936e-205670d59ede","Type":"ContainerDied","Data":"5b732982a0835eb5b45687c64c425c6d675c98581f1ec840e03e88464373b88e"}
Nov 02 15:52:51.300166 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:52:51.300029    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-6af50f/distinct-w0-6af50f-kserve-c6d994647-hbj9z" event={"ID":"3dca5bf3-8e08-47a0-936e-205670d59ede","Type":"ContainerDied","Data":"b2e1da033c45801a5beba3421a1deeb7613441bd1df7e6a86d456065404bde69"}
Nov 02 15:52:51.300166 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:52:51.300048    5304 scope.go:122] "RemoveContainer" containerID="5b732982a0835eb5b45687c64c425c6d675c98581f1ec840e03e88464373b88e"
Nov 02 15:52:51.313011 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:52:51.312991    5304 scope.go:122] "RemoveContainer" containerID="5b732982a0835eb5b45687c64c425c6d675c98581f1ec840e03e88464373b88e"
Nov 02 15:52:51.313370 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:52:51.313346    5304 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5b732982a0835eb5b45687c64c425c6d675c98581f1ec840e03e88464373b88e\": container with ID starting with 5b732982a0835eb5b45687c64c425c6d675c98581f1ec840e03e88464373b88e not found: ID does not exist" containerID="5b732982a0835eb5b45687c64c425c6d675c98581f1ec840e03e88464373b88e"
Nov 02 15:52:51.313468 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:52:51.313379    5304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b732982a0835eb5b45687c64c425c6d675c98581f1ec840e03e88464373b88e"} err="failed to get container status \"5b732982a0835eb5b45687c64c425c6d675c98581f1ec840e03e88464373b88e\": rpc error: code = NotFound desc = could not find container \"5b732982a0835eb5b45687c64c425c6d675c98581f1ec840e03e88464373b88e\": container with ID starting with 5b732982a0835eb5b45687c64c425c6d675c98581f1ec840e03e88464373b88e not found: ID does not exist"
Nov 02 15:52:51.373434 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:52:51.373401    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3dca5bf3-8e08-47a0-936e-205670d59ede-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3dca5bf3-8e08-47a0-936e-205670d59ede-tmp-dir\") pod \"3dca5bf3-8e08-47a0-936e-205670d59ede\" (UID: \"3dca5bf3-8e08-47a0-936e-205670d59ede\") "
Nov 02 15:52:51.373614 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:52:51.373495    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3dca5bf3-8e08-47a0-936e-205670d59ede-kube-api-access-zxs7c\" (UniqueName: \"kubernetes.io/projected/3dca5bf3-8e08-47a0-936e-205670d59ede-kube-api-access-zxs7c\") pod \"3dca5bf3-8e08-47a0-936e-205670d59ede\" (UID: \"3dca5bf3-8e08-47a0-936e-205670d59ede\") "
Nov 02 15:52:51.373614 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:52:51.373534    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3dca5bf3-8e08-47a0-936e-205670d59ede-home\" (UniqueName: \"kubernetes.io/empty-dir/3dca5bf3-8e08-47a0-936e-205670d59ede-home\") pod \"3dca5bf3-8e08-47a0-936e-205670d59ede\" (UID: \"3dca5bf3-8e08-47a0-936e-205670d59ede\") "
Nov 02 15:52:51.373614 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:52:51.373567    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3dca5bf3-8e08-47a0-936e-205670d59ede-tls-certs\" (UniqueName: \"kubernetes.io/secret/3dca5bf3-8e08-47a0-936e-205670d59ede-tls-certs\") pod \"3dca5bf3-8e08-47a0-936e-205670d59ede\" (UID: \"3dca5bf3-8e08-47a0-936e-205670d59ede\") "
Nov 02 15:52:51.374393 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:52:51.373710    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3dca5bf3-8e08-47a0-936e-205670d59ede-dshm\" (UniqueName: \"kubernetes.io/empty-dir/3dca5bf3-8e08-47a0-936e-205670d59ede-dshm\") pod \"3dca5bf3-8e08-47a0-936e-205670d59ede\" (UID: \"3dca5bf3-8e08-47a0-936e-205670d59ede\") "
Nov 02 15:52:51.374393 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:52:51.373720    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3dca5bf3-8e08-47a0-936e-205670d59ede-tmp-dir" pod "3dca5bf3-8e08-47a0-936e-205670d59ede" (UID: "3dca5bf3-8e08-47a0-936e-205670d59ede"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:52:51.374393 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:52:51.373763    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3dca5bf3-8e08-47a0-936e-205670d59ede-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3dca5bf3-8e08-47a0-936e-205670d59ede-model-cache\") pod \"3dca5bf3-8e08-47a0-936e-205670d59ede\" (UID: \"3dca5bf3-8e08-47a0-936e-205670d59ede\") "
Nov 02 15:52:51.374393 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:52:51.373788    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3dca5bf3-8e08-47a0-936e-205670d59ede-home" pod "3dca5bf3-8e08-47a0-936e-205670d59ede" (UID: "3dca5bf3-8e08-47a0-936e-205670d59ede"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:52:51.374393 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:52:51.373928    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3dca5bf3-8e08-47a0-936e-205670d59ede-model-cache" pod "3dca5bf3-8e08-47a0-936e-205670d59ede" (UID: "3dca5bf3-8e08-47a0-936e-205670d59ede"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:52:51.374393 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:52:51.374184    5304 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3dca5bf3-8e08-47a0-936e-205670d59ede-model-cache\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:52:51.374393 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:52:51.374200    5304 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3dca5bf3-8e08-47a0-936e-205670d59ede-tmp-dir\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:52:51.374393 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:52:51.374210    5304 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3dca5bf3-8e08-47a0-936e-205670d59ede-home\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:52:51.375931 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:52:51.375897    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3dca5bf3-8e08-47a0-936e-205670d59ede-kube-api-access-zxs7c" pod "3dca5bf3-8e08-47a0-936e-205670d59ede" (UID: "3dca5bf3-8e08-47a0-936e-205670d59ede"). InnerVolumeSpecName "kube-api-access-zxs7c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:52:51.376283 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:52:51.376260    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3dca5bf3-8e08-47a0-936e-205670d59ede-dshm" pod "3dca5bf3-8e08-47a0-936e-205670d59ede" (UID: "3dca5bf3-8e08-47a0-936e-205670d59ede"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:52:51.376283 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:52:51.376267    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3dca5bf3-8e08-47a0-936e-205670d59ede-tls-certs" pod "3dca5bf3-8e08-47a0-936e-205670d59ede" (UID: "3dca5bf3-8e08-47a0-936e-205670d59ede"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:52:51.475213 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:52:51.475092    5304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zxs7c\" (UniqueName: \"kubernetes.io/projected/3dca5bf3-8e08-47a0-936e-205670d59ede-kube-api-access-zxs7c\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:52:51.475213 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:52:51.475122    5304 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3dca5bf3-8e08-47a0-936e-205670d59ede-tls-certs\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:52:51.475213 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:52:51.475149    5304 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3dca5bf3-8e08-47a0-936e-205670d59ede-dshm\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:52:51.651105 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:52:51.651066    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-6af50f/distinct-w0-6af50f-kserve-c6d994647-hbj9z"]
Nov 02 15:52:51.662881 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:52:51.662846    5304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-6af50f/distinct-w0-6af50f-kserve-c6d994647-hbj9z"]
Nov 02 15:52:52.118766 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:52:52.118724    5304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3dca5bf3-8e08-47a0-936e-205670d59ede" path="/var/lib/kubelet/pods/3dca5bf3-8e08-47a0-936e-205670d59ede/volumes"
Nov 02 15:52:54.108627 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:52:54.108589    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-6f2e0c-6568cc97f7-c7h8r"]
Nov 02 15:52:54.848872 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:52:54.848830    5304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k6fjk" containerName="extractor"
Nov 02 15:52:54.853568 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:52:54.853543    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-e40940-openshift-default-6fb44b465f-75t4b"
Nov 02 15:52:54.913024 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:52:54.912990    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7ab8a29-4410-4fde-8975-49c13b3ff1fb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7ab8a29-4410-4fde-8975-49c13b3ff1fb-workload-socket\") pod \"d7ab8a29-4410-4fde-8975-49c13b3ff1fb\" (UID: \"d7ab8a29-4410-4fde-8975-49c13b3ff1fb\") "
Nov 02 15:52:54.913248 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:52:54.913053    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d7ab8a29-4410-4fde-8975-49c13b3ff1fb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7ab8a29-4410-4fde-8975-49c13b3ff1fb-istio-podinfo\") pod \"d7ab8a29-4410-4fde-8975-49c13b3ff1fb\" (UID: \"d7ab8a29-4410-4fde-8975-49c13b3ff1fb\") "
Nov 02 15:52:54.913248 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:52:54.913082    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7ab8a29-4410-4fde-8975-49c13b3ff1fb-istio-token\" (UniqueName: \"kubernetes.io/projected/d7ab8a29-4410-4fde-8975-49c13b3ff1fb-istio-token\") pod \"d7ab8a29-4410-4fde-8975-49c13b3ff1fb\" (UID: \"d7ab8a29-4410-4fde-8975-49c13b3ff1fb\") "
Nov 02 15:52:54.913248 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:52:54.913106    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7ab8a29-4410-4fde-8975-49c13b3ff1fb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7ab8a29-4410-4fde-8975-49c13b3ff1fb-workload-certs\") pod \"d7ab8a29-4410-4fde-8975-49c13b3ff1fb\" (UID: \"d7ab8a29-4410-4fde-8975-49c13b3ff1fb\") "
Nov 02 15:52:54.913248 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:52:54.913244    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d7ab8a29-4410-4fde-8975-49c13b3ff1fb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7ab8a29-4410-4fde-8975-49c13b3ff1fb-istiod-ca-cert\") pod \"d7ab8a29-4410-4fde-8975-49c13b3ff1fb\" (UID: \"d7ab8a29-4410-4fde-8975-49c13b3ff1fb\") "
Nov 02 15:52:54.913506 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:52:54.913277    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7ab8a29-4410-4fde-8975-49c13b3ff1fb-kube-api-access-qgb8c\" (UniqueName: \"kubernetes.io/projected/d7ab8a29-4410-4fde-8975-49c13b3ff1fb-kube-api-access-qgb8c\") pod \"d7ab8a29-4410-4fde-8975-49c13b3ff1fb\" (UID: \"d7ab8a29-4410-4fde-8975-49c13b3ff1fb\") "
Nov 02 15:52:54.913506 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:52:54.913332    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7ab8a29-4410-4fde-8975-49c13b3ff1fb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7ab8a29-4410-4fde-8975-49c13b3ff1fb-credential-socket\") pod \"d7ab8a29-4410-4fde-8975-49c13b3ff1fb\" (UID: \"d7ab8a29-4410-4fde-8975-49c13b3ff1fb\") "
Nov 02 15:52:54.913506 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:52:54.913372    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7ab8a29-4410-4fde-8975-49c13b3ff1fb-workload-socket" pod "d7ab8a29-4410-4fde-8975-49c13b3ff1fb" (UID: "d7ab8a29-4410-4fde-8975-49c13b3ff1fb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:52:54.913506 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:52:54.913389    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7ab8a29-4410-4fde-8975-49c13b3ff1fb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7ab8a29-4410-4fde-8975-49c13b3ff1fb-istio-envoy\") pod \"d7ab8a29-4410-4fde-8975-49c13b3ff1fb\" (UID: \"d7ab8a29-4410-4fde-8975-49c13b3ff1fb\") "
Nov 02 15:52:54.913506 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:52:54.913439    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7ab8a29-4410-4fde-8975-49c13b3ff1fb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7ab8a29-4410-4fde-8975-49c13b3ff1fb-istio-data\") pod \"d7ab8a29-4410-4fde-8975-49c13b3ff1fb\" (UID: \"d7ab8a29-4410-4fde-8975-49c13b3ff1fb\") "
Nov 02 15:52:54.913906 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:52:54.913620    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7ab8a29-4410-4fde-8975-49c13b3ff1fb-istiod-ca-cert" pod "d7ab8a29-4410-4fde-8975-49c13b3ff1fb" (UID: "d7ab8a29-4410-4fde-8975-49c13b3ff1fb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:52:54.913906 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:52:54.913841    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7ab8a29-4410-4fde-8975-49c13b3ff1fb-istio-data" pod "d7ab8a29-4410-4fde-8975-49c13b3ff1fb" (UID: "d7ab8a29-4410-4fde-8975-49c13b3ff1fb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:52:54.914410 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:52:54.914046    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7ab8a29-4410-4fde-8975-49c13b3ff1fb-credential-socket" pod "d7ab8a29-4410-4fde-8975-49c13b3ff1fb" (UID: "d7ab8a29-4410-4fde-8975-49c13b3ff1fb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:52:54.914410 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:52:54.914070    5304 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7ab8a29-4410-4fde-8975-49c13b3ff1fb-workload-socket\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:52:54.914410 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:52:54.914092    5304 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7ab8a29-4410-4fde-8975-49c13b3ff1fb-istiod-ca-cert\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:52:54.914410 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:52:54.914107    5304 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7ab8a29-4410-4fde-8975-49c13b3ff1fb-istio-data\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:52:54.914410 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:52:54.914266    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7ab8a29-4410-4fde-8975-49c13b3ff1fb-workload-certs" pod "d7ab8a29-4410-4fde-8975-49c13b3ff1fb" (UID: "d7ab8a29-4410-4fde-8975-49c13b3ff1fb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:52:54.915736 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:52:54.915706    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d7ab8a29-4410-4fde-8975-49c13b3ff1fb-istio-podinfo" pod "d7ab8a29-4410-4fde-8975-49c13b3ff1fb" (UID: "d7ab8a29-4410-4fde-8975-49c13b3ff1fb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 15:52:54.916011 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:52:54.915987    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7ab8a29-4410-4fde-8975-49c13b3ff1fb-istio-token" pod "d7ab8a29-4410-4fde-8975-49c13b3ff1fb" (UID: "d7ab8a29-4410-4fde-8975-49c13b3ff1fb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:52:54.916261 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:52:54.916239    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7ab8a29-4410-4fde-8975-49c13b3ff1fb-kube-api-access-qgb8c" pod "d7ab8a29-4410-4fde-8975-49c13b3ff1fb" (UID: "d7ab8a29-4410-4fde-8975-49c13b3ff1fb"). InnerVolumeSpecName "kube-api-access-qgb8c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:52:54.916347 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:52:54.916280    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7ab8a29-4410-4fde-8975-49c13b3ff1fb-istio-envoy" pod "d7ab8a29-4410-4fde-8975-49c13b3ff1fb" (UID: "d7ab8a29-4410-4fde-8975-49c13b3ff1fb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:52:55.019724 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:52:55.017561    5304 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7ab8a29-4410-4fde-8975-49c13b3ff1fb-istio-podinfo\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:52:55.019724 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:52:55.017601    5304 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7ab8a29-4410-4fde-8975-49c13b3ff1fb-istio-token\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:52:55.019724 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:52:55.017619    5304 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7ab8a29-4410-4fde-8975-49c13b3ff1fb-workload-certs\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:52:55.019724 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:52:55.017633    5304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qgb8c\" (UniqueName: \"kubernetes.io/projected/d7ab8a29-4410-4fde-8975-49c13b3ff1fb-kube-api-access-qgb8c\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:52:55.019724 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:52:55.017649    5304 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7ab8a29-4410-4fde-8975-49c13b3ff1fb-credential-socket\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:52:55.019724 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:52:55.017662    5304 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7ab8a29-4410-4fde-8975-49c13b3ff1fb-istio-envoy\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:52:55.324789 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:52:55.324750    5304 generic.go:356] "Generic (PLEG): container finished" podID="d7ab8a29-4410-4fde-8975-49c13b3ff1fb" containerID="334e92c952bf360c8eca5a791970798152aaf9ca276333fe8a507a4a7e643043" exitCode=0
Nov 02 15:52:55.325257 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:52:55.324797    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-e40940-openshift-default-6fb44b465f-75t4b"
Nov 02 15:52:55.325257 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:52:55.324834    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-e40940-openshift-default-6fb44b465f-75t4b" event={"ID":"d7ab8a29-4410-4fde-8975-49c13b3ff1fb","Type":"ContainerDied","Data":"334e92c952bf360c8eca5a791970798152aaf9ca276333fe8a507a4a7e643043"}
Nov 02 15:52:55.325257 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:52:55.324896    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-e40940-openshift-default-6fb44b465f-75t4b" event={"ID":"d7ab8a29-4410-4fde-8975-49c13b3ff1fb","Type":"ContainerDied","Data":"589f75da3f367475110a0d32be806ffd801606686d5381afab1592bdfe09c28d"}
Nov 02 15:52:55.325257 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:52:55.324923    5304 scope.go:122] "RemoveContainer" containerID="334e92c952bf360c8eca5a791970798152aaf9ca276333fe8a507a4a7e643043"
Nov 02 15:52:55.335518 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:52:55.335500    5304 scope.go:122] "RemoveContainer" containerID="c4fc082d0ebbceccdf88601b58447c83a6bc5dd2fe8ba878d8d75c78d9d8729f"
Nov 02 15:52:55.344224 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:52:55.344203    5304 scope.go:122] "RemoveContainer" containerID="334e92c952bf360c8eca5a791970798152aaf9ca276333fe8a507a4a7e643043"
Nov 02 15:52:55.344536 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:52:55.344502    5304 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"334e92c952bf360c8eca5a791970798152aaf9ca276333fe8a507a4a7e643043\": container with ID starting with 334e92c952bf360c8eca5a791970798152aaf9ca276333fe8a507a4a7e643043 not found: ID does not exist" containerID="334e92c952bf360c8eca5a791970798152aaf9ca276333fe8a507a4a7e643043"
Nov 02 15:52:55.344675 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:52:55.344549    5304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"334e92c952bf360c8eca5a791970798152aaf9ca276333fe8a507a4a7e643043"} err="failed to get container status \"334e92c952bf360c8eca5a791970798152aaf9ca276333fe8a507a4a7e643043\": rpc error: code = NotFound desc = could not find container \"334e92c952bf360c8eca5a791970798152aaf9ca276333fe8a507a4a7e643043\": container with ID starting with 334e92c952bf360c8eca5a791970798152aaf9ca276333fe8a507a4a7e643043 not found: ID does not exist"
Nov 02 15:52:55.344675 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:52:55.344572    5304 scope.go:122] "RemoveContainer" containerID="c4fc082d0ebbceccdf88601b58447c83a6bc5dd2fe8ba878d8d75c78d9d8729f"
Nov 02 15:52:55.344850 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:52:55.344830    5304 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c4fc082d0ebbceccdf88601b58447c83a6bc5dd2fe8ba878d8d75c78d9d8729f\": container with ID starting with c4fc082d0ebbceccdf88601b58447c83a6bc5dd2fe8ba878d8d75c78d9d8729f not found: ID does not exist" containerID="c4fc082d0ebbceccdf88601b58447c83a6bc5dd2fe8ba878d8d75c78d9d8729f"
Nov 02 15:52:55.344903 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:52:55.344855    5304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4fc082d0ebbceccdf88601b58447c83a6bc5dd2fe8ba878d8d75c78d9d8729f"} err="failed to get container status \"c4fc082d0ebbceccdf88601b58447c83a6bc5dd2fe8ba878d8d75c78d9d8729f\": rpc error: code = NotFound desc = could not find container \"c4fc082d0ebbceccdf88601b58447c83a6bc5dd2fe8ba878d8d75c78d9d8729f\": container with ID starting with c4fc082d0ebbceccdf88601b58447c83a6bc5dd2fe8ba878d8d75c78d9d8729f not found: ID does not exist"
Nov 02 15:52:55.379928 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:52:55.379899    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-e40940-openshift-default-6fb44b465f-75t4b"]
Nov 02 15:52:55.391816 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:52:55.391782    5304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-e40940-openshift-default-6fb44b465f-75t4b"]
Nov 02 15:52:56.117617 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:52:56.117586    5304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d7ab8a29-4410-4fde-8975-49c13b3ff1fb" path="/var/lib/kubelet/pods/d7ab8a29-4410-4fde-8975-49c13b3ff1fb/volumes"
Nov 02 15:52:59.109473 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:52:59.109411    5304 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-6f2e0c-6568cc97f7-c7h8r" podUID="9c90c9e1-7216-4ffb-b513-f5ca7fec318a" containerName="payload-processing" containerID="cri-o://31267b61932edcb7a3e585accc0405f9f7597f0b04201107e586506712fcd64d" gracePeriod=25
Nov 02 15:53:01.410395 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:01.410355    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-6f2e0c-6568cc97f7-c7h8r" podUID="9c90c9e1-7216-4ffb-b513-f5ca7fec318a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 15:53:03.561397 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:03.561357    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-f0039c-7cdbfcf875-tptc9"]
Nov 02 15:53:03.561909 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:03.561606    5304 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f0039c-7cdbfcf875-tptc9" podUID="39e13479-f61d-4215-bbb3-03aa8b678618" containerName="maas-api" containerID="cri-o://15725e87a6a3cbe9577f5debe711bf87ca634270346ca83b12faee6047b797f5" gracePeriod=30
Nov 02 15:53:03.870402 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:03.870369    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f0039c-ff4996964-9pzcn"]
Nov 02 15:53:03.870653 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:03.870629    5304 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-f0039c-ff4996964-9pzcn" podUID="43c5260b-b089-4c06-ac76-e6e170cac754" containerName="payload-processing" containerID="cri-o://0b61a4b3ffd0614509c9c252c532a59aea8ad299ff31a9d866ae255a83d9d02d" gracePeriod=30
Nov 02 15:53:03.874162 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:03.874091    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f0039c-5597f65b87-tqz8b"]
Nov 02 15:53:03.874343 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:03.874316    5304 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f0039c-5597f65b87-tqz8b" podUID="d35d4668-8376-487a-9c91-8b6801eb9cc6" containerName="payload-pre-processing" containerID="cri-o://cf002bb8f28bd1c2b000f567c7eafb6aa6b554b5231e6d516ba5b096c60275b4" gracePeriod=30
Nov 02 15:53:04.064403 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:04.064366    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f0039c-7cdbfcf875-tptc9"
Nov 02 15:53:04.128211 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:04.128108    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/39e13479-f61d-4215-bbb3-03aa8b678618-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/39e13479-f61d-4215-bbb3-03aa8b678618-maas-api-tls\") pod \"39e13479-f61d-4215-bbb3-03aa8b678618\" (UID: \"39e13479-f61d-4215-bbb3-03aa8b678618\") "
Nov 02 15:53:04.128390 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:04.128255    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/39e13479-f61d-4215-bbb3-03aa8b678618-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/39e13479-f61d-4215-bbb3-03aa8b678618-openshift-service-ca\") pod \"39e13479-f61d-4215-bbb3-03aa8b678618\" (UID: \"39e13479-f61d-4215-bbb3-03aa8b678618\") "
Nov 02 15:53:04.128390 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:04.128306    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/39e13479-f61d-4215-bbb3-03aa8b678618-metrics-certs\" (UniqueName: \"kubernetes.io/secret/39e13479-f61d-4215-bbb3-03aa8b678618-metrics-certs\") pod \"39e13479-f61d-4215-bbb3-03aa8b678618\" (UID: \"39e13479-f61d-4215-bbb3-03aa8b678618\") "
Nov 02 15:53:04.128526 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:04.128405    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/39e13479-f61d-4215-bbb3-03aa8b678618-kube-api-access-jnq7h\" (UniqueName: \"kubernetes.io/projected/39e13479-f61d-4215-bbb3-03aa8b678618-kube-api-access-jnq7h\") pod \"39e13479-f61d-4215-bbb3-03aa8b678618\" (UID: \"39e13479-f61d-4215-bbb3-03aa8b678618\") "
Nov 02 15:53:04.128975 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:04.128927    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39e13479-f61d-4215-bbb3-03aa8b678618-openshift-service-ca" pod "39e13479-f61d-4215-bbb3-03aa8b678618" (UID: "39e13479-f61d-4215-bbb3-03aa8b678618"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:53:04.134462 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:04.134431    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/39e13479-f61d-4215-bbb3-03aa8b678618-metrics-certs" pod "39e13479-f61d-4215-bbb3-03aa8b678618" (UID: "39e13479-f61d-4215-bbb3-03aa8b678618"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:53:04.138314 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:04.138282    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/39e13479-f61d-4215-bbb3-03aa8b678618-maas-api-tls" pod "39e13479-f61d-4215-bbb3-03aa8b678618" (UID: "39e13479-f61d-4215-bbb3-03aa8b678618"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:53:04.138570 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:04.138545    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39e13479-f61d-4215-bbb3-03aa8b678618-kube-api-access-jnq7h" pod "39e13479-f61d-4215-bbb3-03aa8b678618" (UID: "39e13479-f61d-4215-bbb3-03aa8b678618"). InnerVolumeSpecName "kube-api-access-jnq7h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:53:04.168920 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:04.168891    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f0039c-5597f65b87-tqz8b"
Nov 02 15:53:04.218140 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:04.218103    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-f0039c-ff4996964-9pzcn"
Nov 02 15:53:04.230096 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:04.230053    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d35d4668-8376-487a-9c91-8b6801eb9cc6-kube-api-access-8pkxw\" (UniqueName: \"kubernetes.io/projected/d35d4668-8376-487a-9c91-8b6801eb9cc6-kube-api-access-8pkxw\") pod \"d35d4668-8376-487a-9c91-8b6801eb9cc6\" (UID: \"d35d4668-8376-487a-9c91-8b6801eb9cc6\") "
Nov 02 15:53:04.230306 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:04.230242    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d35d4668-8376-487a-9c91-8b6801eb9cc6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d35d4668-8376-487a-9c91-8b6801eb9cc6-plugins-config-volume\") pod \"d35d4668-8376-487a-9c91-8b6801eb9cc6\" (UID: \"d35d4668-8376-487a-9c91-8b6801eb9cc6\") "
Nov 02 15:53:04.234248 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:04.230680    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d35d4668-8376-487a-9c91-8b6801eb9cc6-plugins-config-volume" pod "d35d4668-8376-487a-9c91-8b6801eb9cc6" (UID: "d35d4668-8376-487a-9c91-8b6801eb9cc6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:53:04.234248 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:04.230955    5304 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/39e13479-f61d-4215-bbb3-03aa8b678618-maas-api-tls\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:53:04.234248 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:04.231232    5304 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/39e13479-f61d-4215-bbb3-03aa8b678618-openshift-service-ca\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:53:04.234248 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:04.231268    5304 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/39e13479-f61d-4215-bbb3-03aa8b678618-metrics-certs\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:53:04.234248 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:04.231293    5304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jnq7h\" (UniqueName: \"kubernetes.io/projected/39e13479-f61d-4215-bbb3-03aa8b678618-kube-api-access-jnq7h\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:53:04.234248 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:04.232464    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d35d4668-8376-487a-9c91-8b6801eb9cc6-kube-api-access-8pkxw" pod "d35d4668-8376-487a-9c91-8b6801eb9cc6" (UID: "d35d4668-8376-487a-9c91-8b6801eb9cc6"). InnerVolumeSpecName "kube-api-access-8pkxw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:53:04.331864 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:04.331823    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43c5260b-b089-4c06-ac76-e6e170cac754-kube-api-access-p4dl8\" (UniqueName: \"kubernetes.io/projected/43c5260b-b089-4c06-ac76-e6e170cac754-kube-api-access-p4dl8\") pod \"43c5260b-b089-4c06-ac76-e6e170cac754\" (UID: \"43c5260b-b089-4c06-ac76-e6e170cac754\") "
Nov 02 15:53:04.332108 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:04.331954    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/43c5260b-b089-4c06-ac76-e6e170cac754-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43c5260b-b089-4c06-ac76-e6e170cac754-plugins-config-volume\") pod \"43c5260b-b089-4c06-ac76-e6e170cac754\" (UID: \"43c5260b-b089-4c06-ac76-e6e170cac754\") "
Nov 02 15:53:04.334329 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:04.332285    5304 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d35d4668-8376-487a-9c91-8b6801eb9cc6-plugins-config-volume\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:53:04.334329 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:04.332308    5304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8pkxw\" (UniqueName: \"kubernetes.io/projected/d35d4668-8376-487a-9c91-8b6801eb9cc6-kube-api-access-8pkxw\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:53:04.334329 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:04.332672    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43c5260b-b089-4c06-ac76-e6e170cac754-plugins-config-volume" pod "43c5260b-b089-4c06-ac76-e6e170cac754" (UID: "43c5260b-b089-4c06-ac76-e6e170cac754"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:53:04.337456 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:04.335386    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43c5260b-b089-4c06-ac76-e6e170cac754-kube-api-access-p4dl8" pod "43c5260b-b089-4c06-ac76-e6e170cac754" (UID: "43c5260b-b089-4c06-ac76-e6e170cac754"). InnerVolumeSpecName "kube-api-access-p4dl8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:53:04.394215 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:04.389964    5304 generic.go:356] "Generic (PLEG): container finished" podID="39e13479-f61d-4215-bbb3-03aa8b678618" containerID="15725e87a6a3cbe9577f5debe711bf87ca634270346ca83b12faee6047b797f5" exitCode=0
Nov 02 15:53:04.394215 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:04.390098    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f0039c-7cdbfcf875-tptc9" event={"ID":"39e13479-f61d-4215-bbb3-03aa8b678618","Type":"ContainerDied","Data":"15725e87a6a3cbe9577f5debe711bf87ca634270346ca83b12faee6047b797f5"}
Nov 02 15:53:04.394215 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:04.390188    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f0039c-7cdbfcf875-tptc9" event={"ID":"39e13479-f61d-4215-bbb3-03aa8b678618","Type":"ContainerDied","Data":"847019b1ed96dae9cec53de4c21a908d10f9cd5d33cddf8c0095e40a13cb0727"}
Nov 02 15:53:04.394215 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:04.390220    5304 scope.go:122] "RemoveContainer" containerID="15725e87a6a3cbe9577f5debe711bf87ca634270346ca83b12faee6047b797f5"
Nov 02 15:53:04.394215 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:04.390386    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f0039c-7cdbfcf875-tptc9"
Nov 02 15:53:04.398054 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:04.398024    5304 generic.go:356] "Generic (PLEG): container finished" podID="43c5260b-b089-4c06-ac76-e6e170cac754" containerID="0b61a4b3ffd0614509c9c252c532a59aea8ad299ff31a9d866ae255a83d9d02d" exitCode=0
Nov 02 15:53:04.398238 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:04.398157    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-f0039c-ff4996964-9pzcn"
Nov 02 15:53:04.398356 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:04.398329    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-f0039c-ff4996964-9pzcn" event={"ID":"43c5260b-b089-4c06-ac76-e6e170cac754","Type":"ContainerDied","Data":"0b61a4b3ffd0614509c9c252c532a59aea8ad299ff31a9d866ae255a83d9d02d"}
Nov 02 15:53:04.398464 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:04.398393    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-f0039c-ff4996964-9pzcn" event={"ID":"43c5260b-b089-4c06-ac76-e6e170cac754","Type":"ContainerDied","Data":"61f396b65ff959727476bfd4399eb5f80a014ccf1ea1dc4b9ab593ea489d50a5"}
Nov 02 15:53:04.407691 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:04.403031    5304 generic.go:356] "Generic (PLEG): container finished" podID="d35d4668-8376-487a-9c91-8b6801eb9cc6" containerID="cf002bb8f28bd1c2b000f567c7eafb6aa6b554b5231e6d516ba5b096c60275b4" exitCode=0
Nov 02 15:53:04.407691 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:04.403081    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f0039c-5597f65b87-tqz8b"
Nov 02 15:53:04.407691 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:04.403114    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f0039c-5597f65b87-tqz8b" event={"ID":"d35d4668-8376-487a-9c91-8b6801eb9cc6","Type":"ContainerDied","Data":"cf002bb8f28bd1c2b000f567c7eafb6aa6b554b5231e6d516ba5b096c60275b4"}
Nov 02 15:53:04.407691 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:04.403175    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f0039c-5597f65b87-tqz8b" event={"ID":"d35d4668-8376-487a-9c91-8b6801eb9cc6","Type":"ContainerDied","Data":"2118d95c6fc5095d52ca8a55cb216bdc9b4525c1c6f048e647b089fcbaf27336"}
Nov 02 15:53:04.409925 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:04.409907    5304 scope.go:122] "RemoveContainer" containerID="15725e87a6a3cbe9577f5debe711bf87ca634270346ca83b12faee6047b797f5"
Nov 02 15:53:04.410281 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:53:04.410262    5304 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"15725e87a6a3cbe9577f5debe711bf87ca634270346ca83b12faee6047b797f5\": container with ID starting with 15725e87a6a3cbe9577f5debe711bf87ca634270346ca83b12faee6047b797f5 not found: ID does not exist" containerID="15725e87a6a3cbe9577f5debe711bf87ca634270346ca83b12faee6047b797f5"
Nov 02 15:53:04.410373 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:04.410293    5304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"15725e87a6a3cbe9577f5debe711bf87ca634270346ca83b12faee6047b797f5"} err="failed to get container status \"15725e87a6a3cbe9577f5debe711bf87ca634270346ca83b12faee6047b797f5\": rpc error: code = NotFound desc = could not find container \"15725e87a6a3cbe9577f5debe711bf87ca634270346ca83b12faee6047b797f5\": container with ID starting with 15725e87a6a3cbe9577f5debe711bf87ca634270346ca83b12faee6047b797f5 not found: ID does not exist"
Nov 02 15:53:04.410373 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:04.410316    5304 scope.go:122] "RemoveContainer" containerID="0b61a4b3ffd0614509c9c252c532a59aea8ad299ff31a9d866ae255a83d9d02d"
Nov 02 15:53:04.421504 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:04.421462    5304 scope.go:122] "RemoveContainer" containerID="0b61a4b3ffd0614509c9c252c532a59aea8ad299ff31a9d866ae255a83d9d02d"
Nov 02 15:53:04.421890 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:53:04.421859    5304 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b61a4b3ffd0614509c9c252c532a59aea8ad299ff31a9d866ae255a83d9d02d\": container with ID starting with 0b61a4b3ffd0614509c9c252c532a59aea8ad299ff31a9d866ae255a83d9d02d not found: ID does not exist" containerID="0b61a4b3ffd0614509c9c252c532a59aea8ad299ff31a9d866ae255a83d9d02d"
Nov 02 15:53:04.421998 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:04.421897    5304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b61a4b3ffd0614509c9c252c532a59aea8ad299ff31a9d866ae255a83d9d02d"} err="failed to get container status \"0b61a4b3ffd0614509c9c252c532a59aea8ad299ff31a9d866ae255a83d9d02d\": rpc error: code = NotFound desc = could not find container \"0b61a4b3ffd0614509c9c252c532a59aea8ad299ff31a9d866ae255a83d9d02d\": container with ID starting with 0b61a4b3ffd0614509c9c252c532a59aea8ad299ff31a9d866ae255a83d9d02d not found: ID does not exist"
Nov 02 15:53:04.421998 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:04.421920    5304 scope.go:122] "RemoveContainer" containerID="cf002bb8f28bd1c2b000f567c7eafb6aa6b554b5231e6d516ba5b096c60275b4"
Nov 02 15:53:04.431414 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:04.431391    5304 scope.go:122] "RemoveContainer" containerID="cf002bb8f28bd1c2b000f567c7eafb6aa6b554b5231e6d516ba5b096c60275b4"
Nov 02 15:53:04.431724 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:53:04.431705    5304 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf002bb8f28bd1c2b000f567c7eafb6aa6b554b5231e6d516ba5b096c60275b4\": container with ID starting with cf002bb8f28bd1c2b000f567c7eafb6aa6b554b5231e6d516ba5b096c60275b4 not found: ID does not exist" containerID="cf002bb8f28bd1c2b000f567c7eafb6aa6b554b5231e6d516ba5b096c60275b4"
Nov 02 15:53:04.431774 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:04.431732    5304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf002bb8f28bd1c2b000f567c7eafb6aa6b554b5231e6d516ba5b096c60275b4"} err="failed to get container status \"cf002bb8f28bd1c2b000f567c7eafb6aa6b554b5231e6d516ba5b096c60275b4\": rpc error: code = NotFound desc = could not find container \"cf002bb8f28bd1c2b000f567c7eafb6aa6b554b5231e6d516ba5b096c60275b4\": container with ID starting with cf002bb8f28bd1c2b000f567c7eafb6aa6b554b5231e6d516ba5b096c60275b4 not found: ID does not exist"
Nov 02 15:53:04.433488 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:04.433462    5304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p4dl8\" (UniqueName: \"kubernetes.io/projected/43c5260b-b089-4c06-ac76-e6e170cac754-kube-api-access-p4dl8\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:53:04.433488 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:04.433485    5304 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43c5260b-b089-4c06-ac76-e6e170cac754-plugins-config-volume\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:53:04.456546 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:04.456495    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f0039c-5597f65b87-tqz8b"]
Nov 02 15:53:04.460163 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:04.460118    5304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f0039c-5597f65b87-tqz8b"]
Nov 02 15:53:04.496217 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:04.496181    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f0039c-ff4996964-9pzcn"]
Nov 02 15:53:04.498823 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:04.498793    5304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f0039c-ff4996964-9pzcn"]
Nov 02 15:53:04.546448 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:04.545856    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-f0039c-7cdbfcf875-tptc9"]
Nov 02 15:53:04.551156 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:04.547276    5304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-f0039c-7cdbfcf875-tptc9"]
Nov 02 15:53:06.117304 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:06.117270    5304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="39e13479-f61d-4215-bbb3-03aa8b678618" path="/var/lib/kubelet/pods/39e13479-f61d-4215-bbb3-03aa8b678618/volumes"
Nov 02 15:53:06.117702 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:06.117691    5304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="43c5260b-b089-4c06-ac76-e6e170cac754" path="/var/lib/kubelet/pods/43c5260b-b089-4c06-ac76-e6e170cac754/volumes"
Nov 02 15:53:06.118036 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:06.118023    5304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d35d4668-8376-487a-9c91-8b6801eb9cc6" path="/var/lib/kubelet/pods/d35d4668-8376-487a-9c91-8b6801eb9cc6/volumes"
Nov 02 15:53:06.367018 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:06.366981    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-6af50f-865547d8c5-dvg56"]
Nov 02 15:53:06.367362 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:06.367293    5304 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-6af50f-865547d8c5-dvg56" podUID="ab7216bc-cf46-4280-acd1-3a15d1ead1c0" containerName="maas-api" containerID="cri-o://f5c343e6eb800855b293ef6faec0fcf5f4c6b3b43409fb692d685bae2fffa4e1" gracePeriod=30
Nov 02 15:53:06.680355 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:06.680330    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-6af50f-865547d8c5-dvg56"
Nov 02 15:53:06.775150 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:06.773270    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab7216bc-cf46-4280-acd1-3a15d1ead1c0-kube-api-access-5t56c\" (UniqueName: \"kubernetes.io/projected/ab7216bc-cf46-4280-acd1-3a15d1ead1c0-kube-api-access-5t56c\") pod \"ab7216bc-cf46-4280-acd1-3a15d1ead1c0\" (UID: \"ab7216bc-cf46-4280-acd1-3a15d1ead1c0\") "
Nov 02 15:53:06.775150 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:06.773827    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ab7216bc-cf46-4280-acd1-3a15d1ead1c0-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ab7216bc-cf46-4280-acd1-3a15d1ead1c0-metrics-certs\") pod \"ab7216bc-cf46-4280-acd1-3a15d1ead1c0\" (UID: \"ab7216bc-cf46-4280-acd1-3a15d1ead1c0\") "
Nov 02 15:53:06.775150 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:06.773982    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ab7216bc-cf46-4280-acd1-3a15d1ead1c0-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab7216bc-cf46-4280-acd1-3a15d1ead1c0-openshift-service-ca\") pod \"ab7216bc-cf46-4280-acd1-3a15d1ead1c0\" (UID: \"ab7216bc-cf46-4280-acd1-3a15d1ead1c0\") "
Nov 02 15:53:06.775150 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:06.774063    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ab7216bc-cf46-4280-acd1-3a15d1ead1c0-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ab7216bc-cf46-4280-acd1-3a15d1ead1c0-maas-api-tls\") pod \"ab7216bc-cf46-4280-acd1-3a15d1ead1c0\" (UID: \"ab7216bc-cf46-4280-acd1-3a15d1ead1c0\") "
Nov 02 15:53:06.775734 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:06.775698    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab7216bc-cf46-4280-acd1-3a15d1ead1c0-openshift-service-ca" pod "ab7216bc-cf46-4280-acd1-3a15d1ead1c0" (UID: "ab7216bc-cf46-4280-acd1-3a15d1ead1c0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:53:06.793925 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:06.793006    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ab7216bc-cf46-4280-acd1-3a15d1ead1c0-maas-api-tls" pod "ab7216bc-cf46-4280-acd1-3a15d1ead1c0" (UID: "ab7216bc-cf46-4280-acd1-3a15d1ead1c0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:53:06.795495 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:06.795427    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab7216bc-cf46-4280-acd1-3a15d1ead1c0-kube-api-access-5t56c" pod "ab7216bc-cf46-4280-acd1-3a15d1ead1c0" (UID: "ab7216bc-cf46-4280-acd1-3a15d1ead1c0"). InnerVolumeSpecName "kube-api-access-5t56c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:53:06.796154 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:06.796105    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ab7216bc-cf46-4280-acd1-3a15d1ead1c0-metrics-certs" pod "ab7216bc-cf46-4280-acd1-3a15d1ead1c0" (UID: "ab7216bc-cf46-4280-acd1-3a15d1ead1c0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:53:06.876114 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:06.876078    5304 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab7216bc-cf46-4280-acd1-3a15d1ead1c0-openshift-service-ca\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:53:06.876114 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:06.876111    5304 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ab7216bc-cf46-4280-acd1-3a15d1ead1c0-maas-api-tls\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:53:06.876346 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:06.876143    5304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5t56c\" (UniqueName: \"kubernetes.io/projected/ab7216bc-cf46-4280-acd1-3a15d1ead1c0-kube-api-access-5t56c\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:53:06.876346 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:06.876157    5304 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ab7216bc-cf46-4280-acd1-3a15d1ead1c0-metrics-certs\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:53:07.424255 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:07.424209    5304 generic.go:356] "Generic (PLEG): container finished" podID="ab7216bc-cf46-4280-acd1-3a15d1ead1c0" containerID="f5c343e6eb800855b293ef6faec0fcf5f4c6b3b43409fb692d685bae2fffa4e1" exitCode=0
Nov 02 15:53:07.424740 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:07.424260    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-6af50f-865547d8c5-dvg56"
Nov 02 15:53:07.424740 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:07.424300    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-6af50f-865547d8c5-dvg56" event={"ID":"ab7216bc-cf46-4280-acd1-3a15d1ead1c0","Type":"ContainerDied","Data":"f5c343e6eb800855b293ef6faec0fcf5f4c6b3b43409fb692d685bae2fffa4e1"}
Nov 02 15:53:07.424740 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:07.424356    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-6af50f-865547d8c5-dvg56" event={"ID":"ab7216bc-cf46-4280-acd1-3a15d1ead1c0","Type":"ContainerDied","Data":"b7d1e39ea100be76085e97ac54b814764583c82231ab512544018c306af2d345"}
Nov 02 15:53:07.424740 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:07.424386    5304 scope.go:122] "RemoveContainer" containerID="f5c343e6eb800855b293ef6faec0fcf5f4c6b3b43409fb692d685bae2fffa4e1"
Nov 02 15:53:07.434731 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:07.434711    5304 scope.go:122] "RemoveContainer" containerID="f5c343e6eb800855b293ef6faec0fcf5f4c6b3b43409fb692d685bae2fffa4e1"
Nov 02 15:53:07.435063 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:53:07.435032    5304 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f5c343e6eb800855b293ef6faec0fcf5f4c6b3b43409fb692d685bae2fffa4e1\": container with ID starting with f5c343e6eb800855b293ef6faec0fcf5f4c6b3b43409fb692d685bae2fffa4e1 not found: ID does not exist" containerID="f5c343e6eb800855b293ef6faec0fcf5f4c6b3b43409fb692d685bae2fffa4e1"
Nov 02 15:53:07.435182 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:07.435074    5304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5c343e6eb800855b293ef6faec0fcf5f4c6b3b43409fb692d685bae2fffa4e1"} err="failed to get container status \"f5c343e6eb800855b293ef6faec0fcf5f4c6b3b43409fb692d685bae2fffa4e1\": rpc error: code = NotFound desc = could not find container \"f5c343e6eb800855b293ef6faec0fcf5f4c6b3b43409fb692d685bae2fffa4e1\": container with ID starting with f5c343e6eb800855b293ef6faec0fcf5f4c6b3b43409fb692d685bae2fffa4e1 not found: ID does not exist"
Nov 02 15:53:07.477524 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:07.477487    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-6af50f-865547d8c5-dvg56"]
Nov 02 15:53:07.488868 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:07.488840    5304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-6af50f-865547d8c5-dvg56"]
Nov 02 15:53:08.118891 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:08.118857    5304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ab7216bc-cf46-4280-acd1-3a15d1ead1c0" path="/var/lib/kubelet/pods/ab7216bc-cf46-4280-acd1-3a15d1ead1c0/volumes"
Nov 02 15:53:10.691514 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:10.691478    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-6f2e0c-openshift-default-767db5d9b7-6jmzj"]
Nov 02 15:53:10.691989 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:10.691757    5304 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-6f2e0c-openshift-default-767db5d9b7-6jmzj" podUID="9819202d-bd76-4f6f-b077-59ec79c0c50a" containerName="istio-proxy" containerID="cri-o://c48074e374290de95f1882f29f363dfe1128af91498a3b92a31a69287ba520af" gracePeriod=30
Nov 02 15:53:11.409469 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:11.409424    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-6f2e0c-6568cc97f7-c7h8r" podUID="9c90c9e1-7216-4ffb-b513-f5ca7fec318a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 15:53:14.325170 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:14.325117    5304 patch_prober.go:29] interesting pod/e2e-shared-a-w2-6f2e0c-openshift-default-767db5d9b7-6jmzj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 15:53:14.325608 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:14.325192    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w2-6f2e0c-openshift-default-767db5d9b7-6jmzj" podUID="9819202d-bd76-4f6f-b077-59ec79c0c50a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 15:53:15.953787 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:15.953759    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-6f2e0c-openshift-default-767db5d9b7-6jmzj"
Nov 02 15:53:16.079987 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:16.079958    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9819202d-bd76-4f6f-b077-59ec79c0c50a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9819202d-bd76-4f6f-b077-59ec79c0c50a-istio-podinfo\") pod \"9819202d-bd76-4f6f-b077-59ec79c0c50a\" (UID: \"9819202d-bd76-4f6f-b077-59ec79c0c50a\") "
Nov 02 15:53:16.080185 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:16.080070    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9819202d-bd76-4f6f-b077-59ec79c0c50a-kube-api-access-6d28d\" (UniqueName: \"kubernetes.io/projected/9819202d-bd76-4f6f-b077-59ec79c0c50a-kube-api-access-6d28d\") pod \"9819202d-bd76-4f6f-b077-59ec79c0c50a\" (UID: \"9819202d-bd76-4f6f-b077-59ec79c0c50a\") "
Nov 02 15:53:16.080185 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:16.080153    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9819202d-bd76-4f6f-b077-59ec79c0c50a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9819202d-bd76-4f6f-b077-59ec79c0c50a-istiod-ca-cert\") pod \"9819202d-bd76-4f6f-b077-59ec79c0c50a\" (UID: \"9819202d-bd76-4f6f-b077-59ec79c0c50a\") "
Nov 02 15:53:16.080322 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:16.080224    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9819202d-bd76-4f6f-b077-59ec79c0c50a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9819202d-bd76-4f6f-b077-59ec79c0c50a-workload-certs\") pod \"9819202d-bd76-4f6f-b077-59ec79c0c50a\" (UID: \"9819202d-bd76-4f6f-b077-59ec79c0c50a\") "
Nov 02 15:53:16.080322 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:16.080263    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9819202d-bd76-4f6f-b077-59ec79c0c50a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9819202d-bd76-4f6f-b077-59ec79c0c50a-istio-data\") pod \"9819202d-bd76-4f6f-b077-59ec79c0c50a\" (UID: \"9819202d-bd76-4f6f-b077-59ec79c0c50a\") "
Nov 02 15:53:16.080322 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:16.080293    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9819202d-bd76-4f6f-b077-59ec79c0c50a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9819202d-bd76-4f6f-b077-59ec79c0c50a-workload-socket\") pod \"9819202d-bd76-4f6f-b077-59ec79c0c50a\" (UID: \"9819202d-bd76-4f6f-b077-59ec79c0c50a\") "
Nov 02 15:53:16.080506 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:16.080351    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9819202d-bd76-4f6f-b077-59ec79c0c50a-istio-token\" (UniqueName: \"kubernetes.io/projected/9819202d-bd76-4f6f-b077-59ec79c0c50a-istio-token\") pod \"9819202d-bd76-4f6f-b077-59ec79c0c50a\" (UID: \"9819202d-bd76-4f6f-b077-59ec79c0c50a\") "
Nov 02 15:53:16.080506 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:16.080402    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9819202d-bd76-4f6f-b077-59ec79c0c50a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9819202d-bd76-4f6f-b077-59ec79c0c50a-credential-socket\") pod \"9819202d-bd76-4f6f-b077-59ec79c0c50a\" (UID: \"9819202d-bd76-4f6f-b077-59ec79c0c50a\") "
Nov 02 15:53:16.080506 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:16.080445    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9819202d-bd76-4f6f-b077-59ec79c0c50a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9819202d-bd76-4f6f-b077-59ec79c0c50a-istio-envoy\") pod \"9819202d-bd76-4f6f-b077-59ec79c0c50a\" (UID: \"9819202d-bd76-4f6f-b077-59ec79c0c50a\") "
Nov 02 15:53:16.080701 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:16.080629    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9819202d-bd76-4f6f-b077-59ec79c0c50a-istiod-ca-cert" pod "9819202d-bd76-4f6f-b077-59ec79c0c50a" (UID: "9819202d-bd76-4f6f-b077-59ec79c0c50a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:53:16.080701 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:16.080641    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9819202d-bd76-4f6f-b077-59ec79c0c50a-workload-certs" pod "9819202d-bd76-4f6f-b077-59ec79c0c50a" (UID: "9819202d-bd76-4f6f-b077-59ec79c0c50a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:53:16.081027 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:16.080957    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9819202d-bd76-4f6f-b077-59ec79c0c50a-credential-socket" pod "9819202d-bd76-4f6f-b077-59ec79c0c50a" (UID: "9819202d-bd76-4f6f-b077-59ec79c0c50a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:53:16.081580 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:16.081517    5304 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9819202d-bd76-4f6f-b077-59ec79c0c50a-credential-socket\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:53:16.081580 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:16.081545    5304 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9819202d-bd76-4f6f-b077-59ec79c0c50a-istiod-ca-cert\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:53:16.081580 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:16.081560    5304 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9819202d-bd76-4f6f-b077-59ec79c0c50a-workload-certs\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:53:16.081949 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:16.081703    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9819202d-bd76-4f6f-b077-59ec79c0c50a-workload-socket" pod "9819202d-bd76-4f6f-b077-59ec79c0c50a" (UID: "9819202d-bd76-4f6f-b077-59ec79c0c50a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:53:16.081949 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:16.081863    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9819202d-bd76-4f6f-b077-59ec79c0c50a-istio-data" pod "9819202d-bd76-4f6f-b077-59ec79c0c50a" (UID: "9819202d-bd76-4f6f-b077-59ec79c0c50a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:53:16.083830 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:16.083103    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9819202d-bd76-4f6f-b077-59ec79c0c50a-istio-podinfo" pod "9819202d-bd76-4f6f-b077-59ec79c0c50a" (UID: "9819202d-bd76-4f6f-b077-59ec79c0c50a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 15:53:16.089012 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:16.088980    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9819202d-bd76-4f6f-b077-59ec79c0c50a-kube-api-access-6d28d" pod "9819202d-bd76-4f6f-b077-59ec79c0c50a" (UID: "9819202d-bd76-4f6f-b077-59ec79c0c50a"). InnerVolumeSpecName "kube-api-access-6d28d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:53:16.120181 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:16.120087    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9819202d-bd76-4f6f-b077-59ec79c0c50a-istio-token" pod "9819202d-bd76-4f6f-b077-59ec79c0c50a" (UID: "9819202d-bd76-4f6f-b077-59ec79c0c50a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:53:16.121699 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:16.121668    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9819202d-bd76-4f6f-b077-59ec79c0c50a-istio-envoy" pod "9819202d-bd76-4f6f-b077-59ec79c0c50a" (UID: "9819202d-bd76-4f6f-b077-59ec79c0c50a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:53:16.182366 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:16.182323    5304 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9819202d-bd76-4f6f-b077-59ec79c0c50a-istio-token\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:53:16.182366 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:16.182363    5304 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9819202d-bd76-4f6f-b077-59ec79c0c50a-istio-envoy\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:53:16.182602 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:16.182378    5304 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9819202d-bd76-4f6f-b077-59ec79c0c50a-istio-podinfo\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:53:16.182602 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:16.182394    5304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6d28d\" (UniqueName: \"kubernetes.io/projected/9819202d-bd76-4f6f-b077-59ec79c0c50a-kube-api-access-6d28d\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:53:16.182602 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:16.182409    5304 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9819202d-bd76-4f6f-b077-59ec79c0c50a-istio-data\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:53:16.182602 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:16.182424    5304 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9819202d-bd76-4f6f-b077-59ec79c0c50a-workload-socket\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:53:16.472984 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:16.472892    5304 generic.go:356] "Generic (PLEG): container finished" podID="9819202d-bd76-4f6f-b077-59ec79c0c50a" containerID="c48074e374290de95f1882f29f363dfe1128af91498a3b92a31a69287ba520af" exitCode=0
Nov 02 15:53:16.472984 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:16.472945    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-6f2e0c-openshift-default-767db5d9b7-6jmzj"
Nov 02 15:53:16.472984 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:16.472969    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-6f2e0c-openshift-default-767db5d9b7-6jmzj" event={"ID":"9819202d-bd76-4f6f-b077-59ec79c0c50a","Type":"ContainerDied","Data":"c48074e374290de95f1882f29f363dfe1128af91498a3b92a31a69287ba520af"}
Nov 02 15:53:16.473270 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:16.473024    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-6f2e0c-openshift-default-767db5d9b7-6jmzj" event={"ID":"9819202d-bd76-4f6f-b077-59ec79c0c50a","Type":"ContainerDied","Data":"b81724df4b358b287df51821862409e240c265e9e2efd3d7a25c037717459339"}
Nov 02 15:53:16.473270 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:16.473047    5304 scope.go:122] "RemoveContainer" containerID="c48074e374290de95f1882f29f363dfe1128af91498a3b92a31a69287ba520af"
Nov 02 15:53:16.483009 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:16.482991    5304 scope.go:122] "RemoveContainer" containerID="46d8b1f7cadd9125636d38189583e802ef17f667738eec2f0998339ef29ba8f9"
Nov 02 15:53:16.493291 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:16.493271    5304 scope.go:122] "RemoveContainer" containerID="c48074e374290de95f1882f29f363dfe1128af91498a3b92a31a69287ba520af"
Nov 02 15:53:16.493563 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:53:16.493540    5304 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c48074e374290de95f1882f29f363dfe1128af91498a3b92a31a69287ba520af\": container with ID starting with c48074e374290de95f1882f29f363dfe1128af91498a3b92a31a69287ba520af not found: ID does not exist" containerID="c48074e374290de95f1882f29f363dfe1128af91498a3b92a31a69287ba520af"
Nov 02 15:53:16.493691 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:16.493576    5304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c48074e374290de95f1882f29f363dfe1128af91498a3b92a31a69287ba520af"} err="failed to get container status \"c48074e374290de95f1882f29f363dfe1128af91498a3b92a31a69287ba520af\": rpc error: code = NotFound desc = could not find container \"c48074e374290de95f1882f29f363dfe1128af91498a3b92a31a69287ba520af\": container with ID starting with c48074e374290de95f1882f29f363dfe1128af91498a3b92a31a69287ba520af not found: ID does not exist"
Nov 02 15:53:16.493691 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:16.493603    5304 scope.go:122] "RemoveContainer" containerID="46d8b1f7cadd9125636d38189583e802ef17f667738eec2f0998339ef29ba8f9"
Nov 02 15:53:16.493857 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:53:16.493838    5304 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"46d8b1f7cadd9125636d38189583e802ef17f667738eec2f0998339ef29ba8f9\": container with ID starting with 46d8b1f7cadd9125636d38189583e802ef17f667738eec2f0998339ef29ba8f9 not found: ID does not exist" containerID="46d8b1f7cadd9125636d38189583e802ef17f667738eec2f0998339ef29ba8f9"
Nov 02 15:53:16.493898 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:16.493864    5304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"46d8b1f7cadd9125636d38189583e802ef17f667738eec2f0998339ef29ba8f9"} err="failed to get container status \"46d8b1f7cadd9125636d38189583e802ef17f667738eec2f0998339ef29ba8f9\": rpc error: code = NotFound desc = could not find container \"46d8b1f7cadd9125636d38189583e802ef17f667738eec2f0998339ef29ba8f9\": container with ID starting with 46d8b1f7cadd9125636d38189583e802ef17f667738eec2f0998339ef29ba8f9 not found: ID does not exist"
Nov 02 15:53:16.531596 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:16.531561    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-6f2e0c-openshift-default-767db5d9b7-6jmzj"]
Nov 02 15:53:16.543270 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:16.543240    5304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-6f2e0c-openshift-default-767db5d9b7-6jmzj"]
Nov 02 15:53:18.118288 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:18.118257    5304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9819202d-bd76-4f6f-b077-59ec79c0c50a" path="/var/lib/kubelet/pods/9819202d-bd76-4f6f-b077-59ec79c0c50a/volumes"
Nov 02 15:53:19.375992 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:19.375931    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-6f2e0c-6568cc97f7-c7h8r"
Nov 02 15:53:19.490775 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:19.490741    5304 generic.go:356] "Generic (PLEG): container finished" podID="9c90c9e1-7216-4ffb-b513-f5ca7fec318a" containerID="31267b61932edcb7a3e585accc0405f9f7597f0b04201107e586506712fcd64d" exitCode=0
Nov 02 15:53:19.490956 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:19.490789    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-6f2e0c-6568cc97f7-c7h8r"
Nov 02 15:53:19.490956 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:19.490827    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-6f2e0c-6568cc97f7-c7h8r" event={"ID":"9c90c9e1-7216-4ffb-b513-f5ca7fec318a","Type":"ContainerDied","Data":"31267b61932edcb7a3e585accc0405f9f7597f0b04201107e586506712fcd64d"}
Nov 02 15:53:19.490956 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:19.490895    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-6f2e0c-6568cc97f7-c7h8r" event={"ID":"9c90c9e1-7216-4ffb-b513-f5ca7fec318a","Type":"ContainerDied","Data":"6a10d8720260ebc50803b06f2c87a6b6c7da679a62837a307c1cc7e510bc1498"}
Nov 02 15:53:19.490956 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:19.490925    5304 scope.go:122] "RemoveContainer" containerID="31267b61932edcb7a3e585accc0405f9f7597f0b04201107e586506712fcd64d"
Nov 02 15:53:19.501002 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:19.500983    5304 scope.go:122] "RemoveContainer" containerID="31267b61932edcb7a3e585accc0405f9f7597f0b04201107e586506712fcd64d"
Nov 02 15:53:19.501325 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:53:19.501305    5304 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"31267b61932edcb7a3e585accc0405f9f7597f0b04201107e586506712fcd64d\": container with ID starting with 31267b61932edcb7a3e585accc0405f9f7597f0b04201107e586506712fcd64d not found: ID does not exist" containerID="31267b61932edcb7a3e585accc0405f9f7597f0b04201107e586506712fcd64d"
Nov 02 15:53:19.501389 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:19.501333    5304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"31267b61932edcb7a3e585accc0405f9f7597f0b04201107e586506712fcd64d"} err="failed to get container status \"31267b61932edcb7a3e585accc0405f9f7597f0b04201107e586506712fcd64d\": rpc error: code = NotFound desc = could not find container \"31267b61932edcb7a3e585accc0405f9f7597f0b04201107e586506712fcd64d\": container with ID starting with 31267b61932edcb7a3e585accc0405f9f7597f0b04201107e586506712fcd64d not found: ID does not exist"
Nov 02 15:53:19.518896 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:19.518869    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9c90c9e1-7216-4ffb-b513-f5ca7fec318a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c90c9e1-7216-4ffb-b513-f5ca7fec318a-plugins-config-volume\") pod \"9c90c9e1-7216-4ffb-b513-f5ca7fec318a\" (UID: \"9c90c9e1-7216-4ffb-b513-f5ca7fec318a\") "
Nov 02 15:53:19.519039 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:19.518911    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c90c9e1-7216-4ffb-b513-f5ca7fec318a-kube-api-access-4bpfq\" (UniqueName: \"kubernetes.io/projected/9c90c9e1-7216-4ffb-b513-f5ca7fec318a-kube-api-access-4bpfq\") pod \"9c90c9e1-7216-4ffb-b513-f5ca7fec318a\" (UID: \"9c90c9e1-7216-4ffb-b513-f5ca7fec318a\") "
Nov 02 15:53:19.519218 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:19.519192    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c90c9e1-7216-4ffb-b513-f5ca7fec318a-plugins-config-volume" pod "9c90c9e1-7216-4ffb-b513-f5ca7fec318a" (UID: "9c90c9e1-7216-4ffb-b513-f5ca7fec318a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:53:19.521014 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:19.520994    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c90c9e1-7216-4ffb-b513-f5ca7fec318a-kube-api-access-4bpfq" pod "9c90c9e1-7216-4ffb-b513-f5ca7fec318a" (UID: "9c90c9e1-7216-4ffb-b513-f5ca7fec318a"). InnerVolumeSpecName "kube-api-access-4bpfq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:53:19.624546 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:19.624506    5304 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c90c9e1-7216-4ffb-b513-f5ca7fec318a-plugins-config-volume\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:53:19.624546 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:19.624542    5304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4bpfq\" (UniqueName: \"kubernetes.io/projected/9c90c9e1-7216-4ffb-b513-f5ca7fec318a-kube-api-access-4bpfq\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:53:19.836805 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:19.836769    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-6f2e0c-6568cc97f7-c7h8r"]
Nov 02 15:53:19.848622 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:19.848589    5304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-6f2e0c-6568cc97f7-c7h8r"]
Nov 02 15:53:20.117165 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:20.117098    5304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9c90c9e1-7216-4ffb-b513-f5ca7fec318a" path="/var/lib/kubelet/pods/9c90c9e1-7216-4ffb-b513-f5ca7fec318a/volumes"
Nov 02 15:53:24.801857 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:53:24.801817    5304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k6fjk" containerName="extractor"
Nov 02 15:53:31.298260 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:31.298221    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-f6f1a0-bb944cbdc-m2w4j"]
Nov 02 15:53:33.440638 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:33.440607    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-797488-openshift-default-5f79c8745f-7mjt4"]
Nov 02 15:53:33.441088 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:33.440819    5304 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-797488-openshift-default-5f79c8745f-7mjt4" podUID="ecdcaba8-df41-4c1b-a442-d03f9994a5e4" containerName="istio-proxy" containerID="cri-o://34672f21390ce22237ff0ab63fe954251fd27d630e81bd9d33ffad0b12ba168f" gracePeriod=30
Nov 02 15:53:34.109514 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:34.109481    5304 patch_prober.go:29] interesting pod/e2e-shared-b-w2-797488-openshift-default-5f79c8745f-7mjt4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 15:53:34.109748 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:34.109536    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w2-797488-openshift-default-5f79c8745f-7mjt4" podUID="ecdcaba8-df41-4c1b-a442-d03f9994a5e4" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 15:53:36.299264 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:36.299205    5304 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-f6f1a0-bb944cbdc-m2w4j" podUID="0fc51d17-635f-4f36-90d6-f2c546a457ba" containerName="payload-processing" containerID="cri-o://8f9bf1e387ebe321eb770ccd6dbd945057133d3e238474c297ed03ced5bd7bcf" gracePeriod=25
Nov 02 15:53:38.598998 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:38.598965    5304 generic.go:356] "Generic (PLEG): container finished" podID="ecdcaba8-df41-4c1b-a442-d03f9994a5e4" containerID="34672f21390ce22237ff0ab63fe954251fd27d630e81bd9d33ffad0b12ba168f" exitCode=0
Nov 02 15:53:38.599410 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:38.599047    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-797488-openshift-default-5f79c8745f-7mjt4" event={"ID":"ecdcaba8-df41-4c1b-a442-d03f9994a5e4","Type":"ContainerDied","Data":"34672f21390ce22237ff0ab63fe954251fd27d630e81bd9d33ffad0b12ba168f"}
Nov 02 15:53:38.707640 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:38.707586    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-797488-openshift-default-5f79c8745f-7mjt4"
Nov 02 15:53:38.827757 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:38.827721    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ecdcaba8-df41-4c1b-a442-d03f9994a5e4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ecdcaba8-df41-4c1b-a442-d03f9994a5e4-istio-envoy\") pod \"ecdcaba8-df41-4c1b-a442-d03f9994a5e4\" (UID: \"ecdcaba8-df41-4c1b-a442-d03f9994a5e4\") "
Nov 02 15:53:38.827952 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:38.827762    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ecdcaba8-df41-4c1b-a442-d03f9994a5e4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ecdcaba8-df41-4c1b-a442-d03f9994a5e4-workload-certs\") pod \"ecdcaba8-df41-4c1b-a442-d03f9994a5e4\" (UID: \"ecdcaba8-df41-4c1b-a442-d03f9994a5e4\") "
Nov 02 15:53:38.827952 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:38.827798    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ecdcaba8-df41-4c1b-a442-d03f9994a5e4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ecdcaba8-df41-4c1b-a442-d03f9994a5e4-istio-data\") pod \"ecdcaba8-df41-4c1b-a442-d03f9994a5e4\" (UID: \"ecdcaba8-df41-4c1b-a442-d03f9994a5e4\") "
Nov 02 15:53:38.827952 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:38.827817    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ecdcaba8-df41-4c1b-a442-d03f9994a5e4-istio-token\" (UniqueName: \"kubernetes.io/projected/ecdcaba8-df41-4c1b-a442-d03f9994a5e4-istio-token\") pod \"ecdcaba8-df41-4c1b-a442-d03f9994a5e4\" (UID: \"ecdcaba8-df41-4c1b-a442-d03f9994a5e4\") "
Nov 02 15:53:38.827952 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:38.827871    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ecdcaba8-df41-4c1b-a442-d03f9994a5e4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecdcaba8-df41-4c1b-a442-d03f9994a5e4-credential-socket\") pod \"ecdcaba8-df41-4c1b-a442-d03f9994a5e4\" (UID: \"ecdcaba8-df41-4c1b-a442-d03f9994a5e4\") "
Nov 02 15:53:38.827952 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:38.827897    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ecdcaba8-df41-4c1b-a442-d03f9994a5e4-kube-api-access-d6fqb\" (UniqueName: \"kubernetes.io/projected/ecdcaba8-df41-4c1b-a442-d03f9994a5e4-kube-api-access-d6fqb\") pod \"ecdcaba8-df41-4c1b-a442-d03f9994a5e4\" (UID: \"ecdcaba8-df41-4c1b-a442-d03f9994a5e4\") "
Nov 02 15:53:38.828636 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:38.827961    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ecdcaba8-df41-4c1b-a442-d03f9994a5e4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecdcaba8-df41-4c1b-a442-d03f9994a5e4-workload-socket\") pod \"ecdcaba8-df41-4c1b-a442-d03f9994a5e4\" (UID: \"ecdcaba8-df41-4c1b-a442-d03f9994a5e4\") "
Nov 02 15:53:38.828636 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:38.827992    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ecdcaba8-df41-4c1b-a442-d03f9994a5e4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ecdcaba8-df41-4c1b-a442-d03f9994a5e4-istio-podinfo\") pod \"ecdcaba8-df41-4c1b-a442-d03f9994a5e4\" (UID: \"ecdcaba8-df41-4c1b-a442-d03f9994a5e4\") "
Nov 02 15:53:38.828636 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:38.828121    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecdcaba8-df41-4c1b-a442-d03f9994a5e4-istio-data" pod "ecdcaba8-df41-4c1b-a442-d03f9994a5e4" (UID: "ecdcaba8-df41-4c1b-a442-d03f9994a5e4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:53:38.828636 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:38.828167    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ecdcaba8-df41-4c1b-a442-d03f9994a5e4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ecdcaba8-df41-4c1b-a442-d03f9994a5e4-istiod-ca-cert\") pod \"ecdcaba8-df41-4c1b-a442-d03f9994a5e4\" (UID: \"ecdcaba8-df41-4c1b-a442-d03f9994a5e4\") "
Nov 02 15:53:38.828636 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:38.828458    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ecdcaba8-df41-4c1b-a442-d03f9994a5e4-istiod-ca-cert" pod "ecdcaba8-df41-4c1b-a442-d03f9994a5e4" (UID: "ecdcaba8-df41-4c1b-a442-d03f9994a5e4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:53:38.828636 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:38.828552    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecdcaba8-df41-4c1b-a442-d03f9994a5e4-workload-socket" pod "ecdcaba8-df41-4c1b-a442-d03f9994a5e4" (UID: "ecdcaba8-df41-4c1b-a442-d03f9994a5e4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:53:38.828636 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:38.828574    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecdcaba8-df41-4c1b-a442-d03f9994a5e4-workload-certs" pod "ecdcaba8-df41-4c1b-a442-d03f9994a5e4" (UID: "ecdcaba8-df41-4c1b-a442-d03f9994a5e4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:53:38.829061 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:38.828666    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecdcaba8-df41-4c1b-a442-d03f9994a5e4-credential-socket" pod "ecdcaba8-df41-4c1b-a442-d03f9994a5e4" (UID: "ecdcaba8-df41-4c1b-a442-d03f9994a5e4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:53:38.829061 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:38.828809    5304 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ecdcaba8-df41-4c1b-a442-d03f9994a5e4-istiod-ca-cert\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:53:38.829061 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:38.828831    5304 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ecdcaba8-df41-4c1b-a442-d03f9994a5e4-workload-certs\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:53:38.829061 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:38.828845    5304 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ecdcaba8-df41-4c1b-a442-d03f9994a5e4-istio-data\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:53:38.829061 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:38.828860    5304 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecdcaba8-df41-4c1b-a442-d03f9994a5e4-credential-socket\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:53:38.829061 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:38.828875    5304 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecdcaba8-df41-4c1b-a442-d03f9994a5e4-workload-socket\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:53:38.830214 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:38.830186    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecdcaba8-df41-4c1b-a442-d03f9994a5e4-istio-envoy" pod "ecdcaba8-df41-4c1b-a442-d03f9994a5e4" (UID: "ecdcaba8-df41-4c1b-a442-d03f9994a5e4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:53:38.830405 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:38.830375    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ecdcaba8-df41-4c1b-a442-d03f9994a5e4-kube-api-access-d6fqb" pod "ecdcaba8-df41-4c1b-a442-d03f9994a5e4" (UID: "ecdcaba8-df41-4c1b-a442-d03f9994a5e4"). InnerVolumeSpecName "kube-api-access-d6fqb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:53:38.830525 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:38.830427    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ecdcaba8-df41-4c1b-a442-d03f9994a5e4-istio-token" pod "ecdcaba8-df41-4c1b-a442-d03f9994a5e4" (UID: "ecdcaba8-df41-4c1b-a442-d03f9994a5e4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:53:38.830756 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:38.830738    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ecdcaba8-df41-4c1b-a442-d03f9994a5e4-istio-podinfo" pod "ecdcaba8-df41-4c1b-a442-d03f9994a5e4" (UID: "ecdcaba8-df41-4c1b-a442-d03f9994a5e4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 15:53:38.874091 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:38.874048    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-739bf1/distinct-2-w2-739bf1-kserve-5c58479c48-qgfkh"]
Nov 02 15:53:38.901227 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:38.901195    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-739bf1/simulator-w2-739bf1-kserve-7c74f85465-ltk4t"]
Nov 02 15:53:38.929814 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:38.929783    5304 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ecdcaba8-df41-4c1b-a442-d03f9994a5e4-istio-envoy\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:53:38.929814 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:38.929812    5304 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ecdcaba8-df41-4c1b-a442-d03f9994a5e4-istio-token\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:53:38.929814 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:38.929822    5304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d6fqb\" (UniqueName: \"kubernetes.io/projected/ecdcaba8-df41-4c1b-a442-d03f9994a5e4-kube-api-access-d6fqb\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:53:38.930120 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:38.929833    5304 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ecdcaba8-df41-4c1b-a442-d03f9994a5e4-istio-podinfo\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:53:39.606573 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:39.606541    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-797488-openshift-default-5f79c8745f-7mjt4"
Nov 02 15:53:39.606573 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:39.606561    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-797488-openshift-default-5f79c8745f-7mjt4" event={"ID":"ecdcaba8-df41-4c1b-a442-d03f9994a5e4","Type":"ContainerDied","Data":"2ed90fd35e803742244f5cc2b268269be2116a35afedc228fe7ba66f8e6fda34"}
Nov 02 15:53:39.607119 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:39.606630    5304 scope.go:122] "RemoveContainer" containerID="34672f21390ce22237ff0ab63fe954251fd27d630e81bd9d33ffad0b12ba168f"
Nov 02 15:53:39.673715 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:39.673682    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-797488-openshift-default-5f79c8745f-7mjt4"]
Nov 02 15:53:39.685457 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:39.685428    5304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-797488-openshift-default-5f79c8745f-7mjt4"]
Nov 02 15:53:40.118202 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:40.118168    5304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ecdcaba8-df41-4c1b-a442-d03f9994a5e4" path="/var/lib/kubelet/pods/ecdcaba8-df41-4c1b-a442-d03f9994a5e4/volumes"
Nov 02 15:53:43.218747 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:43.218712    5304 scope.go:122] "RemoveContainer" containerID="1f5839bf02cbda85b66db58ada186123f878967d3b3ed2c8a98aa959be4211aa"
Nov 02 15:53:43.620791 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:43.620757    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-f6f1a0-bb944cbdc-m2w4j" podUID="0fc51d17-635f-4f36-90d6-f2c546a457ba" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 15:53:45.512695 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:45.512655    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-f6f1a0-openshift-default-cd57f47bf-2p5bx"]
Nov 02 15:53:45.513221 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:45.512874    5304 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-f6f1a0-openshift-default-cd57f47bf-2p5bx" podUID="55aea2b2-3109-4d02-a3d5-c65835f4d6bf" containerName="istio-proxy" containerID="cri-o://92d99e959f1dcec0272ffbe25756aab6e0c5b705c3507c2463e1d0eeb51f793b" gracePeriod=30
Nov 02 15:53:50.672789 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:50.672753    5304 generic.go:356] "Generic (PLEG): container finished" podID="55aea2b2-3109-4d02-a3d5-c65835f4d6bf" containerID="92d99e959f1dcec0272ffbe25756aab6e0c5b705c3507c2463e1d0eeb51f793b" exitCode=0
Nov 02 15:53:50.673243 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:50.672792    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-f6f1a0-openshift-default-cd57f47bf-2p5bx" event={"ID":"55aea2b2-3109-4d02-a3d5-c65835f4d6bf","Type":"ContainerDied","Data":"92d99e959f1dcec0272ffbe25756aab6e0c5b705c3507c2463e1d0eeb51f793b"}
Nov 02 15:53:50.673243 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:50.672861    5304 scope.go:122] "RemoveContainer" containerID="fcc4d456dad10bb63c54e6a834b29b56725b6ca722ddf1fd64f22249c1641da9"
Nov 02 15:53:50.769074 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:50.769047    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-f6f1a0-openshift-default-cd57f47bf-2p5bx"
Nov 02 15:53:50.852635 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:50.852599    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55aea2b2-3109-4d02-a3d5-c65835f4d6bf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55aea2b2-3109-4d02-a3d5-c65835f4d6bf-credential-socket\") pod \"55aea2b2-3109-4d02-a3d5-c65835f4d6bf\" (UID: \"55aea2b2-3109-4d02-a3d5-c65835f4d6bf\") "
Nov 02 15:53:50.852809 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:50.852651    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55aea2b2-3109-4d02-a3d5-c65835f4d6bf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55aea2b2-3109-4d02-a3d5-c65835f4d6bf-workload-socket\") pod \"55aea2b2-3109-4d02-a3d5-c65835f4d6bf\" (UID: \"55aea2b2-3109-4d02-a3d5-c65835f4d6bf\") "
Nov 02 15:53:50.852809 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:50.852685    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/55aea2b2-3109-4d02-a3d5-c65835f4d6bf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55aea2b2-3109-4d02-a3d5-c65835f4d6bf-istio-podinfo\") pod \"55aea2b2-3109-4d02-a3d5-c65835f4d6bf\" (UID: \"55aea2b2-3109-4d02-a3d5-c65835f4d6bf\") "
Nov 02 15:53:50.852809 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:50.852722    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55aea2b2-3109-4d02-a3d5-c65835f4d6bf-kube-api-access-8qbxk\" (UniqueName: \"kubernetes.io/projected/55aea2b2-3109-4d02-a3d5-c65835f4d6bf-kube-api-access-8qbxk\") pod \"55aea2b2-3109-4d02-a3d5-c65835f4d6bf\" (UID: \"55aea2b2-3109-4d02-a3d5-c65835f4d6bf\") "
Nov 02 15:53:50.853006 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:50.852822    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55aea2b2-3109-4d02-a3d5-c65835f4d6bf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55aea2b2-3109-4d02-a3d5-c65835f4d6bf-istio-data\") pod \"55aea2b2-3109-4d02-a3d5-c65835f4d6bf\" (UID: \"55aea2b2-3109-4d02-a3d5-c65835f4d6bf\") "
Nov 02 15:53:50.853006 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:50.852861    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55aea2b2-3109-4d02-a3d5-c65835f4d6bf-istio-token\" (UniqueName: \"kubernetes.io/projected/55aea2b2-3109-4d02-a3d5-c65835f4d6bf-istio-token\") pod \"55aea2b2-3109-4d02-a3d5-c65835f4d6bf\" (UID: \"55aea2b2-3109-4d02-a3d5-c65835f4d6bf\") "
Nov 02 15:53:50.853006 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:50.852890    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55aea2b2-3109-4d02-a3d5-c65835f4d6bf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55aea2b2-3109-4d02-a3d5-c65835f4d6bf-istio-envoy\") pod \"55aea2b2-3109-4d02-a3d5-c65835f4d6bf\" (UID: \"55aea2b2-3109-4d02-a3d5-c65835f4d6bf\") "
Nov 02 15:53:50.853006 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:50.852895    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55aea2b2-3109-4d02-a3d5-c65835f4d6bf-credential-socket" pod "55aea2b2-3109-4d02-a3d5-c65835f4d6bf" (UID: "55aea2b2-3109-4d02-a3d5-c65835f4d6bf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:53:50.853006 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:50.852987    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55aea2b2-3109-4d02-a3d5-c65835f4d6bf-workload-socket" pod "55aea2b2-3109-4d02-a3d5-c65835f4d6bf" (UID: "55aea2b2-3109-4d02-a3d5-c65835f4d6bf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:53:50.853327 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:50.853024    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55aea2b2-3109-4d02-a3d5-c65835f4d6bf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55aea2b2-3109-4d02-a3d5-c65835f4d6bf-workload-certs\") pod \"55aea2b2-3109-4d02-a3d5-c65835f4d6bf\" (UID: \"55aea2b2-3109-4d02-a3d5-c65835f4d6bf\") "
Nov 02 15:53:50.853327 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:50.853062    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/55aea2b2-3109-4d02-a3d5-c65835f4d6bf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55aea2b2-3109-4d02-a3d5-c65835f4d6bf-istiod-ca-cert\") pod \"55aea2b2-3109-4d02-a3d5-c65835f4d6bf\" (UID: \"55aea2b2-3109-4d02-a3d5-c65835f4d6bf\") "
Nov 02 15:53:50.853327 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:50.853263    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55aea2b2-3109-4d02-a3d5-c65835f4d6bf-workload-certs" pod "55aea2b2-3109-4d02-a3d5-c65835f4d6bf" (UID: "55aea2b2-3109-4d02-a3d5-c65835f4d6bf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:53:50.853495 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:50.853375    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55aea2b2-3109-4d02-a3d5-c65835f4d6bf-istio-data" pod "55aea2b2-3109-4d02-a3d5-c65835f4d6bf" (UID: "55aea2b2-3109-4d02-a3d5-c65835f4d6bf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:53:50.853495 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:50.853453    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55aea2b2-3109-4d02-a3d5-c65835f4d6bf-istiod-ca-cert" pod "55aea2b2-3109-4d02-a3d5-c65835f4d6bf" (UID: "55aea2b2-3109-4d02-a3d5-c65835f4d6bf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:53:50.853616 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:50.853554    5304 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55aea2b2-3109-4d02-a3d5-c65835f4d6bf-istio-data\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:53:50.853616 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:50.853574    5304 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55aea2b2-3109-4d02-a3d5-c65835f4d6bf-workload-certs\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:53:50.853616 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:50.853587    5304 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55aea2b2-3109-4d02-a3d5-c65835f4d6bf-istiod-ca-cert\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:53:50.853616 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:50.853599    5304 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55aea2b2-3109-4d02-a3d5-c65835f4d6bf-credential-socket\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:53:50.853616 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:50.853615    5304 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55aea2b2-3109-4d02-a3d5-c65835f4d6bf-workload-socket\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:53:50.855084 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:50.855058    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55aea2b2-3109-4d02-a3d5-c65835f4d6bf-kube-api-access-8qbxk" pod "55aea2b2-3109-4d02-a3d5-c65835f4d6bf" (UID: "55aea2b2-3109-4d02-a3d5-c65835f4d6bf"). InnerVolumeSpecName "kube-api-access-8qbxk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:53:50.855228 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:50.855191    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55aea2b2-3109-4d02-a3d5-c65835f4d6bf-istio-envoy" pod "55aea2b2-3109-4d02-a3d5-c65835f4d6bf" (UID: "55aea2b2-3109-4d02-a3d5-c65835f4d6bf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:53:50.855493 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:50.855469    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/55aea2b2-3109-4d02-a3d5-c65835f4d6bf-istio-podinfo" pod "55aea2b2-3109-4d02-a3d5-c65835f4d6bf" (UID: "55aea2b2-3109-4d02-a3d5-c65835f4d6bf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 15:53:50.855540 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:50.855475    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55aea2b2-3109-4d02-a3d5-c65835f4d6bf-istio-token" pod "55aea2b2-3109-4d02-a3d5-c65835f4d6bf" (UID: "55aea2b2-3109-4d02-a3d5-c65835f4d6bf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:53:50.954781 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:50.954752    5304 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55aea2b2-3109-4d02-a3d5-c65835f4d6bf-istio-podinfo\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:53:50.954781 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:50.954778    5304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8qbxk\" (UniqueName: \"kubernetes.io/projected/55aea2b2-3109-4d02-a3d5-c65835f4d6bf-kube-api-access-8qbxk\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:53:50.955007 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:50.954790    5304 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/55aea2b2-3109-4d02-a3d5-c65835f4d6bf-istio-token\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:53:50.955007 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:50.954798    5304 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55aea2b2-3109-4d02-a3d5-c65835f4d6bf-istio-envoy\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:53:51.687214 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:51.687179    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-f6f1a0-openshift-default-cd57f47bf-2p5bx"
Nov 02 15:53:51.687214 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:51.687192    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-f6f1a0-openshift-default-cd57f47bf-2p5bx" event={"ID":"55aea2b2-3109-4d02-a3d5-c65835f4d6bf","Type":"ContainerDied","Data":"0e8682bd4b5937755a2e6cbd56a44e749b4ca1609acb12e265c477fccc9b1c0e"}
Nov 02 15:53:51.687809 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:51.687285    5304 scope.go:122] "RemoveContainer" containerID="92d99e959f1dcec0272ffbe25756aab6e0c5b705c3507c2463e1d0eeb51f793b"
Nov 02 15:53:51.753539 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:51.753504    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-f6f1a0-openshift-default-cd57f47bf-2p5bx"]
Nov 02 15:53:51.765605 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:51.765511    5304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-f6f1a0-openshift-default-cd57f47bf-2p5bx"]
Nov 02 15:53:52.118671 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:52.118632    5304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="55aea2b2-3109-4d02-a3d5-c65835f4d6bf" path="/var/lib/kubelet/pods/55aea2b2-3109-4d02-a3d5-c65835f4d6bf/volumes"
Nov 02 15:53:53.621594 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:53.621559    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-f6f1a0-bb944cbdc-m2w4j" podUID="0fc51d17-635f-4f36-90d6-f2c546a457ba" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 15:53:53.894168 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:53.894052    5304 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-739bf1/distinct-2-w2-739bf1-kserve-5c58479c48-qgfkh" podUID="9746bf8a-91bd-4715-a02e-ae2e651ca54d" containerName="main" containerID="cri-o://318eb6e36e1c1b0c12dfd74bf206603c50ab68c02996f556680b6f712cde3d5d" gracePeriod=45
Nov 02 15:53:53.919845 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:53.919804    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-739bf1/distinct-2-w2-739bf1-kserve-5c58479c48-qgfkh" podUID="9746bf8a-91bd-4715-a02e-ae2e651ca54d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": dial tcp 10.133.0.61:8000: connect: connection refused"
Nov 02 15:53:53.920653 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:53.920623    5304 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-739bf1/simulator-w2-739bf1-kserve-7c74f85465-ltk4t" podUID="cf6a1ded-28a3-49eb-878b-b4ad77a0bb5c" containerName="main" containerID="cri-o://5475afc0733bb8493b073d13d49dd895ba6cba428a1433374b0572ec4b1c5ed9" gracePeriod=45
Nov 02 15:53:54.217039 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:54.217012    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-739bf1/distinct-2-w2-739bf1-kserve-5c58479c48-qgfkh"
Nov 02 15:53:54.220192 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:54.220169    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-739bf1/simulator-w2-739bf1-kserve-7c74f85465-ltk4t"
Nov 02 15:53:54.288695 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:54.288665    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9746bf8a-91bd-4715-a02e-ae2e651ca54d-home\" (UniqueName: \"kubernetes.io/empty-dir/9746bf8a-91bd-4715-a02e-ae2e651ca54d-home\") pod \"9746bf8a-91bd-4715-a02e-ae2e651ca54d\" (UID: \"9746bf8a-91bd-4715-a02e-ae2e651ca54d\") "
Nov 02 15:53:54.288880 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:54.288721    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf6a1ded-28a3-49eb-878b-b4ad77a0bb5c-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cf6a1ded-28a3-49eb-878b-b4ad77a0bb5c-tmp-dir\") pod \"cf6a1ded-28a3-49eb-878b-b4ad77a0bb5c\" (UID: \"cf6a1ded-28a3-49eb-878b-b4ad77a0bb5c\") "
Nov 02 15:53:54.288880 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:54.288765    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9746bf8a-91bd-4715-a02e-ae2e651ca54d-kube-api-access-cwwcn\" (UniqueName: \"kubernetes.io/projected/9746bf8a-91bd-4715-a02e-ae2e651ca54d-kube-api-access-cwwcn\") pod \"9746bf8a-91bd-4715-a02e-ae2e651ca54d\" (UID: \"9746bf8a-91bd-4715-a02e-ae2e651ca54d\") "
Nov 02 15:53:54.288880 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:54.288868    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf6a1ded-28a3-49eb-878b-b4ad77a0bb5c-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cf6a1ded-28a3-49eb-878b-b4ad77a0bb5c-model-cache\") pod \"cf6a1ded-28a3-49eb-878b-b4ad77a0bb5c\" (UID: \"cf6a1ded-28a3-49eb-878b-b4ad77a0bb5c\") "
Nov 02 15:53:54.289087 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:54.288939    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9746bf8a-91bd-4715-a02e-ae2e651ca54d-tls-certs\" (UniqueName: \"kubernetes.io/secret/9746bf8a-91bd-4715-a02e-ae2e651ca54d-tls-certs\") pod \"9746bf8a-91bd-4715-a02e-ae2e651ca54d\" (UID: \"9746bf8a-91bd-4715-a02e-ae2e651ca54d\") "
Nov 02 15:53:54.289087 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:54.289028    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cf6a1ded-28a3-49eb-878b-b4ad77a0bb5c-tls-certs\" (UniqueName: \"kubernetes.io/secret/cf6a1ded-28a3-49eb-878b-b4ad77a0bb5c-tls-certs\") pod \"cf6a1ded-28a3-49eb-878b-b4ad77a0bb5c\" (UID: \"cf6a1ded-28a3-49eb-878b-b4ad77a0bb5c\") "
Nov 02 15:53:54.289087 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:54.289037    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9746bf8a-91bd-4715-a02e-ae2e651ca54d-home" pod "9746bf8a-91bd-4715-a02e-ae2e651ca54d" (UID: "9746bf8a-91bd-4715-a02e-ae2e651ca54d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:53:54.289087 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:54.289063    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf6a1ded-28a3-49eb-878b-b4ad77a0bb5c-kube-api-access-92w48\" (UniqueName: \"kubernetes.io/projected/cf6a1ded-28a3-49eb-878b-b4ad77a0bb5c-kube-api-access-92w48\") pod \"cf6a1ded-28a3-49eb-878b-b4ad77a0bb5c\" (UID: \"cf6a1ded-28a3-49eb-878b-b4ad77a0bb5c\") "
Nov 02 15:53:54.289325 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:54.289044    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf6a1ded-28a3-49eb-878b-b4ad77a0bb5c-tmp-dir" pod "cf6a1ded-28a3-49eb-878b-b4ad77a0bb5c" (UID: "cf6a1ded-28a3-49eb-878b-b4ad77a0bb5c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:53:54.289325 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:54.289254    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf6a1ded-28a3-49eb-878b-b4ad77a0bb5c-model-cache" pod "cf6a1ded-28a3-49eb-878b-b4ad77a0bb5c" (UID: "cf6a1ded-28a3-49eb-878b-b4ad77a0bb5c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:53:54.289574 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:54.289553    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf6a1ded-28a3-49eb-878b-b4ad77a0bb5c-dshm\" (UniqueName: \"kubernetes.io/empty-dir/cf6a1ded-28a3-49eb-878b-b4ad77a0bb5c-dshm\") pod \"cf6a1ded-28a3-49eb-878b-b4ad77a0bb5c\" (UID: \"cf6a1ded-28a3-49eb-878b-b4ad77a0bb5c\") "
Nov 02 15:53:54.289685 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:54.289593    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9746bf8a-91bd-4715-a02e-ae2e651ca54d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9746bf8a-91bd-4715-a02e-ae2e651ca54d-model-cache\") pod \"9746bf8a-91bd-4715-a02e-ae2e651ca54d\" (UID: \"9746bf8a-91bd-4715-a02e-ae2e651ca54d\") "
Nov 02 15:53:54.289685 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:54.289635    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9746bf8a-91bd-4715-a02e-ae2e651ca54d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9746bf8a-91bd-4715-a02e-ae2e651ca54d-tmp-dir\") pod \"9746bf8a-91bd-4715-a02e-ae2e651ca54d\" (UID: \"9746bf8a-91bd-4715-a02e-ae2e651ca54d\") "
Nov 02 15:53:54.289685 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:54.289669    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9746bf8a-91bd-4715-a02e-ae2e651ca54d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/9746bf8a-91bd-4715-a02e-ae2e651ca54d-dshm\") pod \"9746bf8a-91bd-4715-a02e-ae2e651ca54d\" (UID: \"9746bf8a-91bd-4715-a02e-ae2e651ca54d\") "
Nov 02 15:53:54.289865 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:54.289701    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf6a1ded-28a3-49eb-878b-b4ad77a0bb5c-home\" (UniqueName: \"kubernetes.io/empty-dir/cf6a1ded-28a3-49eb-878b-b4ad77a0bb5c-home\") pod \"cf6a1ded-28a3-49eb-878b-b4ad77a0bb5c\" (UID: \"cf6a1ded-28a3-49eb-878b-b4ad77a0bb5c\") "
Nov 02 15:53:54.290780 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:54.290242    5304 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9746bf8a-91bd-4715-a02e-ae2e651ca54d-home\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:53:54.290780 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:54.290270    5304 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cf6a1ded-28a3-49eb-878b-b4ad77a0bb5c-tmp-dir\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:53:54.290780 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:54.290285    5304 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cf6a1ded-28a3-49eb-878b-b4ad77a0bb5c-model-cache\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:53:54.290780 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:54.290489    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf6a1ded-28a3-49eb-878b-b4ad77a0bb5c-home" pod "cf6a1ded-28a3-49eb-878b-b4ad77a0bb5c" (UID: "cf6a1ded-28a3-49eb-878b-b4ad77a0bb5c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:53:54.290780 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:54.290650    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9746bf8a-91bd-4715-a02e-ae2e651ca54d-tmp-dir" pod "9746bf8a-91bd-4715-a02e-ae2e651ca54d" (UID: "9746bf8a-91bd-4715-a02e-ae2e651ca54d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:53:54.290780 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:54.290742    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9746bf8a-91bd-4715-a02e-ae2e651ca54d-model-cache" pod "9746bf8a-91bd-4715-a02e-ae2e651ca54d" (UID: "9746bf8a-91bd-4715-a02e-ae2e651ca54d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:53:54.292075 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:54.292047    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9746bf8a-91bd-4715-a02e-ae2e651ca54d-kube-api-access-cwwcn" pod "9746bf8a-91bd-4715-a02e-ae2e651ca54d" (UID: "9746bf8a-91bd-4715-a02e-ae2e651ca54d"). InnerVolumeSpecName "kube-api-access-cwwcn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:53:54.292075 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:54.292066    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf6a1ded-28a3-49eb-878b-b4ad77a0bb5c-kube-api-access-92w48" pod "cf6a1ded-28a3-49eb-878b-b4ad77a0bb5c" (UID: "cf6a1ded-28a3-49eb-878b-b4ad77a0bb5c"). InnerVolumeSpecName "kube-api-access-92w48". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:53:54.292251 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:54.292074    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9746bf8a-91bd-4715-a02e-ae2e651ca54d-tls-certs" pod "9746bf8a-91bd-4715-a02e-ae2e651ca54d" (UID: "9746bf8a-91bd-4715-a02e-ae2e651ca54d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:53:54.292468 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:54.292442    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cf6a1ded-28a3-49eb-878b-b4ad77a0bb5c-tls-certs" pod "cf6a1ded-28a3-49eb-878b-b4ad77a0bb5c" (UID: "cf6a1ded-28a3-49eb-878b-b4ad77a0bb5c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:53:54.292693 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:54.292657    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf6a1ded-28a3-49eb-878b-b4ad77a0bb5c-dshm" pod "cf6a1ded-28a3-49eb-878b-b4ad77a0bb5c" (UID: "cf6a1ded-28a3-49eb-878b-b4ad77a0bb5c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:53:54.293153 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:54.293103    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9746bf8a-91bd-4715-a02e-ae2e651ca54d-dshm" pod "9746bf8a-91bd-4715-a02e-ae2e651ca54d" (UID: "9746bf8a-91bd-4715-a02e-ae2e651ca54d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:53:54.391629 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:54.391595    5304 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9746bf8a-91bd-4715-a02e-ae2e651ca54d-tls-certs\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:53:54.391629 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:54.391628    5304 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cf6a1ded-28a3-49eb-878b-b4ad77a0bb5c-tls-certs\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:53:54.391853 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:54.391638    5304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-92w48\" (UniqueName: \"kubernetes.io/projected/cf6a1ded-28a3-49eb-878b-b4ad77a0bb5c-kube-api-access-92w48\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:53:54.391853 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:54.391649    5304 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cf6a1ded-28a3-49eb-878b-b4ad77a0bb5c-dshm\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:53:54.391853 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:54.391659    5304 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9746bf8a-91bd-4715-a02e-ae2e651ca54d-model-cache\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:53:54.391853 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:54.391668    5304 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9746bf8a-91bd-4715-a02e-ae2e651ca54d-tmp-dir\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:53:54.391853 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:54.391675    5304 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9746bf8a-91bd-4715-a02e-ae2e651ca54d-dshm\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:53:54.391853 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:54.391683    5304 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cf6a1ded-28a3-49eb-878b-b4ad77a0bb5c-home\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:53:54.391853 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:54.391692    5304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cwwcn\" (UniqueName: \"kubernetes.io/projected/9746bf8a-91bd-4715-a02e-ae2e651ca54d-kube-api-access-cwwcn\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:53:54.703442 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:54.703408    5304 generic.go:356] "Generic (PLEG): container finished" podID="9746bf8a-91bd-4715-a02e-ae2e651ca54d" containerID="318eb6e36e1c1b0c12dfd74bf206603c50ab68c02996f556680b6f712cde3d5d" exitCode=0
Nov 02 15:53:54.703919 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:54.703455    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-739bf1/distinct-2-w2-739bf1-kserve-5c58479c48-qgfkh"
Nov 02 15:53:54.703919 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:54.703490    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-739bf1/distinct-2-w2-739bf1-kserve-5c58479c48-qgfkh" event={"ID":"9746bf8a-91bd-4715-a02e-ae2e651ca54d","Type":"ContainerDied","Data":"318eb6e36e1c1b0c12dfd74bf206603c50ab68c02996f556680b6f712cde3d5d"}
Nov 02 15:53:54.703919 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:54.703533    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-739bf1/distinct-2-w2-739bf1-kserve-5c58479c48-qgfkh" event={"ID":"9746bf8a-91bd-4715-a02e-ae2e651ca54d","Type":"ContainerDied","Data":"3453060ccb0010a2764e10c301fb0adf36b53e281c3dbd7aded2faff5c826399"}
Nov 02 15:53:54.703919 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:54.703550    5304 scope.go:122] "RemoveContainer" containerID="318eb6e36e1c1b0c12dfd74bf206603c50ab68c02996f556680b6f712cde3d5d"
Nov 02 15:53:54.705150 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:54.705101    5304 generic.go:356] "Generic (PLEG): container finished" podID="cf6a1ded-28a3-49eb-878b-b4ad77a0bb5c" containerID="5475afc0733bb8493b073d13d49dd895ba6cba428a1433374b0572ec4b1c5ed9" exitCode=0
Nov 02 15:53:54.705258 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:54.705166    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-739bf1/simulator-w2-739bf1-kserve-7c74f85465-ltk4t"
Nov 02 15:53:54.705316 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:54.705161    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-739bf1/simulator-w2-739bf1-kserve-7c74f85465-ltk4t" event={"ID":"cf6a1ded-28a3-49eb-878b-b4ad77a0bb5c","Type":"ContainerDied","Data":"5475afc0733bb8493b073d13d49dd895ba6cba428a1433374b0572ec4b1c5ed9"}
Nov 02 15:53:54.705431 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:54.705332    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-739bf1/simulator-w2-739bf1-kserve-7c74f85465-ltk4t" event={"ID":"cf6a1ded-28a3-49eb-878b-b4ad77a0bb5c","Type":"ContainerDied","Data":"b47f48bf334bbae1e02521b22179b9069b7e76b7a1cf8ccae1e93dab301c0c6a"}
Nov 02 15:53:54.717230 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:54.717208    5304 scope.go:122] "RemoveContainer" containerID="318eb6e36e1c1b0c12dfd74bf206603c50ab68c02996f556680b6f712cde3d5d"
Nov 02 15:53:54.717693 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:53:54.717592    5304 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"318eb6e36e1c1b0c12dfd74bf206603c50ab68c02996f556680b6f712cde3d5d\": container with ID starting with 318eb6e36e1c1b0c12dfd74bf206603c50ab68c02996f556680b6f712cde3d5d not found: ID does not exist" containerID="318eb6e36e1c1b0c12dfd74bf206603c50ab68c02996f556680b6f712cde3d5d"
Nov 02 15:53:54.717693 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:54.717627    5304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"318eb6e36e1c1b0c12dfd74bf206603c50ab68c02996f556680b6f712cde3d5d"} err="failed to get container status \"318eb6e36e1c1b0c12dfd74bf206603c50ab68c02996f556680b6f712cde3d5d\": rpc error: code = NotFound desc = could not find container \"318eb6e36e1c1b0c12dfd74bf206603c50ab68c02996f556680b6f712cde3d5d\": container with ID starting with 318eb6e36e1c1b0c12dfd74bf206603c50ab68c02996f556680b6f712cde3d5d not found: ID does not exist"
Nov 02 15:53:54.717693 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:54.717654    5304 scope.go:122] "RemoveContainer" containerID="5475afc0733bb8493b073d13d49dd895ba6cba428a1433374b0572ec4b1c5ed9"
Nov 02 15:53:54.726919 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:54.726901    5304 scope.go:122] "RemoveContainer" containerID="5475afc0733bb8493b073d13d49dd895ba6cba428a1433374b0572ec4b1c5ed9"
Nov 02 15:53:54.727221 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:53:54.727194    5304 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5475afc0733bb8493b073d13d49dd895ba6cba428a1433374b0572ec4b1c5ed9\": container with ID starting with 5475afc0733bb8493b073d13d49dd895ba6cba428a1433374b0572ec4b1c5ed9 not found: ID does not exist" containerID="5475afc0733bb8493b073d13d49dd895ba6cba428a1433374b0572ec4b1c5ed9"
Nov 02 15:53:54.727329 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:54.727226    5304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5475afc0733bb8493b073d13d49dd895ba6cba428a1433374b0572ec4b1c5ed9"} err="failed to get container status \"5475afc0733bb8493b073d13d49dd895ba6cba428a1433374b0572ec4b1c5ed9\": rpc error: code = NotFound desc = could not find container \"5475afc0733bb8493b073d13d49dd895ba6cba428a1433374b0572ec4b1c5ed9\": container with ID starting with 5475afc0733bb8493b073d13d49dd895ba6cba428a1433374b0572ec4b1c5ed9 not found: ID does not exist"
Nov 02 15:53:54.772728 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:54.772695    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-739bf1/simulator-w2-739bf1-kserve-7c74f85465-ltk4t"]
Nov 02 15:53:54.785632 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:54.779262    5304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-739bf1/simulator-w2-739bf1-kserve-7c74f85465-ltk4t"]
Nov 02 15:53:54.800688 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:53:54.800653    5304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k6fjk" containerName="extractor"
Nov 02 15:53:54.812717 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:54.812673    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-739bf1/distinct-2-w2-739bf1-kserve-5c58479c48-qgfkh"]
Nov 02 15:53:54.819109 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:54.819085    5304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-739bf1/distinct-2-w2-739bf1-kserve-5c58479c48-qgfkh"]
Nov 02 15:53:56.117839 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:56.117808    5304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9746bf8a-91bd-4715-a02e-ae2e651ca54d" path="/var/lib/kubelet/pods/9746bf8a-91bd-4715-a02e-ae2e651ca54d/volumes"
Nov 02 15:53:56.118339 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:56.118263    5304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cf6a1ded-28a3-49eb-878b-b4ad77a0bb5c" path="/var/lib/kubelet/pods/cf6a1ded-28a3-49eb-878b-b4ad77a0bb5c/volumes"
Nov 02 15:53:56.558651 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:56.558624    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-f6f1a0-bb944cbdc-m2w4j"
Nov 02 15:53:56.615142 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:56.615109    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0fc51d17-635f-4f36-90d6-f2c546a457ba-kube-api-access-zvrh5\" (UniqueName: \"kubernetes.io/projected/0fc51d17-635f-4f36-90d6-f2c546a457ba-kube-api-access-zvrh5\") pod \"0fc51d17-635f-4f36-90d6-f2c546a457ba\" (UID: \"0fc51d17-635f-4f36-90d6-f2c546a457ba\") "
Nov 02 15:53:56.615331 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:56.615191    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0fc51d17-635f-4f36-90d6-f2c546a457ba-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0fc51d17-635f-4f36-90d6-f2c546a457ba-plugins-config-volume\") pod \"0fc51d17-635f-4f36-90d6-f2c546a457ba\" (UID: \"0fc51d17-635f-4f36-90d6-f2c546a457ba\") "
Nov 02 15:53:56.615549 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:56.615525    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0fc51d17-635f-4f36-90d6-f2c546a457ba-plugins-config-volume" pod "0fc51d17-635f-4f36-90d6-f2c546a457ba" (UID: "0fc51d17-635f-4f36-90d6-f2c546a457ba"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:53:56.617339 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:56.617314    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0fc51d17-635f-4f36-90d6-f2c546a457ba-kube-api-access-zvrh5" pod "0fc51d17-635f-4f36-90d6-f2c546a457ba" (UID: "0fc51d17-635f-4f36-90d6-f2c546a457ba"). InnerVolumeSpecName "kube-api-access-zvrh5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:53:56.716341 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:56.716262    5304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zvrh5\" (UniqueName: \"kubernetes.io/projected/0fc51d17-635f-4f36-90d6-f2c546a457ba-kube-api-access-zvrh5\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:53:56.716341 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:56.716292    5304 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0fc51d17-635f-4f36-90d6-f2c546a457ba-plugins-config-volume\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:53:56.718278 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:56.718246    5304 generic.go:356] "Generic (PLEG): container finished" podID="0fc51d17-635f-4f36-90d6-f2c546a457ba" containerID="8f9bf1e387ebe321eb770ccd6dbd945057133d3e238474c297ed03ced5bd7bcf" exitCode=0
Nov 02 15:53:56.718407 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:56.718296    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-f6f1a0-bb944cbdc-m2w4j"
Nov 02 15:53:56.718407 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:56.718309    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-f6f1a0-bb944cbdc-m2w4j" event={"ID":"0fc51d17-635f-4f36-90d6-f2c546a457ba","Type":"ContainerDied","Data":"8f9bf1e387ebe321eb770ccd6dbd945057133d3e238474c297ed03ced5bd7bcf"}
Nov 02 15:53:56.718407 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:56.718360    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-f6f1a0-bb944cbdc-m2w4j" event={"ID":"0fc51d17-635f-4f36-90d6-f2c546a457ba","Type":"ContainerDied","Data":"b390064ff57c54db0dd21ea71f3e46818cdef48f162e1dde5a3dd5b519636623"}
Nov 02 15:53:56.718407 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:56.718390    5304 scope.go:122] "RemoveContainer" containerID="8f9bf1e387ebe321eb770ccd6dbd945057133d3e238474c297ed03ced5bd7bcf"
Nov 02 15:53:56.728626 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:56.728601    5304 scope.go:122] "RemoveContainer" containerID="8f9bf1e387ebe321eb770ccd6dbd945057133d3e238474c297ed03ced5bd7bcf"
Nov 02 15:53:56.728914 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:53:56.728892    5304 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f9bf1e387ebe321eb770ccd6dbd945057133d3e238474c297ed03ced5bd7bcf\": container with ID starting with 8f9bf1e387ebe321eb770ccd6dbd945057133d3e238474c297ed03ced5bd7bcf not found: ID does not exist" containerID="8f9bf1e387ebe321eb770ccd6dbd945057133d3e238474c297ed03ced5bd7bcf"
Nov 02 15:53:56.728969 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:56.728923    5304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f9bf1e387ebe321eb770ccd6dbd945057133d3e238474c297ed03ced5bd7bcf"} err="failed to get container status \"8f9bf1e387ebe321eb770ccd6dbd945057133d3e238474c297ed03ced5bd7bcf\": rpc error: code = NotFound desc = could not find container \"8f9bf1e387ebe321eb770ccd6dbd945057133d3e238474c297ed03ced5bd7bcf\": container with ID starting with 8f9bf1e387ebe321eb770ccd6dbd945057133d3e238474c297ed03ced5bd7bcf not found: ID does not exist"
Nov 02 15:53:56.797195 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:56.797150    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-f6f1a0-bb944cbdc-m2w4j"]
Nov 02 15:53:56.804064 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:56.804032    5304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-f6f1a0-bb944cbdc-m2w4j"]
Nov 02 15:53:57.570529 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:57.570493    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-f0039c-openshift-default-5f656fc965-rjf2w"]
Nov 02 15:53:57.571007 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:57.570782    5304 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-f0039c-openshift-default-5f656fc965-rjf2w" podUID="9178b9f8-6fda-40a9-8850-fa0097f2d39a" containerName="istio-proxy" containerID="cri-o://75004946a6197bf76c248dc08a3e0e00c4632a344e8913e3d4bb7dddfa8ec931" gracePeriod=30
Nov 02 15:53:58.086342 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:58.086305    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-28da58-openshift-default-76446c9f79-rg6rm"]
Nov 02 15:53:58.086590 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:58.086566    5304 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-28da58-openshift-default-76446c9f79-rg6rm" podUID="e80dea97-6fba-4ad0-b574-da148b1979db" containerName="istio-proxy" containerID="cri-o://af24a6c1d1d4fd1dbe13f28dd9e59a5861478f779204f9941d54d12f7ec52d9c" gracePeriod=30
Nov 02 15:53:58.117793 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:53:58.117759    5304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0fc51d17-635f-4f36-90d6-f2c546a457ba" path="/var/lib/kubelet/pods/0fc51d17-635f-4f36-90d6-f2c546a457ba/volumes"
Nov 02 15:54:02.755635 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:54:02.755600    5304 generic.go:356] "Generic (PLEG): container finished" podID="9178b9f8-6fda-40a9-8850-fa0097f2d39a" containerID="75004946a6197bf76c248dc08a3e0e00c4632a344e8913e3d4bb7dddfa8ec931" exitCode=0
Nov 02 15:54:02.756117 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:54:02.755675    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f0039c-openshift-default-5f656fc965-rjf2w" event={"ID":"9178b9f8-6fda-40a9-8850-fa0097f2d39a","Type":"ContainerDied","Data":"75004946a6197bf76c248dc08a3e0e00c4632a344e8913e3d4bb7dddfa8ec931"}
Nov 02 15:54:02.756117 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:54:02.755743    5304 scope.go:122] "RemoveContainer" containerID="7f93fabea8b9718b8713ce2ac008f397f62178433855a93d33fdd815291278ee"
Nov 02 15:54:02.857782 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:54:02.857758    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-f0039c-openshift-default-5f656fc965-rjf2w"
Nov 02 15:54:02.978720 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:54:02.978689    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9178b9f8-6fda-40a9-8850-fa0097f2d39a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9178b9f8-6fda-40a9-8850-fa0097f2d39a-workload-certs\") pod \"9178b9f8-6fda-40a9-8850-fa0097f2d39a\" (UID: \"9178b9f8-6fda-40a9-8850-fa0097f2d39a\") "
Nov 02 15:54:02.978922 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:54:02.978733    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9178b9f8-6fda-40a9-8850-fa0097f2d39a-istio-token\" (UniqueName: \"kubernetes.io/projected/9178b9f8-6fda-40a9-8850-fa0097f2d39a-istio-token\") pod \"9178b9f8-6fda-40a9-8850-fa0097f2d39a\" (UID: \"9178b9f8-6fda-40a9-8850-fa0097f2d39a\") "
Nov 02 15:54:02.978922 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:54:02.978759    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9178b9f8-6fda-40a9-8850-fa0097f2d39a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9178b9f8-6fda-40a9-8850-fa0097f2d39a-istio-envoy\") pod \"9178b9f8-6fda-40a9-8850-fa0097f2d39a\" (UID: \"9178b9f8-6fda-40a9-8850-fa0097f2d39a\") "
Nov 02 15:54:02.978922 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:54:02.978836    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9178b9f8-6fda-40a9-8850-fa0097f2d39a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9178b9f8-6fda-40a9-8850-fa0097f2d39a-istio-podinfo\") pod \"9178b9f8-6fda-40a9-8850-fa0097f2d39a\" (UID: \"9178b9f8-6fda-40a9-8850-fa0097f2d39a\") "
Nov 02 15:54:02.978922 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:54:02.978868    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9178b9f8-6fda-40a9-8850-fa0097f2d39a-kube-api-access-zxl5t\" (UniqueName: \"kubernetes.io/projected/9178b9f8-6fda-40a9-8850-fa0097f2d39a-kube-api-access-zxl5t\") pod \"9178b9f8-6fda-40a9-8850-fa0097f2d39a\" (UID: \"9178b9f8-6fda-40a9-8850-fa0097f2d39a\") "
Nov 02 15:54:02.978922 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:54:02.978915    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9178b9f8-6fda-40a9-8850-fa0097f2d39a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9178b9f8-6fda-40a9-8850-fa0097f2d39a-credential-socket\") pod \"9178b9f8-6fda-40a9-8850-fa0097f2d39a\" (UID: \"9178b9f8-6fda-40a9-8850-fa0097f2d39a\") "
Nov 02 15:54:02.979284 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:54:02.978987    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9178b9f8-6fda-40a9-8850-fa0097f2d39a-workload-certs" pod "9178b9f8-6fda-40a9-8850-fa0097f2d39a" (UID: "9178b9f8-6fda-40a9-8850-fa0097f2d39a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:54:02.979284 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:54:02.979119    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9178b9f8-6fda-40a9-8850-fa0097f2d39a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9178b9f8-6fda-40a9-8850-fa0097f2d39a-istiod-ca-cert\") pod \"9178b9f8-6fda-40a9-8850-fa0097f2d39a\" (UID: \"9178b9f8-6fda-40a9-8850-fa0097f2d39a\") "
Nov 02 15:54:02.979284 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:54:02.979218    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9178b9f8-6fda-40a9-8850-fa0097f2d39a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9178b9f8-6fda-40a9-8850-fa0097f2d39a-workload-socket\") pod \"9178b9f8-6fda-40a9-8850-fa0097f2d39a\" (UID: \"9178b9f8-6fda-40a9-8850-fa0097f2d39a\") "
Nov 02 15:54:02.979284 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:54:02.979244    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9178b9f8-6fda-40a9-8850-fa0097f2d39a-credential-socket" pod "9178b9f8-6fda-40a9-8850-fa0097f2d39a" (UID: "9178b9f8-6fda-40a9-8850-fa0097f2d39a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:54:02.979498 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:54:02.979307    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9178b9f8-6fda-40a9-8850-fa0097f2d39a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9178b9f8-6fda-40a9-8850-fa0097f2d39a-istio-data\") pod \"9178b9f8-6fda-40a9-8850-fa0097f2d39a\" (UID: \"9178b9f8-6fda-40a9-8850-fa0097f2d39a\") "
Nov 02 15:54:02.979555 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:54:02.979519    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9178b9f8-6fda-40a9-8850-fa0097f2d39a-workload-socket" pod "9178b9f8-6fda-40a9-8850-fa0097f2d39a" (UID: "9178b9f8-6fda-40a9-8850-fa0097f2d39a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:54:02.979618 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:54:02.979568    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9178b9f8-6fda-40a9-8850-fa0097f2d39a-istiod-ca-cert" pod "9178b9f8-6fda-40a9-8850-fa0097f2d39a" (UID: "9178b9f8-6fda-40a9-8850-fa0097f2d39a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:54:02.979767 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:54:02.979743    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9178b9f8-6fda-40a9-8850-fa0097f2d39a-istio-data" pod "9178b9f8-6fda-40a9-8850-fa0097f2d39a" (UID: "9178b9f8-6fda-40a9-8850-fa0097f2d39a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:54:02.979961 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:54:02.979854    5304 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9178b9f8-6fda-40a9-8850-fa0097f2d39a-credential-socket\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:54:02.980025 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:54:02.979975    5304 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9178b9f8-6fda-40a9-8850-fa0097f2d39a-istiod-ca-cert\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:54:02.980025 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:54:02.979998    5304 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9178b9f8-6fda-40a9-8850-fa0097f2d39a-workload-socket\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:54:02.980025 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:54:02.980013    5304 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9178b9f8-6fda-40a9-8850-fa0097f2d39a-workload-certs\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:54:02.981227 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:54:02.981203    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9178b9f8-6fda-40a9-8850-fa0097f2d39a-istio-envoy" pod "9178b9f8-6fda-40a9-8850-fa0097f2d39a" (UID: "9178b9f8-6fda-40a9-8850-fa0097f2d39a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:54:02.981345 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:54:02.981322    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9178b9f8-6fda-40a9-8850-fa0097f2d39a-istio-token" pod "9178b9f8-6fda-40a9-8850-fa0097f2d39a" (UID: "9178b9f8-6fda-40a9-8850-fa0097f2d39a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:54:02.981534 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:54:02.981510    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9178b9f8-6fda-40a9-8850-fa0097f2d39a-kube-api-access-zxl5t" pod "9178b9f8-6fda-40a9-8850-fa0097f2d39a" (UID: "9178b9f8-6fda-40a9-8850-fa0097f2d39a"). InnerVolumeSpecName "kube-api-access-zxl5t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:54:02.981727 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:54:02.981706    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9178b9f8-6fda-40a9-8850-fa0097f2d39a-istio-podinfo" pod "9178b9f8-6fda-40a9-8850-fa0097f2d39a" (UID: "9178b9f8-6fda-40a9-8850-fa0097f2d39a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 15:54:03.081239 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:54:03.081198    5304 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9178b9f8-6fda-40a9-8850-fa0097f2d39a-istio-data\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:54:03.081239 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:54:03.081231    5304 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9178b9f8-6fda-40a9-8850-fa0097f2d39a-istio-token\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:54:03.081239 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:54:03.081242    5304 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9178b9f8-6fda-40a9-8850-fa0097f2d39a-istio-envoy\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:54:03.081501 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:54:03.081253    5304 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9178b9f8-6fda-40a9-8850-fa0097f2d39a-istio-podinfo\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:54:03.081501 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:54:03.081267    5304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zxl5t\" (UniqueName: \"kubernetes.io/projected/9178b9f8-6fda-40a9-8850-fa0097f2d39a-kube-api-access-zxl5t\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:54:03.327565 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:54:03.327538    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-28da58-openshift-default-76446c9f79-rg6rm"
Nov 02 15:54:03.384715 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:54:03.384609    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e80dea97-6fba-4ad0-b574-da148b1979db-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e80dea97-6fba-4ad0-b574-da148b1979db-workload-socket\") pod \"e80dea97-6fba-4ad0-b574-da148b1979db\" (UID: \"e80dea97-6fba-4ad0-b574-da148b1979db\") "
Nov 02 15:54:03.384715 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:54:03.384685    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e80dea97-6fba-4ad0-b574-da148b1979db-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e80dea97-6fba-4ad0-b574-da148b1979db-istio-podinfo\") pod \"e80dea97-6fba-4ad0-b574-da148b1979db\" (UID: \"e80dea97-6fba-4ad0-b574-da148b1979db\") "
Nov 02 15:54:03.384715 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:54:03.384709    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e80dea97-6fba-4ad0-b574-da148b1979db-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e80dea97-6fba-4ad0-b574-da148b1979db-istiod-ca-cert\") pod \"e80dea97-6fba-4ad0-b574-da148b1979db\" (UID: \"e80dea97-6fba-4ad0-b574-da148b1979db\") "
Nov 02 15:54:03.385050 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:54:03.384728    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e80dea97-6fba-4ad0-b574-da148b1979db-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e80dea97-6fba-4ad0-b574-da148b1979db-istio-envoy\") pod \"e80dea97-6fba-4ad0-b574-da148b1979db\" (UID: \"e80dea97-6fba-4ad0-b574-da148b1979db\") "
Nov 02 15:54:03.385050 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:54:03.384797    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e80dea97-6fba-4ad0-b574-da148b1979db-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e80dea97-6fba-4ad0-b574-da148b1979db-istio-data\") pod \"e80dea97-6fba-4ad0-b574-da148b1979db\" (UID: \"e80dea97-6fba-4ad0-b574-da148b1979db\") "
Nov 02 15:54:03.385050 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:54:03.384841    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e80dea97-6fba-4ad0-b574-da148b1979db-kube-api-access-wzcdr\" (UniqueName: \"kubernetes.io/projected/e80dea97-6fba-4ad0-b574-da148b1979db-kube-api-access-wzcdr\") pod \"e80dea97-6fba-4ad0-b574-da148b1979db\" (UID: \"e80dea97-6fba-4ad0-b574-da148b1979db\") "
Nov 02 15:54:03.385050 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:54:03.384884    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e80dea97-6fba-4ad0-b574-da148b1979db-istio-token\" (UniqueName: \"kubernetes.io/projected/e80dea97-6fba-4ad0-b574-da148b1979db-istio-token\") pod \"e80dea97-6fba-4ad0-b574-da148b1979db\" (UID: \"e80dea97-6fba-4ad0-b574-da148b1979db\") "
Nov 02 15:54:03.385050 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:54:03.384919    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e80dea97-6fba-4ad0-b574-da148b1979db-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e80dea97-6fba-4ad0-b574-da148b1979db-workload-certs\") pod \"e80dea97-6fba-4ad0-b574-da148b1979db\" (UID: \"e80dea97-6fba-4ad0-b574-da148b1979db\") "
Nov 02 15:54:03.385050 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:54:03.384922    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e80dea97-6fba-4ad0-b574-da148b1979db-workload-socket" pod "e80dea97-6fba-4ad0-b574-da148b1979db" (UID: "e80dea97-6fba-4ad0-b574-da148b1979db"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:54:03.385050 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:54:03.384948    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e80dea97-6fba-4ad0-b574-da148b1979db-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e80dea97-6fba-4ad0-b574-da148b1979db-credential-socket\") pod \"e80dea97-6fba-4ad0-b574-da148b1979db\" (UID: \"e80dea97-6fba-4ad0-b574-da148b1979db\") "
Nov 02 15:54:03.385532 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:54:03.385250    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e80dea97-6fba-4ad0-b574-da148b1979db-istio-data" pod "e80dea97-6fba-4ad0-b574-da148b1979db" (UID: "e80dea97-6fba-4ad0-b574-da148b1979db"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:54:03.385532 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:54:03.385343    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e80dea97-6fba-4ad0-b574-da148b1979db-workload-certs" pod "e80dea97-6fba-4ad0-b574-da148b1979db" (UID: "e80dea97-6fba-4ad0-b574-da148b1979db"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:54:03.385872 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:54:03.385838    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e80dea97-6fba-4ad0-b574-da148b1979db-istiod-ca-cert" pod "e80dea97-6fba-4ad0-b574-da148b1979db" (UID: "e80dea97-6fba-4ad0-b574-da148b1979db"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:54:03.386146 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:54:03.386088    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e80dea97-6fba-4ad0-b574-da148b1979db-credential-socket" pod "e80dea97-6fba-4ad0-b574-da148b1979db" (UID: "e80dea97-6fba-4ad0-b574-da148b1979db"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:54:03.386998 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:54:03.386904    5304 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e80dea97-6fba-4ad0-b574-da148b1979db-workload-socket\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:54:03.388032 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:54:03.387139    5304 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e80dea97-6fba-4ad0-b574-da148b1979db-istiod-ca-cert\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:54:03.388334 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:54:03.388223    5304 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e80dea97-6fba-4ad0-b574-da148b1979db-istio-data\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:54:03.388334 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:54:03.388275    5304 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e80dea97-6fba-4ad0-b574-da148b1979db-workload-certs\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:54:03.388334 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:54:03.388289    5304 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e80dea97-6fba-4ad0-b574-da148b1979db-credential-socket\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:54:03.388334 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:54:03.387631    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e80dea97-6fba-4ad0-b574-da148b1979db-istio-podinfo" pod "e80dea97-6fba-4ad0-b574-da148b1979db" (UID: "e80dea97-6fba-4ad0-b574-da148b1979db"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 15:54:03.389306 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:54:03.389096    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e80dea97-6fba-4ad0-b574-da148b1979db-istio-token" pod "e80dea97-6fba-4ad0-b574-da148b1979db" (UID: "e80dea97-6fba-4ad0-b574-da148b1979db"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:54:03.389485 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:54:03.389450    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e80dea97-6fba-4ad0-b574-da148b1979db-istio-envoy" pod "e80dea97-6fba-4ad0-b574-da148b1979db" (UID: "e80dea97-6fba-4ad0-b574-da148b1979db"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:54:03.389627 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:54:03.389605    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e80dea97-6fba-4ad0-b574-da148b1979db-kube-api-access-wzcdr" pod "e80dea97-6fba-4ad0-b574-da148b1979db" (UID: "e80dea97-6fba-4ad0-b574-da148b1979db"). InnerVolumeSpecName "kube-api-access-wzcdr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:54:03.489622 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:54:03.489587    5304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wzcdr\" (UniqueName: \"kubernetes.io/projected/e80dea97-6fba-4ad0-b574-da148b1979db-kube-api-access-wzcdr\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:54:03.489622 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:54:03.489616    5304 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e80dea97-6fba-4ad0-b574-da148b1979db-istio-token\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:54:03.489622 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:54:03.489625    5304 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e80dea97-6fba-4ad0-b574-da148b1979db-istio-podinfo\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:54:03.489870 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:54:03.489634    5304 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e80dea97-6fba-4ad0-b574-da148b1979db-istio-envoy\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:54:03.762216 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:54:03.762108    5304 generic.go:356] "Generic (PLEG): container finished" podID="e80dea97-6fba-4ad0-b574-da148b1979db" containerID="af24a6c1d1d4fd1dbe13f28dd9e59a5861478f779204f9941d54d12f7ec52d9c" exitCode=0
Nov 02 15:54:03.762216 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:54:03.762156    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-28da58-openshift-default-76446c9f79-rg6rm" event={"ID":"e80dea97-6fba-4ad0-b574-da148b1979db","Type":"ContainerDied","Data":"af24a6c1d1d4fd1dbe13f28dd9e59a5861478f779204f9941d54d12f7ec52d9c"}
Nov 02 15:54:03.762216 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:54:03.762184    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-28da58-openshift-default-76446c9f79-rg6rm"
Nov 02 15:54:03.762819 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:54:03.762222    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-28da58-openshift-default-76446c9f79-rg6rm" event={"ID":"e80dea97-6fba-4ad0-b574-da148b1979db","Type":"ContainerDied","Data":"8e19dc84487f1a25cd147f8dabc05fabcc2fba0d4d8c8d732a1cb6ee607f24f0"}
Nov 02 15:54:03.762819 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:54:03.762253    5304 scope.go:122] "RemoveContainer" containerID="af24a6c1d1d4fd1dbe13f28dd9e59a5861478f779204f9941d54d12f7ec52d9c"
Nov 02 15:54:03.764422 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:54:03.764403    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-f0039c-openshift-default-5f656fc965-rjf2w"
Nov 02 15:54:03.764569 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:54:03.764435    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f0039c-openshift-default-5f656fc965-rjf2w" event={"ID":"9178b9f8-6fda-40a9-8850-fa0097f2d39a","Type":"ContainerDied","Data":"7c63663b5576566f525d1dc79c17c5bd9cca7ceb738a6219a05b00c237e80aed"}
Nov 02 15:54:03.784434 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:54:03.784412    5304 scope.go:122] "RemoveContainer" containerID="af24a6c1d1d4fd1dbe13f28dd9e59a5861478f779204f9941d54d12f7ec52d9c"
Nov 02 15:54:03.784759 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:54:03.784736    5304 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af24a6c1d1d4fd1dbe13f28dd9e59a5861478f779204f9941d54d12f7ec52d9c\": container with ID starting with af24a6c1d1d4fd1dbe13f28dd9e59a5861478f779204f9941d54d12f7ec52d9c not found: ID does not exist" containerID="af24a6c1d1d4fd1dbe13f28dd9e59a5861478f779204f9941d54d12f7ec52d9c"
Nov 02 15:54:03.784864 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:54:03.784765    5304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af24a6c1d1d4fd1dbe13f28dd9e59a5861478f779204f9941d54d12f7ec52d9c"} err="failed to get container status \"af24a6c1d1d4fd1dbe13f28dd9e59a5861478f779204f9941d54d12f7ec52d9c\": rpc error: code = NotFound desc = could not find container \"af24a6c1d1d4fd1dbe13f28dd9e59a5861478f779204f9941d54d12f7ec52d9c\": container with ID starting with af24a6c1d1d4fd1dbe13f28dd9e59a5861478f779204f9941d54d12f7ec52d9c not found: ID does not exist"
Nov 02 15:54:03.784864 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:54:03.784783    5304 scope.go:122] "RemoveContainer" containerID="75004946a6197bf76c248dc08a3e0e00c4632a344e8913e3d4bb7dddfa8ec931"
Nov 02 15:54:03.826541 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:54:03.826515    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-f0039c-openshift-default-5f656fc965-rjf2w"]
Nov 02 15:54:03.841778 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:54:03.841751    5304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-f0039c-openshift-default-5f656fc965-rjf2w"]
Nov 02 15:54:03.890911 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:54:03.890877    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-28da58-openshift-default-76446c9f79-rg6rm"]
Nov 02 15:54:03.894020 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:54:03.893987    5304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-28da58-openshift-default-76446c9f79-rg6rm"]
Nov 02 15:54:04.119231 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:54:04.119197    5304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9178b9f8-6fda-40a9-8850-fa0097f2d39a" path="/var/lib/kubelet/pods/9178b9f8-6fda-40a9-8850-fa0097f2d39a/volumes"
Nov 02 15:54:04.119831 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:54:04.119810    5304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e80dea97-6fba-4ad0-b574-da148b1979db" path="/var/lib/kubelet/pods/e80dea97-6fba-4ad0-b574-da148b1979db/volumes"
Nov 02 15:54:14.050890 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:54:14.050846    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-624254/distinct-w3-624254-kserve-5465dc4765-dmq67"]
Nov 02 15:54:14.091159 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:54:14.091060    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-624254/simulator-w3-624254-kserve-56866b7699-z47sd"]
Nov 02 15:54:24.796862 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:54:24.796825    5304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k6fjk" containerName="extractor"
Nov 02 15:54:29.070917 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:54:29.070852    5304 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-624254/distinct-w3-624254-kserve-5465dc4765-dmq67" podUID="1a5aa910-a6de-417b-8638-81e4ec5c1825" containerName="main" containerID="cri-o://010a8321841e466f631db5a26ccc650099bcbda947dd780929cd5f1cb1a3c922" gracePeriod=45
Nov 02 15:54:29.110767 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:54:29.110726    5304 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-624254/simulator-w3-624254-kserve-56866b7699-z47sd" podUID="345c710f-b414-4e62-86c7-d51ff4e5be15" containerName="main" containerID="cri-o://9f9134166e5dd2fc996adf28f0f4cf600048bbae39ae4cc4cf8e9a0a8d7bd87f" gracePeriod=45
Nov 02 15:54:29.362420 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:54:29.362394    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-624254/distinct-w3-624254-kserve-5465dc4765-dmq67"
Nov 02 15:54:29.391298 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:54:29.391273    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-624254/simulator-w3-624254-kserve-56866b7699-z47sd"
Nov 02 15:54:29.549568 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:54:29.549532    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a5aa910-a6de-417b-8638-81e4ec5c1825-kube-api-access-c9r42\" (UniqueName: \"kubernetes.io/projected/1a5aa910-a6de-417b-8638-81e4ec5c1825-kube-api-access-c9r42\") pod \"1a5aa910-a6de-417b-8638-81e4ec5c1825\" (UID: \"1a5aa910-a6de-417b-8638-81e4ec5c1825\") "
Nov 02 15:54:29.549783 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:54:29.549610    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a5aa910-a6de-417b-8638-81e4ec5c1825-home\" (UniqueName: \"kubernetes.io/empty-dir/1a5aa910-a6de-417b-8638-81e4ec5c1825-home\") pod \"1a5aa910-a6de-417b-8638-81e4ec5c1825\" (UID: \"1a5aa910-a6de-417b-8638-81e4ec5c1825\") "
Nov 02 15:54:29.549783 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:54:29.549652    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a5aa910-a6de-417b-8638-81e4ec5c1825-dshm\" (UniqueName: \"kubernetes.io/empty-dir/1a5aa910-a6de-417b-8638-81e4ec5c1825-dshm\") pod \"1a5aa910-a6de-417b-8638-81e4ec5c1825\" (UID: \"1a5aa910-a6de-417b-8638-81e4ec5c1825\") "
Nov 02 15:54:29.549783 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:54:29.549679    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a5aa910-a6de-417b-8638-81e4ec5c1825-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a5aa910-a6de-417b-8638-81e4ec5c1825-tmp-dir\") pod \"1a5aa910-a6de-417b-8638-81e4ec5c1825\" (UID: \"1a5aa910-a6de-417b-8638-81e4ec5c1825\") "
Nov 02 15:54:29.549783 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:54:29.549747    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/345c710f-b414-4e62-86c7-d51ff4e5be15-kube-api-access-5rl6k\" (UniqueName: \"kubernetes.io/projected/345c710f-b414-4e62-86c7-d51ff4e5be15-kube-api-access-5rl6k\") pod \"345c710f-b414-4e62-86c7-d51ff4e5be15\" (UID: \"345c710f-b414-4e62-86c7-d51ff4e5be15\") "
Nov 02 15:54:29.550050 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:54:29.549792    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/345c710f-b414-4e62-86c7-d51ff4e5be15-home\" (UniqueName: \"kubernetes.io/empty-dir/345c710f-b414-4e62-86c7-d51ff4e5be15-home\") pod \"345c710f-b414-4e62-86c7-d51ff4e5be15\" (UID: \"345c710f-b414-4e62-86c7-d51ff4e5be15\") "
Nov 02 15:54:29.550050 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:54:29.549858    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/345c710f-b414-4e62-86c7-d51ff4e5be15-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/345c710f-b414-4e62-86c7-d51ff4e5be15-model-cache\") pod \"345c710f-b414-4e62-86c7-d51ff4e5be15\" (UID: \"345c710f-b414-4e62-86c7-d51ff4e5be15\") "
Nov 02 15:54:29.550050 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:54:29.549888    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/345c710f-b414-4e62-86c7-d51ff4e5be15-dshm\" (UniqueName: \"kubernetes.io/empty-dir/345c710f-b414-4e62-86c7-d51ff4e5be15-dshm\") pod \"345c710f-b414-4e62-86c7-d51ff4e5be15\" (UID: \"345c710f-b414-4e62-86c7-d51ff4e5be15\") "
Nov 02 15:54:29.550050 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:54:29.549942    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1a5aa910-a6de-417b-8638-81e4ec5c1825-tls-certs\" (UniqueName: \"kubernetes.io/secret/1a5aa910-a6de-417b-8638-81e4ec5c1825-tls-certs\") pod \"1a5aa910-a6de-417b-8638-81e4ec5c1825\" (UID: \"1a5aa910-a6de-417b-8638-81e4ec5c1825\") "
Nov 02 15:54:29.550050 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:54:29.549996    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/345c710f-b414-4e62-86c7-d51ff4e5be15-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/345c710f-b414-4e62-86c7-d51ff4e5be15-tmp-dir\") pod \"345c710f-b414-4e62-86c7-d51ff4e5be15\" (UID: \"345c710f-b414-4e62-86c7-d51ff4e5be15\") "
Nov 02 15:54:29.550050 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:54:29.550041    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a5aa910-a6de-417b-8638-81e4ec5c1825-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1a5aa910-a6de-417b-8638-81e4ec5c1825-model-cache\") pod \"1a5aa910-a6de-417b-8638-81e4ec5c1825\" (UID: \"1a5aa910-a6de-417b-8638-81e4ec5c1825\") "
Nov 02 15:54:29.550407 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:54:29.550112    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/345c710f-b414-4e62-86c7-d51ff4e5be15-tls-certs\" (UniqueName: \"kubernetes.io/secret/345c710f-b414-4e62-86c7-d51ff4e5be15-tls-certs\") pod \"345c710f-b414-4e62-86c7-d51ff4e5be15\" (UID: \"345c710f-b414-4e62-86c7-d51ff4e5be15\") "
Nov 02 15:54:29.550591 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:54:29.550562    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/345c710f-b414-4e62-86c7-d51ff4e5be15-home" pod "345c710f-b414-4e62-86c7-d51ff4e5be15" (UID: "345c710f-b414-4e62-86c7-d51ff4e5be15"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:54:29.551374 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:54:29.551340    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/345c710f-b414-4e62-86c7-d51ff4e5be15-model-cache" pod "345c710f-b414-4e62-86c7-d51ff4e5be15" (UID: "345c710f-b414-4e62-86c7-d51ff4e5be15"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:54:29.551637 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:54:29.551588    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a5aa910-a6de-417b-8638-81e4ec5c1825-home" pod "1a5aa910-a6de-417b-8638-81e4ec5c1825" (UID: "1a5aa910-a6de-417b-8638-81e4ec5c1825"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:54:29.552421 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:54:29.551968    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/345c710f-b414-4e62-86c7-d51ff4e5be15-tmp-dir" pod "345c710f-b414-4e62-86c7-d51ff4e5be15" (UID: "345c710f-b414-4e62-86c7-d51ff4e5be15"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:54:29.552421 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:54:29.552154    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a5aa910-a6de-417b-8638-81e4ec5c1825-model-cache" pod "1a5aa910-a6de-417b-8638-81e4ec5c1825" (UID: "1a5aa910-a6de-417b-8638-81e4ec5c1825"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:54:29.552421 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:54:29.552245    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a5aa910-a6de-417b-8638-81e4ec5c1825-tmp-dir" pod "1a5aa910-a6de-417b-8638-81e4ec5c1825" (UID: "1a5aa910-a6de-417b-8638-81e4ec5c1825"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:54:29.553408 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:54:29.553376    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/345c710f-b414-4e62-86c7-d51ff4e5be15-tls-certs" pod "345c710f-b414-4e62-86c7-d51ff4e5be15" (UID: "345c710f-b414-4e62-86c7-d51ff4e5be15"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:54:29.553519 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:54:29.553457    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a5aa910-a6de-417b-8638-81e4ec5c1825-kube-api-access-c9r42" pod "1a5aa910-a6de-417b-8638-81e4ec5c1825" (UID: "1a5aa910-a6de-417b-8638-81e4ec5c1825"). InnerVolumeSpecName "kube-api-access-c9r42". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:54:29.553625 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:54:29.553602    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/345c710f-b414-4e62-86c7-d51ff4e5be15-kube-api-access-5rl6k" pod "345c710f-b414-4e62-86c7-d51ff4e5be15" (UID: "345c710f-b414-4e62-86c7-d51ff4e5be15"). InnerVolumeSpecName "kube-api-access-5rl6k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:54:29.554004 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:54:29.553986    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1a5aa910-a6de-417b-8638-81e4ec5c1825-tls-certs" pod "1a5aa910-a6de-417b-8638-81e4ec5c1825" (UID: "1a5aa910-a6de-417b-8638-81e4ec5c1825"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:54:29.554289 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:54:29.554265    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a5aa910-a6de-417b-8638-81e4ec5c1825-dshm" pod "1a5aa910-a6de-417b-8638-81e4ec5c1825" (UID: "1a5aa910-a6de-417b-8638-81e4ec5c1825"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:54:29.554593 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:54:29.554572    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/345c710f-b414-4e62-86c7-d51ff4e5be15-dshm" pod "345c710f-b414-4e62-86c7-d51ff4e5be15" (UID: "345c710f-b414-4e62-86c7-d51ff4e5be15"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:54:29.651267 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:54:29.651185    5304 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/345c710f-b414-4e62-86c7-d51ff4e5be15-model-cache\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:54:29.651267 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:54:29.651216    5304 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/345c710f-b414-4e62-86c7-d51ff4e5be15-dshm\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:54:29.651267 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:54:29.651224    5304 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a5aa910-a6de-417b-8638-81e4ec5c1825-tls-certs\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:54:29.651267 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:54:29.651232    5304 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/345c710f-b414-4e62-86c7-d51ff4e5be15-tmp-dir\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:54:29.651267 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:54:29.651240    5304 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1a5aa910-a6de-417b-8638-81e4ec5c1825-model-cache\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:54:29.651267 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:54:29.651249    5304 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/345c710f-b414-4e62-86c7-d51ff4e5be15-tls-certs\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:54:29.651267 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:54:29.651257    5304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c9r42\" (UniqueName: \"kubernetes.io/projected/1a5aa910-a6de-417b-8638-81e4ec5c1825-kube-api-access-c9r42\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:54:29.651267 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:54:29.651266    5304 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1a5aa910-a6de-417b-8638-81e4ec5c1825-home\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:54:29.651267 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:54:29.651275    5304 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1a5aa910-a6de-417b-8638-81e4ec5c1825-dshm\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:54:29.651694 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:54:29.651283    5304 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a5aa910-a6de-417b-8638-81e4ec5c1825-tmp-dir\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:54:29.651694 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:54:29.651292    5304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5rl6k\" (UniqueName: \"kubernetes.io/projected/345c710f-b414-4e62-86c7-d51ff4e5be15-kube-api-access-5rl6k\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:54:29.651694 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:54:29.651299    5304 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/345c710f-b414-4e62-86c7-d51ff4e5be15-home\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:54:29.895294 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:54:29.895258    5304 generic.go:356] "Generic (PLEG): container finished" podID="345c710f-b414-4e62-86c7-d51ff4e5be15" containerID="9f9134166e5dd2fc996adf28f0f4cf600048bbae39ae4cc4cf8e9a0a8d7bd87f" exitCode=0
Nov 02 15:54:29.895502 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:54:29.895303    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-624254/simulator-w3-624254-kserve-56866b7699-z47sd"
Nov 02 15:54:29.895502 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:54:29.895345    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-624254/simulator-w3-624254-kserve-56866b7699-z47sd" event={"ID":"345c710f-b414-4e62-86c7-d51ff4e5be15","Type":"ContainerDied","Data":"9f9134166e5dd2fc996adf28f0f4cf600048bbae39ae4cc4cf8e9a0a8d7bd87f"}
Nov 02 15:54:29.895502 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:54:29.895392    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-624254/simulator-w3-624254-kserve-56866b7699-z47sd" event={"ID":"345c710f-b414-4e62-86c7-d51ff4e5be15","Type":"ContainerDied","Data":"7e0d992ea9e5c666ebacb9dea2ee81a51e686dc31e30fed251c58d7a190cad00"}
Nov 02 15:54:29.895502 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:54:29.895418    5304 scope.go:122] "RemoveContainer" containerID="9f9134166e5dd2fc996adf28f0f4cf600048bbae39ae4cc4cf8e9a0a8d7bd87f"
Nov 02 15:54:29.896715 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:54:29.896689    5304 generic.go:356] "Generic (PLEG): container finished" podID="1a5aa910-a6de-417b-8638-81e4ec5c1825" containerID="010a8321841e466f631db5a26ccc650099bcbda947dd780929cd5f1cb1a3c922" exitCode=0
Nov 02 15:54:29.896789 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:54:29.896720    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-624254/distinct-w3-624254-kserve-5465dc4765-dmq67" event={"ID":"1a5aa910-a6de-417b-8638-81e4ec5c1825","Type":"ContainerDied","Data":"010a8321841e466f631db5a26ccc650099bcbda947dd780929cd5f1cb1a3c922"}
Nov 02 15:54:29.896789 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:54:29.896753    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-624254/distinct-w3-624254-kserve-5465dc4765-dmq67" event={"ID":"1a5aa910-a6de-417b-8638-81e4ec5c1825","Type":"ContainerDied","Data":"d0e79cbb2555ba5258b965ab5be4703d6a0fe741d9d5d364187904dee3d76cbc"}
Nov 02 15:54:29.896789 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:54:29.896729    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-624254/distinct-w3-624254-kserve-5465dc4765-dmq67"
Nov 02 15:54:29.906300 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:54:29.906285    5304 scope.go:122] "RemoveContainer" containerID="9f9134166e5dd2fc996adf28f0f4cf600048bbae39ae4cc4cf8e9a0a8d7bd87f"
Nov 02 15:54:29.906584 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:54:29.906541    5304 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f9134166e5dd2fc996adf28f0f4cf600048bbae39ae4cc4cf8e9a0a8d7bd87f\": container with ID starting with 9f9134166e5dd2fc996adf28f0f4cf600048bbae39ae4cc4cf8e9a0a8d7bd87f not found: ID does not exist" containerID="9f9134166e5dd2fc996adf28f0f4cf600048bbae39ae4cc4cf8e9a0a8d7bd87f"
Nov 02 15:54:29.906584 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:54:29.906568    5304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f9134166e5dd2fc996adf28f0f4cf600048bbae39ae4cc4cf8e9a0a8d7bd87f"} err="failed to get container status \"9f9134166e5dd2fc996adf28f0f4cf600048bbae39ae4cc4cf8e9a0a8d7bd87f\": rpc error: code = NotFound desc = could not find container \"9f9134166e5dd2fc996adf28f0f4cf600048bbae39ae4cc4cf8e9a0a8d7bd87f\": container with ID starting with 9f9134166e5dd2fc996adf28f0f4cf600048bbae39ae4cc4cf8e9a0a8d7bd87f not found: ID does not exist"
Nov 02 15:54:29.906584 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:54:29.906585    5304 scope.go:122] "RemoveContainer" containerID="010a8321841e466f631db5a26ccc650099bcbda947dd780929cd5f1cb1a3c922"
Nov 02 15:54:29.914581 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:54:29.914559    5304 scope.go:122] "RemoveContainer" containerID="010a8321841e466f631db5a26ccc650099bcbda947dd780929cd5f1cb1a3c922"
Nov 02 15:54:29.914854 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:54:29.914837    5304 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"010a8321841e466f631db5a26ccc650099bcbda947dd780929cd5f1cb1a3c922\": container with ID starting with 010a8321841e466f631db5a26ccc650099bcbda947dd780929cd5f1cb1a3c922 not found: ID does not exist" containerID="010a8321841e466f631db5a26ccc650099bcbda947dd780929cd5f1cb1a3c922"
Nov 02 15:54:29.914912 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:54:29.914860    5304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"010a8321841e466f631db5a26ccc650099bcbda947dd780929cd5f1cb1a3c922"} err="failed to get container status \"010a8321841e466f631db5a26ccc650099bcbda947dd780929cd5f1cb1a3c922\": rpc error: code = NotFound desc = could not find container \"010a8321841e466f631db5a26ccc650099bcbda947dd780929cd5f1cb1a3c922\": container with ID starting with 010a8321841e466f631db5a26ccc650099bcbda947dd780929cd5f1cb1a3c922 not found: ID does not exist"
Nov 02 15:54:29.951668 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:54:29.951634    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-624254/distinct-w3-624254-kserve-5465dc4765-dmq67"]
Nov 02 15:54:29.955822 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:54:29.955751    5304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-624254/distinct-w3-624254-kserve-5465dc4765-dmq67"]
Nov 02 15:54:29.995092 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:54:29.995049    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-624254/simulator-w3-624254-kserve-56866b7699-z47sd"]
Nov 02 15:54:30.000053 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:54:30.000023    5304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-624254/simulator-w3-624254-kserve-56866b7699-z47sd"]
Nov 02 15:54:30.116750 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:54:30.116719    5304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1a5aa910-a6de-417b-8638-81e4ec5c1825" path="/var/lib/kubelet/pods/1a5aa910-a6de-417b-8638-81e4ec5c1825/volumes"
Nov 02 15:54:30.117160 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:54:30.117145    5304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="345c710f-b414-4e62-86c7-d51ff4e5be15" path="/var/lib/kubelet/pods/345c710f-b414-4e62-86c7-d51ff4e5be15/volumes"
Nov 02 15:54:30.328970 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:54:30.328875    5304 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-624254/distinct-w3-624254-kserve-5465dc4765-dmq67" podUID="1a5aa910-a6de-417b-8638-81e4ec5c1825" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded"
Nov 02 15:54:36.241172 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:54:36.241119    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-624254-576d847589-4jfld"]
Nov 02 15:54:36.242502 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:54:36.242468    5304 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-624254-576d847589-4jfld" podUID="2e311645-6f4b-4169-8c27-8ad75ac36da2" containerName="maas-api" containerID="cri-o://2d5898fc59bae3612aa31c443a2dac401dd48e79dd43ef4c9dc0903bf2dcae78" gracePeriod=30
Nov 02 15:54:36.508311 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:54:36.508284    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-624254-576d847589-4jfld"
Nov 02 15:54:36.520744 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:54:36.520716    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2e311645-6f4b-4169-8c27-8ad75ac36da2-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2e311645-6f4b-4169-8c27-8ad75ac36da2-maas-api-tls\") pod \"2e311645-6f4b-4169-8c27-8ad75ac36da2\" (UID: \"2e311645-6f4b-4169-8c27-8ad75ac36da2\") "
Nov 02 15:54:36.520915 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:54:36.520825    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2e311645-6f4b-4169-8c27-8ad75ac36da2-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2e311645-6f4b-4169-8c27-8ad75ac36da2-metrics-certs\") pod \"2e311645-6f4b-4169-8c27-8ad75ac36da2\" (UID: \"2e311645-6f4b-4169-8c27-8ad75ac36da2\") "
Nov 02 15:54:36.520915 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:54:36.520866    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2e311645-6f4b-4169-8c27-8ad75ac36da2-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e311645-6f4b-4169-8c27-8ad75ac36da2-openshift-service-ca\") pod \"2e311645-6f4b-4169-8c27-8ad75ac36da2\" (UID: \"2e311645-6f4b-4169-8c27-8ad75ac36da2\") "
Nov 02 15:54:36.521045 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:54:36.520950    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e311645-6f4b-4169-8c27-8ad75ac36da2-kube-api-access-jxhpr\" (UniqueName: \"kubernetes.io/projected/2e311645-6f4b-4169-8c27-8ad75ac36da2-kube-api-access-jxhpr\") pod \"2e311645-6f4b-4169-8c27-8ad75ac36da2\" (UID: \"2e311645-6f4b-4169-8c27-8ad75ac36da2\") "
Nov 02 15:54:36.521329 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:54:36.521296    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e311645-6f4b-4169-8c27-8ad75ac36da2-openshift-service-ca" pod "2e311645-6f4b-4169-8c27-8ad75ac36da2" (UID: "2e311645-6f4b-4169-8c27-8ad75ac36da2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:54:36.523060 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:54:36.523027    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e311645-6f4b-4169-8c27-8ad75ac36da2-metrics-certs" pod "2e311645-6f4b-4169-8c27-8ad75ac36da2" (UID: "2e311645-6f4b-4169-8c27-8ad75ac36da2"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:54:36.523194 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:54:36.523096    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e311645-6f4b-4169-8c27-8ad75ac36da2-kube-api-access-jxhpr" pod "2e311645-6f4b-4169-8c27-8ad75ac36da2" (UID: "2e311645-6f4b-4169-8c27-8ad75ac36da2"). InnerVolumeSpecName "kube-api-access-jxhpr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:54:36.523308 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:54:36.523285    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e311645-6f4b-4169-8c27-8ad75ac36da2-maas-api-tls" pod "2e311645-6f4b-4169-8c27-8ad75ac36da2" (UID: "2e311645-6f4b-4169-8c27-8ad75ac36da2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:54:36.622585 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:54:36.622547    5304 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2e311645-6f4b-4169-8c27-8ad75ac36da2-maas-api-tls\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:54:36.622585 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:54:36.622579    5304 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2e311645-6f4b-4169-8c27-8ad75ac36da2-metrics-certs\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:54:36.622585 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:54:36.622589    5304 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e311645-6f4b-4169-8c27-8ad75ac36da2-openshift-service-ca\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:54:36.622849 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:54:36.622598    5304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jxhpr\" (UniqueName: \"kubernetes.io/projected/2e311645-6f4b-4169-8c27-8ad75ac36da2-kube-api-access-jxhpr\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:54:36.937050 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:54:36.937009    5304 generic.go:356] "Generic (PLEG): container finished" podID="2e311645-6f4b-4169-8c27-8ad75ac36da2" containerID="2d5898fc59bae3612aa31c443a2dac401dd48e79dd43ef4c9dc0903bf2dcae78" exitCode=0
Nov 02 15:54:36.937279 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:54:36.937062    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-624254-576d847589-4jfld"
Nov 02 15:54:36.937279 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:54:36.937091    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-624254-576d847589-4jfld" event={"ID":"2e311645-6f4b-4169-8c27-8ad75ac36da2","Type":"ContainerDied","Data":"2d5898fc59bae3612aa31c443a2dac401dd48e79dd43ef4c9dc0903bf2dcae78"}
Nov 02 15:54:36.937279 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:54:36.937182    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-624254-576d847589-4jfld" event={"ID":"2e311645-6f4b-4169-8c27-8ad75ac36da2","Type":"ContainerDied","Data":"d4549008cd70092ec77708be5f081cbead433f2366d0a1f421dac2aa44fc636a"}
Nov 02 15:54:36.937279 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:54:36.937215    5304 scope.go:122] "RemoveContainer" containerID="2d5898fc59bae3612aa31c443a2dac401dd48e79dd43ef4c9dc0903bf2dcae78"
Nov 02 15:54:36.947427 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:54:36.947409    5304 scope.go:122] "RemoveContainer" containerID="2d5898fc59bae3612aa31c443a2dac401dd48e79dd43ef4c9dc0903bf2dcae78"
Nov 02 15:54:36.947739 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:54:36.947720    5304 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d5898fc59bae3612aa31c443a2dac401dd48e79dd43ef4c9dc0903bf2dcae78\": container with ID starting with 2d5898fc59bae3612aa31c443a2dac401dd48e79dd43ef4c9dc0903bf2dcae78 not found: ID does not exist" containerID="2d5898fc59bae3612aa31c443a2dac401dd48e79dd43ef4c9dc0903bf2dcae78"
Nov 02 15:54:36.947795 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:54:36.947747    5304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d5898fc59bae3612aa31c443a2dac401dd48e79dd43ef4c9dc0903bf2dcae78"} err="failed to get container status \"2d5898fc59bae3612aa31c443a2dac401dd48e79dd43ef4c9dc0903bf2dcae78\": rpc error: code = NotFound desc = could not find container \"2d5898fc59bae3612aa31c443a2dac401dd48e79dd43ef4c9dc0903bf2dcae78\": container with ID starting with 2d5898fc59bae3612aa31c443a2dac401dd48e79dd43ef4c9dc0903bf2dcae78 not found: ID does not exist"
Nov 02 15:54:37.006043 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:54:37.006005    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-624254-576d847589-4jfld"]
Nov 02 15:54:37.014881 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:54:37.014847    5304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-624254-576d847589-4jfld"]
Nov 02 15:54:38.116871 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:54:38.116837    5304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2e311645-6f4b-4169-8c27-8ad75ac36da2" path="/var/lib/kubelet/pods/2e311645-6f4b-4169-8c27-8ad75ac36da2/volumes"
Nov 02 15:54:54.798467 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:54:54.798425    5304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k6fjk" containerName="extractor"
Nov 02 15:55:16.499968 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:55:16.499925    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-cce791-openshift-default-c54fc95f9-n5z7h"]
Nov 02 15:55:16.500640 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:55:16.500192    5304 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w1-cce791-openshift-default-c54fc95f9-n5z7h" podUID="7bafb51c-17bf-4c9a-8e43-cfdf16c614d5" containerName="istio-proxy" containerID="cri-o://0a7c1bdd6e9168536c04211525a73596221c25a083a8ccdfefe4e72d055e358e" gracePeriod=30
Nov 02 15:55:21.748199 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:55:21.748172    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-cce791-openshift-default-c54fc95f9-n5z7h"
Nov 02 15:55:21.844856 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:55:21.844828    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7bafb51c-17bf-4c9a-8e43-cfdf16c614d5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bafb51c-17bf-4c9a-8e43-cfdf16c614d5-credential-socket\") pod \"7bafb51c-17bf-4c9a-8e43-cfdf16c614d5\" (UID: \"7bafb51c-17bf-4c9a-8e43-cfdf16c614d5\") "
Nov 02 15:55:21.845052 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:55:21.844865    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7bafb51c-17bf-4c9a-8e43-cfdf16c614d5-istio-token\" (UniqueName: \"kubernetes.io/projected/7bafb51c-17bf-4c9a-8e43-cfdf16c614d5-istio-token\") pod \"7bafb51c-17bf-4c9a-8e43-cfdf16c614d5\" (UID: \"7bafb51c-17bf-4c9a-8e43-cfdf16c614d5\") "
Nov 02 15:55:21.845052 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:55:21.844930    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7bafb51c-17bf-4c9a-8e43-cfdf16c614d5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7bafb51c-17bf-4c9a-8e43-cfdf16c614d5-workload-certs\") pod \"7bafb51c-17bf-4c9a-8e43-cfdf16c614d5\" (UID: \"7bafb51c-17bf-4c9a-8e43-cfdf16c614d5\") "
Nov 02 15:55:21.845052 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:55:21.844955    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7bafb51c-17bf-4c9a-8e43-cfdf16c614d5-kube-api-access-9n57j\" (UniqueName: \"kubernetes.io/projected/7bafb51c-17bf-4c9a-8e43-cfdf16c614d5-kube-api-access-9n57j\") pod \"7bafb51c-17bf-4c9a-8e43-cfdf16c614d5\" (UID: \"7bafb51c-17bf-4c9a-8e43-cfdf16c614d5\") "
Nov 02 15:55:21.845052 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:55:21.844976    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7bafb51c-17bf-4c9a-8e43-cfdf16c614d5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7bafb51c-17bf-4c9a-8e43-cfdf16c614d5-istiod-ca-cert\") pod \"7bafb51c-17bf-4c9a-8e43-cfdf16c614d5\" (UID: \"7bafb51c-17bf-4c9a-8e43-cfdf16c614d5\") "
Nov 02 15:55:21.845052 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:55:21.845004    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7bafb51c-17bf-4c9a-8e43-cfdf16c614d5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7bafb51c-17bf-4c9a-8e43-cfdf16c614d5-istio-data\") pod \"7bafb51c-17bf-4c9a-8e43-cfdf16c614d5\" (UID: \"7bafb51c-17bf-4c9a-8e43-cfdf16c614d5\") "
Nov 02 15:55:21.845376 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:55:21.845058    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7bafb51c-17bf-4c9a-8e43-cfdf16c614d5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7bafb51c-17bf-4c9a-8e43-cfdf16c614d5-istio-podinfo\") pod \"7bafb51c-17bf-4c9a-8e43-cfdf16c614d5\" (UID: \"7bafb51c-17bf-4c9a-8e43-cfdf16c614d5\") "
Nov 02 15:55:21.845376 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:55:21.845087    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7bafb51c-17bf-4c9a-8e43-cfdf16c614d5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bafb51c-17bf-4c9a-8e43-cfdf16c614d5-workload-socket\") pod \"7bafb51c-17bf-4c9a-8e43-cfdf16c614d5\" (UID: \"7bafb51c-17bf-4c9a-8e43-cfdf16c614d5\") "
Nov 02 15:55:21.845376 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:55:21.845157    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bafb51c-17bf-4c9a-8e43-cfdf16c614d5-credential-socket" pod "7bafb51c-17bf-4c9a-8e43-cfdf16c614d5" (UID: "7bafb51c-17bf-4c9a-8e43-cfdf16c614d5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:55:21.845376 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:55:21.845182    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7bafb51c-17bf-4c9a-8e43-cfdf16c614d5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7bafb51c-17bf-4c9a-8e43-cfdf16c614d5-istio-envoy\") pod \"7bafb51c-17bf-4c9a-8e43-cfdf16c614d5\" (UID: \"7bafb51c-17bf-4c9a-8e43-cfdf16c614d5\") "
Nov 02 15:55:21.845376 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:55:21.845328    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bafb51c-17bf-4c9a-8e43-cfdf16c614d5-workload-certs" pod "7bafb51c-17bf-4c9a-8e43-cfdf16c614d5" (UID: "7bafb51c-17bf-4c9a-8e43-cfdf16c614d5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:55:21.845702 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:55:21.845457    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bafb51c-17bf-4c9a-8e43-cfdf16c614d5-istio-data" pod "7bafb51c-17bf-4c9a-8e43-cfdf16c614d5" (UID: "7bafb51c-17bf-4c9a-8e43-cfdf16c614d5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:55:21.845702 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:55:21.845496    5304 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7bafb51c-17bf-4c9a-8e43-cfdf16c614d5-workload-certs\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:55:21.845702 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:55:21.845516    5304 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bafb51c-17bf-4c9a-8e43-cfdf16c614d5-credential-socket\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:55:21.845702 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:55:21.845493    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7bafb51c-17bf-4c9a-8e43-cfdf16c614d5-istiod-ca-cert" pod "7bafb51c-17bf-4c9a-8e43-cfdf16c614d5" (UID: "7bafb51c-17bf-4c9a-8e43-cfdf16c614d5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:55:21.845702 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:55:21.845651    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bafb51c-17bf-4c9a-8e43-cfdf16c614d5-workload-socket" pod "7bafb51c-17bf-4c9a-8e43-cfdf16c614d5" (UID: "7bafb51c-17bf-4c9a-8e43-cfdf16c614d5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:55:21.847303 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:55:21.847283    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7bafb51c-17bf-4c9a-8e43-cfdf16c614d5-istio-podinfo" pod "7bafb51c-17bf-4c9a-8e43-cfdf16c614d5" (UID: "7bafb51c-17bf-4c9a-8e43-cfdf16c614d5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 15:55:21.847432 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:55:21.847414    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bafb51c-17bf-4c9a-8e43-cfdf16c614d5-istio-envoy" pod "7bafb51c-17bf-4c9a-8e43-cfdf16c614d5" (UID: "7bafb51c-17bf-4c9a-8e43-cfdf16c614d5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:55:21.847671 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:55:21.847654    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7bafb51c-17bf-4c9a-8e43-cfdf16c614d5-istio-token" pod "7bafb51c-17bf-4c9a-8e43-cfdf16c614d5" (UID: "7bafb51c-17bf-4c9a-8e43-cfdf16c614d5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:55:21.847728 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:55:21.847667    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7bafb51c-17bf-4c9a-8e43-cfdf16c614d5-kube-api-access-9n57j" pod "7bafb51c-17bf-4c9a-8e43-cfdf16c614d5" (UID: "7bafb51c-17bf-4c9a-8e43-cfdf16c614d5"). InnerVolumeSpecName "kube-api-access-9n57j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:55:21.946949 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:55:21.946909    5304 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7bafb51c-17bf-4c9a-8e43-cfdf16c614d5-istio-data\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:55:21.946949 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:55:21.946942    5304 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7bafb51c-17bf-4c9a-8e43-cfdf16c614d5-istio-podinfo\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:55:21.946949 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:55:21.946951    5304 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bafb51c-17bf-4c9a-8e43-cfdf16c614d5-workload-socket\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:55:21.946949 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:55:21.946960    5304 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7bafb51c-17bf-4c9a-8e43-cfdf16c614d5-istio-envoy\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:55:21.947300 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:55:21.946971    5304 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7bafb51c-17bf-4c9a-8e43-cfdf16c614d5-istio-token\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:55:21.947300 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:55:21.946985    5304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9n57j\" (UniqueName: \"kubernetes.io/projected/7bafb51c-17bf-4c9a-8e43-cfdf16c614d5-kube-api-access-9n57j\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:55:21.947300 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:55:21.946994    5304 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7bafb51c-17bf-4c9a-8e43-cfdf16c614d5-istiod-ca-cert\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:55:22.047952 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:55:22.047914    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-5dd8e9/premium-w1-5dd8e9-kserve-67588456fb-j6lph"]
Nov 02 15:55:22.056584 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:55:22.056550    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-5dd8e9/simulator-w1-5dd8e9-kserve-6f9f755d6f-gnw5d"]
Nov 02 15:55:22.134636 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:55:22.134605    5304 generic.go:356] "Generic (PLEG): container finished" podID="7bafb51c-17bf-4c9a-8e43-cfdf16c614d5" containerID="0a7c1bdd6e9168536c04211525a73596221c25a083a8ccdfefe4e72d055e358e" exitCode=0
Nov 02 15:55:22.134813 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:55:22.134652    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-cce791-openshift-default-c54fc95f9-n5z7h"
Nov 02 15:55:22.134813 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:55:22.134692    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-cce791-openshift-default-c54fc95f9-n5z7h" event={"ID":"7bafb51c-17bf-4c9a-8e43-cfdf16c614d5","Type":"ContainerDied","Data":"0a7c1bdd6e9168536c04211525a73596221c25a083a8ccdfefe4e72d055e358e"}
Nov 02 15:55:22.134813 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:55:22.134751    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-cce791-openshift-default-c54fc95f9-n5z7h" event={"ID":"7bafb51c-17bf-4c9a-8e43-cfdf16c614d5","Type":"ContainerDied","Data":"412cf8eefe3ae791c211b2925c174b01bd44a120553c221b8e70b297e7bddc3f"}
Nov 02 15:55:22.134813 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:55:22.134782    5304 scope.go:122] "RemoveContainer" containerID="0a7c1bdd6e9168536c04211525a73596221c25a083a8ccdfefe4e72d055e358e"
Nov 02 15:55:22.144619 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:55:22.144600    5304 scope.go:122] "RemoveContainer" containerID="0a7c1bdd6e9168536c04211525a73596221c25a083a8ccdfefe4e72d055e358e"
Nov 02 15:55:22.144895 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:55:22.144874    5304 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a7c1bdd6e9168536c04211525a73596221c25a083a8ccdfefe4e72d055e358e\": container with ID starting with 0a7c1bdd6e9168536c04211525a73596221c25a083a8ccdfefe4e72d055e358e not found: ID does not exist" containerID="0a7c1bdd6e9168536c04211525a73596221c25a083a8ccdfefe4e72d055e358e"
Nov 02 15:55:22.144948 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:55:22.144904    5304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a7c1bdd6e9168536c04211525a73596221c25a083a8ccdfefe4e72d055e358e"} err="failed to get container status \"0a7c1bdd6e9168536c04211525a73596221c25a083a8ccdfefe4e72d055e358e\": rpc error: code = NotFound desc = could not find container \"0a7c1bdd6e9168536c04211525a73596221c25a083a8ccdfefe4e72d055e358e\": container with ID starting with 0a7c1bdd6e9168536c04211525a73596221c25a083a8ccdfefe4e72d055e358e not found: ID does not exist"
Nov 02 15:55:22.196041 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:55:22.196001    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-cce791-openshift-default-c54fc95f9-n5z7h"]
Nov 02 15:55:22.198385 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:55:22.198360    5304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-cce791-openshift-default-c54fc95f9-n5z7h"]
Nov 02 15:55:24.116543 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:55:24.116509    5304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7bafb51c-17bf-4c9a-8e43-cfdf16c614d5" path="/var/lib/kubelet/pods/7bafb51c-17bf-4c9a-8e43-cfdf16c614d5/volumes"
Nov 02 15:55:24.796867 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:55:24.796822    5304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k6fjk" containerName="extractor"
Nov 02 15:55:37.068818 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:55:37.068761    5304 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-5dd8e9/premium-w1-5dd8e9-kserve-67588456fb-j6lph" podUID="4ccc607a-14e4-43ca-8999-78100d4b799f" containerName="main" containerID="cri-o://fccc8e8f461aa26eee58ae4e1b570d520d7a89e01d72c90fbcb923c77a067303" gracePeriod=45
Nov 02 15:55:37.081974 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:55:37.081925    5304 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-5dd8e9/simulator-w1-5dd8e9-kserve-6f9f755d6f-gnw5d" podUID="66b75e64-cb24-40be-9076-4984c132e208" containerName="main" containerID="cri-o://b4b6aedcdb640eb49cd010d0c3dbddcb760257857b2386f94572d9a21537ea2c" gracePeriod=45
Nov 02 15:55:37.213853 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:55:37.213801    5304 generic.go:356] "Generic (PLEG): container finished" podID="66b75e64-cb24-40be-9076-4984c132e208" containerID="b4b6aedcdb640eb49cd010d0c3dbddcb760257857b2386f94572d9a21537ea2c" exitCode=0
Nov 02 15:55:37.214031 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:55:37.213895    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-5dd8e9/simulator-w1-5dd8e9-kserve-6f9f755d6f-gnw5d" event={"ID":"66b75e64-cb24-40be-9076-4984c132e208","Type":"ContainerDied","Data":"b4b6aedcdb640eb49cd010d0c3dbddcb760257857b2386f94572d9a21537ea2c"}
Nov 02 15:55:37.215409 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:55:37.215385    5304 generic.go:356] "Generic (PLEG): container finished" podID="4ccc607a-14e4-43ca-8999-78100d4b799f" containerID="fccc8e8f461aa26eee58ae4e1b570d520d7a89e01d72c90fbcb923c77a067303" exitCode=0
Nov 02 15:55:37.215509 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:55:37.215432    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-5dd8e9/premium-w1-5dd8e9-kserve-67588456fb-j6lph" event={"ID":"4ccc607a-14e4-43ca-8999-78100d4b799f","Type":"ContainerDied","Data":"fccc8e8f461aa26eee58ae4e1b570d520d7a89e01d72c90fbcb923c77a067303"}
Nov 02 15:55:37.325522 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:55:37.325457    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-5dd8e9/premium-w1-5dd8e9-kserve-67588456fb-j6lph"
Nov 02 15:55:37.352054 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:55:37.352025    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-5dd8e9/simulator-w1-5dd8e9-kserve-6f9f755d6f-gnw5d"
Nov 02 15:55:37.395454 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:55:37.395418    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4ccc607a-14e4-43ca-8999-78100d4b799f-tls-certs\" (UniqueName: \"kubernetes.io/secret/4ccc607a-14e4-43ca-8999-78100d4b799f-tls-certs\") pod \"4ccc607a-14e4-43ca-8999-78100d4b799f\" (UID: \"4ccc607a-14e4-43ca-8999-78100d4b799f\") "
Nov 02 15:55:37.395671 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:55:37.395516    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ccc607a-14e4-43ca-8999-78100d4b799f-home\" (UniqueName: \"kubernetes.io/empty-dir/4ccc607a-14e4-43ca-8999-78100d4b799f-home\") pod \"4ccc607a-14e4-43ca-8999-78100d4b799f\" (UID: \"4ccc607a-14e4-43ca-8999-78100d4b799f\") "
Nov 02 15:55:37.395671 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:55:37.395562    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ccc607a-14e4-43ca-8999-78100d4b799f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4ccc607a-14e4-43ca-8999-78100d4b799f-model-cache\") pod \"4ccc607a-14e4-43ca-8999-78100d4b799f\" (UID: \"4ccc607a-14e4-43ca-8999-78100d4b799f\") "
Nov 02 15:55:37.395671 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:55:37.395614    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ccc607a-14e4-43ca-8999-78100d4b799f-kube-api-access-kjhfg\" (UniqueName: \"kubernetes.io/projected/4ccc607a-14e4-43ca-8999-78100d4b799f-kube-api-access-kjhfg\") pod \"4ccc607a-14e4-43ca-8999-78100d4b799f\" (UID: \"4ccc607a-14e4-43ca-8999-78100d4b799f\") "
Nov 02 15:55:37.395671 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:55:37.395642    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ccc607a-14e4-43ca-8999-78100d4b799f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/4ccc607a-14e4-43ca-8999-78100d4b799f-dshm\") pod \"4ccc607a-14e4-43ca-8999-78100d4b799f\" (UID: \"4ccc607a-14e4-43ca-8999-78100d4b799f\") "
Nov 02 15:55:37.395960 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:55:37.395703    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ccc607a-14e4-43ca-8999-78100d4b799f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4ccc607a-14e4-43ca-8999-78100d4b799f-tmp-dir\") pod \"4ccc607a-14e4-43ca-8999-78100d4b799f\" (UID: \"4ccc607a-14e4-43ca-8999-78100d4b799f\") "
Nov 02 15:55:37.395960 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:55:37.395846    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ccc607a-14e4-43ca-8999-78100d4b799f-home" pod "4ccc607a-14e4-43ca-8999-78100d4b799f" (UID: "4ccc607a-14e4-43ca-8999-78100d4b799f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:55:37.396098 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:55:37.396071    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ccc607a-14e4-43ca-8999-78100d4b799f-model-cache" pod "4ccc607a-14e4-43ca-8999-78100d4b799f" (UID: "4ccc607a-14e4-43ca-8999-78100d4b799f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:55:37.396206 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:55:37.396095    5304 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4ccc607a-14e4-43ca-8999-78100d4b799f-home\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:55:37.396206 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:55:37.396093    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ccc607a-14e4-43ca-8999-78100d4b799f-tmp-dir" pod "4ccc607a-14e4-43ca-8999-78100d4b799f" (UID: "4ccc607a-14e4-43ca-8999-78100d4b799f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:55:37.398259 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:55:37.398222    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ccc607a-14e4-43ca-8999-78100d4b799f-dshm" pod "4ccc607a-14e4-43ca-8999-78100d4b799f" (UID: "4ccc607a-14e4-43ca-8999-78100d4b799f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:55:37.398364 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:55:37.398230    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ccc607a-14e4-43ca-8999-78100d4b799f-kube-api-access-kjhfg" pod "4ccc607a-14e4-43ca-8999-78100d4b799f" (UID: "4ccc607a-14e4-43ca-8999-78100d4b799f"). InnerVolumeSpecName "kube-api-access-kjhfg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:55:37.398364 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:55:37.398275    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4ccc607a-14e4-43ca-8999-78100d4b799f-tls-certs" pod "4ccc607a-14e4-43ca-8999-78100d4b799f" (UID: "4ccc607a-14e4-43ca-8999-78100d4b799f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:55:37.496910 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:55:37.496879    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66b75e64-cb24-40be-9076-4984c132e208-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/66b75e64-cb24-40be-9076-4984c132e208-tmp-dir\") pod \"66b75e64-cb24-40be-9076-4984c132e208\" (UID: \"66b75e64-cb24-40be-9076-4984c132e208\") "
Nov 02 15:55:37.497074 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:55:37.496925    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66b75e64-cb24-40be-9076-4984c132e208-home\" (UniqueName: \"kubernetes.io/empty-dir/66b75e64-cb24-40be-9076-4984c132e208-home\") pod \"66b75e64-cb24-40be-9076-4984c132e208\" (UID: \"66b75e64-cb24-40be-9076-4984c132e208\") "
Nov 02 15:55:37.497074 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:55:37.496986    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66b75e64-cb24-40be-9076-4984c132e208-kube-api-access-br92q\" (UniqueName: \"kubernetes.io/projected/66b75e64-cb24-40be-9076-4984c132e208-kube-api-access-br92q\") pod \"66b75e64-cb24-40be-9076-4984c132e208\" (UID: \"66b75e64-cb24-40be-9076-4984c132e208\") "
Nov 02 15:55:37.497074 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:55:37.497022    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66b75e64-cb24-40be-9076-4984c132e208-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/66b75e64-cb24-40be-9076-4984c132e208-model-cache\") pod \"66b75e64-cb24-40be-9076-4984c132e208\" (UID: \"66b75e64-cb24-40be-9076-4984c132e208\") "
Nov 02 15:55:37.497074 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:55:37.497064    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/66b75e64-cb24-40be-9076-4984c132e208-tls-certs\" (UniqueName: \"kubernetes.io/secret/66b75e64-cb24-40be-9076-4984c132e208-tls-certs\") pod \"66b75e64-cb24-40be-9076-4984c132e208\" (UID: \"66b75e64-cb24-40be-9076-4984c132e208\") "
Nov 02 15:55:37.497349 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:55:37.497108    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66b75e64-cb24-40be-9076-4984c132e208-dshm\" (UniqueName: \"kubernetes.io/empty-dir/66b75e64-cb24-40be-9076-4984c132e208-dshm\") pod \"66b75e64-cb24-40be-9076-4984c132e208\" (UID: \"66b75e64-cb24-40be-9076-4984c132e208\") "
Nov 02 15:55:37.497349 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:55:37.497199    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66b75e64-cb24-40be-9076-4984c132e208-tmp-dir" pod "66b75e64-cb24-40be-9076-4984c132e208" (UID: "66b75e64-cb24-40be-9076-4984c132e208"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:55:37.497349 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:55:37.497248    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66b75e64-cb24-40be-9076-4984c132e208-home" pod "66b75e64-cb24-40be-9076-4984c132e208" (UID: "66b75e64-cb24-40be-9076-4984c132e208"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:55:37.497349 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:55:37.497334    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66b75e64-cb24-40be-9076-4984c132e208-model-cache" pod "66b75e64-cb24-40be-9076-4984c132e208" (UID: "66b75e64-cb24-40be-9076-4984c132e208"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:55:37.497538 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:55:37.497390    5304 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4ccc607a-14e4-43ca-8999-78100d4b799f-model-cache\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:55:37.497538 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:55:37.497405    5304 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/66b75e64-cb24-40be-9076-4984c132e208-tmp-dir\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:55:37.497538 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:55:37.497415    5304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kjhfg\" (UniqueName: \"kubernetes.io/projected/4ccc607a-14e4-43ca-8999-78100d4b799f-kube-api-access-kjhfg\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:55:37.497538 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:55:37.497424    5304 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4ccc607a-14e4-43ca-8999-78100d4b799f-dshm\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:55:37.497538 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:55:37.497433    5304 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/66b75e64-cb24-40be-9076-4984c132e208-home\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:55:37.497538 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:55:37.497440    5304 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4ccc607a-14e4-43ca-8999-78100d4b799f-tmp-dir\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:55:37.497538 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:55:37.497449    5304 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4ccc607a-14e4-43ca-8999-78100d4b799f-tls-certs\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:55:37.499099 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:55:37.499079    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66b75e64-cb24-40be-9076-4984c132e208-dshm" pod "66b75e64-cb24-40be-9076-4984c132e208" (UID: "66b75e64-cb24-40be-9076-4984c132e208"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:55:37.499202 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:55:37.499179    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66b75e64-cb24-40be-9076-4984c132e208-tls-certs" pod "66b75e64-cb24-40be-9076-4984c132e208" (UID: "66b75e64-cb24-40be-9076-4984c132e208"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:55:37.499251 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:55:37.499175    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66b75e64-cb24-40be-9076-4984c132e208-kube-api-access-br92q" pod "66b75e64-cb24-40be-9076-4984c132e208" (UID: "66b75e64-cb24-40be-9076-4984c132e208"). InnerVolumeSpecName "kube-api-access-br92q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:55:37.598768 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:55:37.598733    5304 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/66b75e64-cb24-40be-9076-4984c132e208-dshm\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:55:37.598768 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:55:37.598766    5304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-br92q\" (UniqueName: \"kubernetes.io/projected/66b75e64-cb24-40be-9076-4984c132e208-kube-api-access-br92q\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:55:37.598978 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:55:37.598776    5304 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/66b75e64-cb24-40be-9076-4984c132e208-model-cache\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:55:37.598978 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:55:37.598785    5304 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/66b75e64-cb24-40be-9076-4984c132e208-tls-certs\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:55:38.221164 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:55:38.221103    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-5dd8e9/simulator-w1-5dd8e9-kserve-6f9f755d6f-gnw5d" event={"ID":"66b75e64-cb24-40be-9076-4984c132e208","Type":"ContainerDied","Data":"27ac563a147095630ec60ec88f6fc5d9de9ef241d07a24c9812e1df4ed350cba"}
Nov 02 15:55:38.221633 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:55:38.221114    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-5dd8e9/simulator-w1-5dd8e9-kserve-6f9f755d6f-gnw5d"
Nov 02 15:55:38.221633 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:55:38.221183    5304 scope.go:122] "RemoveContainer" containerID="b4b6aedcdb640eb49cd010d0c3dbddcb760257857b2386f94572d9a21537ea2c"
Nov 02 15:55:38.222683 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:55:38.222501    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-5dd8e9/premium-w1-5dd8e9-kserve-67588456fb-j6lph" event={"ID":"4ccc607a-14e4-43ca-8999-78100d4b799f","Type":"ContainerDied","Data":"5243f93f225cbb4fdfd9e33c57626288296521ec42cce8e931db744d20252823"}
Nov 02 15:55:38.222683 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:55:38.222544    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-5dd8e9/premium-w1-5dd8e9-kserve-67588456fb-j6lph"
Nov 02 15:55:38.230608 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:55:38.230576    5304 scope.go:122] "RemoveContainer" containerID="fccc8e8f461aa26eee58ae4e1b570d520d7a89e01d72c90fbcb923c77a067303"
Nov 02 15:55:38.272341 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:55:38.272295    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-5dd8e9/simulator-w1-5dd8e9-kserve-6f9f755d6f-gnw5d"]
Nov 02 15:55:38.277320 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:55:38.277282    5304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-5dd8e9/simulator-w1-5dd8e9-kserve-6f9f755d6f-gnw5d"]
Nov 02 15:55:38.311542 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:55:38.311508    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-5dd8e9/premium-w1-5dd8e9-kserve-67588456fb-j6lph"]
Nov 02 15:55:38.314423 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:55:38.314393    5304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-5dd8e9/premium-w1-5dd8e9-kserve-67588456fb-j6lph"]
Nov 02 15:55:40.117502 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:55:40.117465    5304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4ccc607a-14e4-43ca-8999-78100d4b799f" path="/var/lib/kubelet/pods/4ccc607a-14e4-43ca-8999-78100d4b799f/volumes"
Nov 02 15:55:40.117884 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:55:40.117862    5304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="66b75e64-cb24-40be-9076-4984c132e208" path="/var/lib/kubelet/pods/66b75e64-cb24-40be-9076-4984c132e208/volumes"
Nov 02 15:55:43.268601 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:55:43.268563    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-5dd8e9-68fff466d7-7dgm5"]
Nov 02 15:55:43.269116 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:55:43.268813    5304 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-5dd8e9-68fff466d7-7dgm5" podUID="f5ef30f3-a503-42ee-a484-caf583967d39" containerName="maas-api" containerID="cri-o://e67722e309f649301fc7d99b2b5087cd346d4a3f9ed6b37b31aa90db878b5dc2" gracePeriod=30
Nov 02 15:55:43.522833 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:55:43.522771    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-5dd8e9-68fff466d7-7dgm5"
Nov 02 15:55:43.650272 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:55:43.650237    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f5ef30f3-a503-42ee-a484-caf583967d39-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f5ef30f3-a503-42ee-a484-caf583967d39-metrics-certs\") pod \"f5ef30f3-a503-42ee-a484-caf583967d39\" (UID: \"f5ef30f3-a503-42ee-a484-caf583967d39\") "
Nov 02 15:55:43.650553 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:55:43.650337    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f5ef30f3-a503-42ee-a484-caf583967d39-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f5ef30f3-a503-42ee-a484-caf583967d39-maas-api-tls\") pod \"f5ef30f3-a503-42ee-a484-caf583967d39\" (UID: \"f5ef30f3-a503-42ee-a484-caf583967d39\") "
Nov 02 15:55:43.650553 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:55:43.650368    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5ef30f3-a503-42ee-a484-caf583967d39-kube-api-access-kh64b\" (UniqueName: \"kubernetes.io/projected/f5ef30f3-a503-42ee-a484-caf583967d39-kube-api-access-kh64b\") pod \"f5ef30f3-a503-42ee-a484-caf583967d39\" (UID: \"f5ef30f3-a503-42ee-a484-caf583967d39\") "
Nov 02 15:55:43.650553 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:55:43.650449    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f5ef30f3-a503-42ee-a484-caf583967d39-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5ef30f3-a503-42ee-a484-caf583967d39-openshift-service-ca\") pod \"f5ef30f3-a503-42ee-a484-caf583967d39\" (UID: \"f5ef30f3-a503-42ee-a484-caf583967d39\") "
Nov 02 15:55:43.650860 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:55:43.650832    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5ef30f3-a503-42ee-a484-caf583967d39-openshift-service-ca" pod "f5ef30f3-a503-42ee-a484-caf583967d39" (UID: "f5ef30f3-a503-42ee-a484-caf583967d39"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:55:43.652478 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:55:43.652454    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f5ef30f3-a503-42ee-a484-caf583967d39-metrics-certs" pod "f5ef30f3-a503-42ee-a484-caf583967d39" (UID: "f5ef30f3-a503-42ee-a484-caf583967d39"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:55:43.652575 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:55:43.652472    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f5ef30f3-a503-42ee-a484-caf583967d39-maas-api-tls" pod "f5ef30f3-a503-42ee-a484-caf583967d39" (UID: "f5ef30f3-a503-42ee-a484-caf583967d39"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:55:43.652575 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:55:43.652492    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5ef30f3-a503-42ee-a484-caf583967d39-kube-api-access-kh64b" pod "f5ef30f3-a503-42ee-a484-caf583967d39" (UID: "f5ef30f3-a503-42ee-a484-caf583967d39"). InnerVolumeSpecName "kube-api-access-kh64b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:55:43.751715 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:55:43.751684    5304 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f5ef30f3-a503-42ee-a484-caf583967d39-maas-api-tls\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:55:43.751715 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:55:43.751713    5304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kh64b\" (UniqueName: \"kubernetes.io/projected/f5ef30f3-a503-42ee-a484-caf583967d39-kube-api-access-kh64b\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:55:43.751715 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:55:43.751723    5304 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5ef30f3-a503-42ee-a484-caf583967d39-openshift-service-ca\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:55:43.751968 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:55:43.751733    5304 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f5ef30f3-a503-42ee-a484-caf583967d39-metrics-certs\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 15:55:44.252448 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:55:44.252406    5304 generic.go:356] "Generic (PLEG): container finished" podID="f5ef30f3-a503-42ee-a484-caf583967d39" containerID="e67722e309f649301fc7d99b2b5087cd346d4a3f9ed6b37b31aa90db878b5dc2" exitCode=0
Nov 02 15:55:44.252643 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:55:44.252461    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-5dd8e9-68fff466d7-7dgm5"
Nov 02 15:55:44.252643 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:55:44.252456    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-5dd8e9-68fff466d7-7dgm5" event={"ID":"f5ef30f3-a503-42ee-a484-caf583967d39","Type":"ContainerDied","Data":"e67722e309f649301fc7d99b2b5087cd346d4a3f9ed6b37b31aa90db878b5dc2"}
Nov 02 15:55:44.252643 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:55:44.252584    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-5dd8e9-68fff466d7-7dgm5" event={"ID":"f5ef30f3-a503-42ee-a484-caf583967d39","Type":"ContainerDied","Data":"32c5f15f5496e2d0861e2df17e31b1019d0c34db2a02f363d15868ee92b6e9cb"}
Nov 02 15:55:44.252643 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:55:44.252611    5304 scope.go:122] "RemoveContainer" containerID="e67722e309f649301fc7d99b2b5087cd346d4a3f9ed6b37b31aa90db878b5dc2"
Nov 02 15:55:44.262454 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:55:44.262432    5304 scope.go:122] "RemoveContainer" containerID="e67722e309f649301fc7d99b2b5087cd346d4a3f9ed6b37b31aa90db878b5dc2"
Nov 02 15:55:44.262765 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:55:44.262740    5304 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e67722e309f649301fc7d99b2b5087cd346d4a3f9ed6b37b31aa90db878b5dc2\": container with ID starting with e67722e309f649301fc7d99b2b5087cd346d4a3f9ed6b37b31aa90db878b5dc2 not found: ID does not exist" containerID="e67722e309f649301fc7d99b2b5087cd346d4a3f9ed6b37b31aa90db878b5dc2"
Nov 02 15:55:44.262839 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:55:44.262774    5304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e67722e309f649301fc7d99b2b5087cd346d4a3f9ed6b37b31aa90db878b5dc2"} err="failed to get container status \"e67722e309f649301fc7d99b2b5087cd346d4a3f9ed6b37b31aa90db878b5dc2\": rpc error: code = NotFound desc = could not find container \"e67722e309f649301fc7d99b2b5087cd346d4a3f9ed6b37b31aa90db878b5dc2\": container with ID starting with e67722e309f649301fc7d99b2b5087cd346d4a3f9ed6b37b31aa90db878b5dc2 not found: ID does not exist"
Nov 02 15:55:44.307066 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:55:44.307022    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-5dd8e9-68fff466d7-7dgm5"]
Nov 02 15:55:44.318893 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:55:44.318855    5304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-5dd8e9-68fff466d7-7dgm5"]
Nov 02 15:55:46.116770 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:55:46.116729    5304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f5ef30f3-a503-42ee-a484-caf583967d39" path="/var/lib/kubelet/pods/f5ef30f3-a503-42ee-a484-caf583967d39/volumes"
Nov 02 15:55:54.797646 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:55:54.797610    5304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k6fjk" containerName="extractor"
Nov 02 15:56:24.798291 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:56:24.798246    5304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k6fjk" containerName="extractor"
Nov 02 15:56:28.879494 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:56:28.879459    5304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-srlvv"]
Nov 02 15:56:28.880072 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:56:28.880008    5304 cpu_manager.go:414] "containerMap: removing container" podUID="2e311645-6f4b-4169-8c27-8ad75ac36da2" containerName="maas-api"
Nov 02 15:56:28.880072 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:56:28.880031    5304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e311645-6f4b-4169-8c27-8ad75ac36da2" containerName="maas-api"
Nov 02 15:56:28.880072 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:56:28.880041    5304 cpu_manager.go:414] "containerMap: removing container" podUID="ab7216bc-cf46-4280-acd1-3a15d1ead1c0" containerName="maas-api"
Nov 02 15:56:28.880072 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:56:28.880047    5304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab7216bc-cf46-4280-acd1-3a15d1ead1c0" containerName="maas-api"
Nov 02 15:56:28.880072 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:56:28.880057    5304 cpu_manager.go:414] "containerMap: removing container" podUID="3dca5bf3-8e08-47a0-936e-205670d59ede" containerName="main"
Nov 02 15:56:28.880072 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:56:28.880062    5304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3dca5bf3-8e08-47a0-936e-205670d59ede" containerName="main"
Nov 02 15:56:28.880390 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:56:28.880078    5304 cpu_manager.go:414] "containerMap: removing container" podUID="4ccc607a-14e4-43ca-8999-78100d4b799f" containerName="main"
Nov 02 15:56:28.880390 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:56:28.880086    5304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ccc607a-14e4-43ca-8999-78100d4b799f" containerName="main"
Nov 02 15:56:28.880390 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:56:28.880095    5304 cpu_manager.go:414] "containerMap: removing container" podUID="d35d4668-8376-487a-9c91-8b6801eb9cc6" containerName="payload-pre-processing"
Nov 02 15:56:28.880390 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:56:28.880101    5304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d35d4668-8376-487a-9c91-8b6801eb9cc6" containerName="payload-pre-processing"
Nov 02 15:56:28.880390 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:56:28.880112    5304 cpu_manager.go:414] "containerMap: removing container" podUID="9746bf8a-91bd-4715-a02e-ae2e651ca54d" containerName="main"
Nov 02 15:56:28.880390 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:56:28.880120    5304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9746bf8a-91bd-4715-a02e-ae2e651ca54d" containerName="main"
Nov 02 15:56:28.880390 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:56:28.880149    5304 cpu_manager.go:414] "containerMap: removing container" podUID="66b75e64-cb24-40be-9076-4984c132e208" containerName="main"
Nov 02 15:56:28.880390 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:56:28.880156    5304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66b75e64-cb24-40be-9076-4984c132e208" containerName="main"
Nov 02 15:56:28.880390 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:56:28.880166    5304 cpu_manager.go:414] "containerMap: removing container" podUID="345c710f-b414-4e62-86c7-d51ff4e5be15" containerName="main"
Nov 02 15:56:28.880390 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:56:28.880173    5304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="345c710f-b414-4e62-86c7-d51ff4e5be15" containerName="main"
Nov 02 15:56:28.880390 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:56:28.880188    5304 cpu_manager.go:414] "containerMap: removing container" podUID="0c18605a-898a-41ca-ba2d-e7f48597e790" containerName="payload-processing"
Nov 02 15:56:28.880390 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:56:28.880195    5304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c18605a-898a-41ca-ba2d-e7f48597e790" containerName="payload-processing"
Nov 02 15:56:28.880390 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:56:28.880205    5304 cpu_manager.go:414] "containerMap: removing container" podUID="73b5a23d-1e75-4beb-920e-8953826899af" containerName="istio-proxy"
Nov 02 15:56:28.880390 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:56:28.880212    5304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73b5a23d-1e75-4beb-920e-8953826899af" containerName="istio-proxy"
Nov 02 15:56:28.880390 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:56:28.880225    5304 cpu_manager.go:414] "containerMap: removing container" podUID="9c90c9e1-7216-4ffb-b513-f5ca7fec318a" containerName="payload-processing"
Nov 02 15:56:28.880390 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:56:28.880232    5304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c90c9e1-7216-4ffb-b513-f5ca7fec318a" containerName="payload-processing"
Nov 02 15:56:28.880390 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:56:28.880243    5304 cpu_manager.go:414] "containerMap: removing container" podUID="43c5260b-b089-4c06-ac76-e6e170cac754" containerName="payload-processing"
Nov 02 15:56:28.880390 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:56:28.880250    5304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43c5260b-b089-4c06-ac76-e6e170cac754" containerName="payload-processing"
Nov 02 15:56:28.880390 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:56:28.880258    5304 cpu_manager.go:414] "containerMap: removing container" podUID="7bafb51c-17bf-4c9a-8e43-cfdf16c614d5" containerName="istio-proxy"
Nov 02 15:56:28.880390 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:56:28.880266    5304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7bafb51c-17bf-4c9a-8e43-cfdf16c614d5" containerName="istio-proxy"
Nov 02 15:56:28.880390 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:56:28.880276    5304 cpu_manager.go:414] "containerMap: removing container" podUID="fe452842-bf9b-4129-a1dc-4a5b98d2672f" containerName="payload-processing"
Nov 02 15:56:28.880390 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:56:28.880283    5304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe452842-bf9b-4129-a1dc-4a5b98d2672f" containerName="payload-processing"
Nov 02 15:56:28.880390 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:56:28.880292    5304 cpu_manager.go:414] "containerMap: removing container" podUID="cf6a1ded-28a3-49eb-878b-b4ad77a0bb5c" containerName="main"
Nov 02 15:56:28.880390 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:56:28.880298    5304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf6a1ded-28a3-49eb-878b-b4ad77a0bb5c" containerName="main"
Nov 02 15:56:28.880390 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:56:28.880308    5304 cpu_manager.go:414] "containerMap: removing container" podUID="39e13479-f61d-4215-bbb3-03aa8b678618" containerName="maas-api"
Nov 02 15:56:28.880390 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:56:28.880315    5304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="39e13479-f61d-4215-bbb3-03aa8b678618" containerName="maas-api"
Nov 02 15:56:28.880390 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:56:28.880323    5304 cpu_manager.go:414] "containerMap: removing container" podUID="e80dea97-6fba-4ad0-b574-da148b1979db" containerName="istio-proxy"
Nov 02 15:56:28.880390 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:56:28.880332    5304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e80dea97-6fba-4ad0-b574-da148b1979db" containerName="istio-proxy"
Nov 02 15:56:28.880390 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:56:28.880343    5304 cpu_manager.go:414] "containerMap: removing container" podUID="1285186b-0b54-48b3-84c6-b640966bb312" containerName="istio-proxy"
Nov 02 15:56:28.880390 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:56:28.880350    5304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1285186b-0b54-48b3-84c6-b640966bb312" containerName="istio-proxy"
Nov 02 15:56:28.880390 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:56:28.880358    5304 cpu_manager.go:414] "containerMap: removing container" podUID="69df020e-436a-4ae9-b07b-b138bb2ff1da" containerName="payload-pre-processing"
Nov 02 15:56:28.880390 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:56:28.880366    5304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="69df020e-436a-4ae9-b07b-b138bb2ff1da" containerName="payload-pre-processing"
Nov 02 15:56:28.880390 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:56:28.880380    5304 cpu_manager.go:414] "containerMap: removing container" podUID="d7ab8a29-4410-4fde-8975-49c13b3ff1fb" containerName="istio-proxy"
Nov 02 15:56:28.880390 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:56:28.880386    5304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7ab8a29-4410-4fde-8975-49c13b3ff1fb" containerName="istio-proxy"
Nov 02 15:56:28.880390 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:56:28.880395    5304 cpu_manager.go:414] "containerMap: removing container" podUID="ecdcaba8-df41-4c1b-a442-d03f9994a5e4" containerName="istio-proxy"
Nov 02 15:56:28.880390 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:56:28.880403    5304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ecdcaba8-df41-4c1b-a442-d03f9994a5e4" containerName="istio-proxy"
Nov 02 15:56:28.881638 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:56:28.880412    5304 cpu_manager.go:414] "containerMap: removing container" podUID="1a5aa910-a6de-417b-8638-81e4ec5c1825" containerName="main"
Nov 02 15:56:28.881638 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:56:28.880418    5304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a5aa910-a6de-417b-8638-81e4ec5c1825" containerName="main"
Nov 02 15:56:28.881638 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:56:28.880426    5304 cpu_manager.go:414] "containerMap: removing container" podUID="9819202d-bd76-4f6f-b077-59ec79c0c50a" containerName="istio-proxy"
Nov 02 15:56:28.881638 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:56:28.880431    5304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9819202d-bd76-4f6f-b077-59ec79c0c50a" containerName="istio-proxy"
Nov 02 15:56:28.881638 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:56:28.880436    5304 cpu_manager.go:414] "containerMap: removing container" podUID="9178b9f8-6fda-40a9-8850-fa0097f2d39a" containerName="istio-proxy"
Nov 02 15:56:28.881638 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:56:28.880441    5304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9178b9f8-6fda-40a9-8850-fa0097f2d39a" containerName="istio-proxy"
Nov 02 15:56:28.881638 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:56:28.880448    5304 cpu_manager.go:414] "containerMap: removing container" podUID="e04e8ce8-dec7-47b8-9eba-a8f87a539790" containerName="payload-pre-processing"
Nov 02 15:56:28.881638 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:56:28.880454    5304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e04e8ce8-dec7-47b8-9eba-a8f87a539790" containerName="payload-pre-processing"
Nov 02 15:56:28.881638 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:56:28.880461    5304 cpu_manager.go:414] "containerMap: removing container" podUID="d7ab8a29-4410-4fde-8975-49c13b3ff1fb" containerName="istio-proxy"
Nov 02 15:56:28.881638 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:56:28.880466    5304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7ab8a29-4410-4fde-8975-49c13b3ff1fb" containerName="istio-proxy"
Nov 02 15:56:28.881638 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:56:28.880473    5304 cpu_manager.go:414] "containerMap: removing container" podUID="f5ef30f3-a503-42ee-a484-caf583967d39" containerName="maas-api"
Nov 02 15:56:28.881638 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:56:28.880477    5304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f5ef30f3-a503-42ee-a484-caf583967d39" containerName="maas-api"
Nov 02 15:56:28.881638 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:56:28.880484    5304 cpu_manager.go:414] "containerMap: removing container" podUID="55aea2b2-3109-4d02-a3d5-c65835f4d6bf" containerName="istio-proxy"
Nov 02 15:56:28.881638 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:56:28.880489    5304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="55aea2b2-3109-4d02-a3d5-c65835f4d6bf" containerName="istio-proxy"
Nov 02 15:56:28.881638 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:56:28.880496    5304 cpu_manager.go:414] "containerMap: removing container" podUID="bdab310f-f084-4e60-9b19-3fd5e0b6a953" containerName="istio-proxy"
Nov 02 15:56:28.881638 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:56:28.880509    5304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bdab310f-f084-4e60-9b19-3fd5e0b6a953" containerName="istio-proxy"
Nov 02 15:56:28.881638 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:56:28.880517    5304 cpu_manager.go:414] "containerMap: removing container" podUID="210f620a-8ae7-4e4c-acc3-46f75ddc6105" containerName="main"
Nov 02 15:56:28.881638 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:56:28.880521    5304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="210f620a-8ae7-4e4c-acc3-46f75ddc6105" containerName="main"
Nov 02 15:56:28.881638 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:56:28.880530    5304 cpu_manager.go:414] "containerMap: removing container" podUID="9178b9f8-6fda-40a9-8850-fa0097f2d39a" containerName="istio-proxy"
Nov 02 15:56:28.881638 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:56:28.880535    5304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9178b9f8-6fda-40a9-8850-fa0097f2d39a" containerName="istio-proxy"
Nov 02 15:56:28.881638 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:56:28.880541    5304 cpu_manager.go:414] "containerMap: removing container" podUID="73b5a23d-1e75-4beb-920e-8953826899af" containerName="istio-proxy"
Nov 02 15:56:28.881638 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:56:28.880545    5304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73b5a23d-1e75-4beb-920e-8953826899af" containerName="istio-proxy"
Nov 02 15:56:28.881638 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:56:28.880551    5304 cpu_manager.go:414] "containerMap: removing container" podUID="55aea2b2-3109-4d02-a3d5-c65835f4d6bf" containerName="istio-proxy"
Nov 02 15:56:28.881638 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:56:28.880555    5304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="55aea2b2-3109-4d02-a3d5-c65835f4d6bf" containerName="istio-proxy"
Nov 02 15:56:28.881638 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:56:28.880567    5304 cpu_manager.go:414] "containerMap: removing container" podUID="0fc51d17-635f-4f36-90d6-f2c546a457ba" containerName="payload-processing"
Nov 02 15:56:28.881638 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:56:28.880571    5304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0fc51d17-635f-4f36-90d6-f2c546a457ba" containerName="payload-processing"
Nov 02 15:56:28.881638 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:56:28.880577    5304 cpu_manager.go:414] "containerMap: removing container" podUID="1020808c-e760-46e7-99e8-66932fca1956" containerName="istio-proxy"
Nov 02 15:56:28.881638 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:56:28.880581    5304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1020808c-e760-46e7-99e8-66932fca1956" containerName="istio-proxy"
Nov 02 15:56:28.881638 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:56:28.880590    5304 cpu_manager.go:414] "containerMap: removing container" podUID="1020808c-e760-46e7-99e8-66932fca1956" containerName="istio-proxy"
Nov 02 15:56:28.881638 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:56:28.880594    5304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1020808c-e760-46e7-99e8-66932fca1956" containerName="istio-proxy"
Nov 02 15:56:28.881638 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:56:28.880600    5304 cpu_manager.go:414] "containerMap: removing container" podUID="9819202d-bd76-4f6f-b077-59ec79c0c50a" containerName="istio-proxy"
Nov 02 15:56:28.881638 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:56:28.880604    5304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9819202d-bd76-4f6f-b077-59ec79c0c50a" containerName="istio-proxy"
Nov 02 15:56:28.881638 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:56:28.880612    5304 cpu_manager.go:414] "containerMap: removing container" podUID="99313de5-a08c-4544-86ec-54aeedfe4ec2" containerName="payload-processing"
Nov 02 15:56:28.881638 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:56:28.880617    5304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="99313de5-a08c-4544-86ec-54aeedfe4ec2" containerName="payload-processing"
Nov 02 15:56:28.881638 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:56:28.880624    5304 cpu_manager.go:414] "containerMap: removing container" podUID="211ad15d-9234-4f57-b1b9-181dd2d74208" containerName="main"
Nov 02 15:56:28.881638 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:56:28.880629    5304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="211ad15d-9234-4f57-b1b9-181dd2d74208" containerName="main"
Nov 02 15:56:28.881638 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:56:28.880699    5304 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7ab8a29-4410-4fde-8975-49c13b3ff1fb" containerName="istio-proxy"
Nov 02 15:56:28.881638 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:56:28.880706    5304 memory_manager.go:365] "RemoveStaleState removing state" podUID="ecdcaba8-df41-4c1b-a442-d03f9994a5e4" containerName="istio-proxy"
Nov 02 15:56:28.881638 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:56:28.880714    5304 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a5aa910-a6de-417b-8638-81e4ec5c1825" containerName="main"
Nov 02 15:56:28.881638 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:56:28.880721    5304 memory_manager.go:365] "RemoveStaleState removing state" podUID="9819202d-bd76-4f6f-b077-59ec79c0c50a" containerName="istio-proxy"
Nov 02 15:56:28.881638 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:56:28.880727    5304 memory_manager.go:365] "RemoveStaleState removing state" podUID="9178b9f8-6fda-40a9-8850-fa0097f2d39a" containerName="istio-proxy"
Nov 02 15:56:28.881638 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:56:28.880732    5304 memory_manager.go:365] "RemoveStaleState removing state" podUID="e04e8ce8-dec7-47b8-9eba-a8f87a539790" containerName="payload-pre-processing"
Nov 02 15:56:28.881638 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:56:28.880739    5304 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7ab8a29-4410-4fde-8975-49c13b3ff1fb" containerName="istio-proxy"
Nov 02 15:56:28.881638 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:56:28.880745    5304 memory_manager.go:365] "RemoveStaleState removing state" podUID="f5ef30f3-a503-42ee-a484-caf583967d39" containerName="maas-api"
Nov 02 15:56:28.881638 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:56:28.880751    5304 memory_manager.go:365] "RemoveStaleState removing state" podUID="55aea2b2-3109-4d02-a3d5-c65835f4d6bf" containerName="istio-proxy"
Nov 02 15:56:28.881638 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:56:28.880759    5304 memory_manager.go:365] "RemoveStaleState removing state" podUID="bdab310f-f084-4e60-9b19-3fd5e0b6a953" containerName="istio-proxy"
Nov 02 15:56:28.881638 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:56:28.880767    5304 memory_manager.go:365] "RemoveStaleState removing state" podUID="210f620a-8ae7-4e4c-acc3-46f75ddc6105" containerName="main"
Nov 02 15:56:28.881638 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:56:28.880775    5304 memory_manager.go:365] "RemoveStaleState removing state" podUID="9178b9f8-6fda-40a9-8850-fa0097f2d39a" containerName="istio-proxy"
Nov 02 15:56:28.881638 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:56:28.880784    5304 memory_manager.go:365] "RemoveStaleState removing state" podUID="73b5a23d-1e75-4beb-920e-8953826899af" containerName="istio-proxy"
Nov 02 15:56:28.881638 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:56:28.880791    5304 memory_manager.go:365] "RemoveStaleState removing state" podUID="55aea2b2-3109-4d02-a3d5-c65835f4d6bf" containerName="istio-proxy"
Nov 02 15:56:28.881638 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:56:28.880800    5304 memory_manager.go:365] "RemoveStaleState removing state" podUID="0fc51d17-635f-4f36-90d6-f2c546a457ba" containerName="payload-processing"
Nov 02 15:56:28.881638 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:56:28.880808    5304 memory_manager.go:365] "RemoveStaleState removing state" podUID="1285186b-0b54-48b3-84c6-b640966bb312" containerName="istio-proxy"
Nov 02 15:56:28.881638 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:56:28.880816    5304 memory_manager.go:365] "RemoveStaleState removing state" podUID="1020808c-e760-46e7-99e8-66932fca1956" containerName="istio-proxy"
Nov 02 15:56:28.881638 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:56:28.880824    5304 memory_manager.go:365] "RemoveStaleState removing state" podUID="1020808c-e760-46e7-99e8-66932fca1956" containerName="istio-proxy"
Nov 02 15:56:28.881638 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:56:28.880832    5304 memory_manager.go:365] "RemoveStaleState removing state" podUID="9819202d-bd76-4f6f-b077-59ec79c0c50a" containerName="istio-proxy"
Nov 02 15:56:28.881638 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:56:28.880842    5304 memory_manager.go:365] "RemoveStaleState removing state" podUID="99313de5-a08c-4544-86ec-54aeedfe4ec2" containerName="payload-processing"
Nov 02 15:56:28.881638 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:56:28.880849    5304 memory_manager.go:365] "RemoveStaleState removing state" podUID="211ad15d-9234-4f57-b1b9-181dd2d74208" containerName="main"
Nov 02 15:56:28.881638 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:56:28.880858    5304 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e311645-6f4b-4169-8c27-8ad75ac36da2" containerName="maas-api"
Nov 02 15:56:28.881638 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:56:28.880866    5304 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab7216bc-cf46-4280-acd1-3a15d1ead1c0" containerName="maas-api"
Nov 02 15:56:28.881638 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:56:28.880875    5304 memory_manager.go:365] "RemoveStaleState removing state" podUID="3dca5bf3-8e08-47a0-936e-205670d59ede" containerName="main"
Nov 02 15:56:28.881638 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:56:28.880884    5304 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ccc607a-14e4-43ca-8999-78100d4b799f" containerName="main"
Nov 02 15:56:28.881638 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:56:28.880893    5304 memory_manager.go:365] "RemoveStaleState removing state" podUID="d35d4668-8376-487a-9c91-8b6801eb9cc6" containerName="payload-pre-processing"
Nov 02 15:56:28.881638 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:56:28.880902    5304 memory_manager.go:365] "RemoveStaleState removing state" podUID="9746bf8a-91bd-4715-a02e-ae2e651ca54d" containerName="main"
Nov 02 15:56:28.881638 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:56:28.880911    5304 memory_manager.go:365] "RemoveStaleState removing state" podUID="66b75e64-cb24-40be-9076-4984c132e208" containerName="main"
Nov 02 15:56:28.881638 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:56:28.880919    5304 memory_manager.go:365] "RemoveStaleState removing state" podUID="345c710f-b414-4e62-86c7-d51ff4e5be15" containerName="main"
Nov 02 15:56:28.881638 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:56:28.880926    5304 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c18605a-898a-41ca-ba2d-e7f48597e790" containerName="payload-processing"
Nov 02 15:56:28.881638 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:56:28.880935    5304 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c90c9e1-7216-4ffb-b513-f5ca7fec318a" containerName="payload-processing"
Nov 02 15:56:28.881638 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:56:28.880943    5304 memory_manager.go:365] "RemoveStaleState removing state" podUID="43c5260b-b089-4c06-ac76-e6e170cac754" containerName="payload-processing"
Nov 02 15:56:28.881638 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:56:28.880949    5304 memory_manager.go:365] "RemoveStaleState removing state" podUID="7bafb51c-17bf-4c9a-8e43-cfdf16c614d5" containerName="istio-proxy"
Nov 02 15:56:28.881638 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:56:28.880954    5304 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe452842-bf9b-4129-a1dc-4a5b98d2672f" containerName="payload-processing"
Nov 02 15:56:28.881638 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:56:28.880959    5304 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf6a1ded-28a3-49eb-878b-b4ad77a0bb5c" containerName="main"
Nov 02 15:56:28.881638 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:56:28.880963    5304 memory_manager.go:365] "RemoveStaleState removing state" podUID="39e13479-f61d-4215-bbb3-03aa8b678618" containerName="maas-api"
Nov 02 15:56:28.881638 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:56:28.880968    5304 memory_manager.go:365] "RemoveStaleState removing state" podUID="e80dea97-6fba-4ad0-b574-da148b1979db" containerName="istio-proxy"
Nov 02 15:56:28.881638 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:56:28.880976    5304 memory_manager.go:365] "RemoveStaleState removing state" podUID="69df020e-436a-4ae9-b07b-b138bb2ff1da" containerName="payload-pre-processing"
Nov 02 15:56:28.884291 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:56:28.884037    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-srlvv"
Nov 02 15:56:28.899099 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:56:28.899070    5304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-pwrm4\""
Nov 02 15:56:28.912292 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:56:28.912264    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-srlvv"]
Nov 02 15:56:28.953727 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:56:28.953693    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98cb3e76-0b44-497d-aaf7-4ee08e76d1b1-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-srlvv\" (UID: \"98cb3e76-0b44-497d-aaf7-4ee08e76d1b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-srlvv"
Nov 02 15:56:28.953888 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:56:28.953743    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98cb3e76-0b44-497d-aaf7-4ee08e76d1b1-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-srlvv\" (UID: \"98cb3e76-0b44-497d-aaf7-4ee08e76d1b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-srlvv"
Nov 02 15:56:28.953888 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:56:28.953819    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98cb3e76-0b44-497d-aaf7-4ee08e76d1b1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-srlvv\" (UID: \"98cb3e76-0b44-497d-aaf7-4ee08e76d1b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-srlvv"
Nov 02 15:56:28.953888 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:56:28.953880    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kx85x\" (UniqueName: \"kubernetes.io/projected/98cb3e76-0b44-497d-aaf7-4ee08e76d1b1-kube-api-access-kx85x\") pod \"maas-default-gateway-openshift-default-fcb75c45f-srlvv\" (UID: \"98cb3e76-0b44-497d-aaf7-4ee08e76d1b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-srlvv"
Nov 02 15:56:28.954076 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:56:28.953991    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98cb3e76-0b44-497d-aaf7-4ee08e76d1b1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-srlvv\" (UID: \"98cb3e76-0b44-497d-aaf7-4ee08e76d1b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-srlvv"
Nov 02 15:56:28.954076 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:56:28.954023    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98cb3e76-0b44-497d-aaf7-4ee08e76d1b1-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-srlvv\" (UID: \"98cb3e76-0b44-497d-aaf7-4ee08e76d1b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-srlvv"
Nov 02 15:56:28.954230 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:56:28.954072    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98cb3e76-0b44-497d-aaf7-4ee08e76d1b1-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-srlvv\" (UID: \"98cb3e76-0b44-497d-aaf7-4ee08e76d1b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-srlvv"
Nov 02 15:56:28.954230 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:56:28.954143    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98cb3e76-0b44-497d-aaf7-4ee08e76d1b1-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-srlvv\" (UID: \"98cb3e76-0b44-497d-aaf7-4ee08e76d1b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-srlvv"
Nov 02 15:56:28.954230 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:56:28.954191    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/98cb3e76-0b44-497d-aaf7-4ee08e76d1b1-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-srlvv\" (UID: \"98cb3e76-0b44-497d-aaf7-4ee08e76d1b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-srlvv"
Nov 02 15:56:29.054730 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:56:29.054695    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98cb3e76-0b44-497d-aaf7-4ee08e76d1b1-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-srlvv\" (UID: \"98cb3e76-0b44-497d-aaf7-4ee08e76d1b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-srlvv"
Nov 02 15:56:29.054730 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:56:29.054732    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98cb3e76-0b44-497d-aaf7-4ee08e76d1b1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-srlvv\" (UID: \"98cb3e76-0b44-497d-aaf7-4ee08e76d1b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-srlvv"
Nov 02 15:56:29.054947 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:56:29.054857    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kx85x\" (UniqueName: \"kubernetes.io/projected/98cb3e76-0b44-497d-aaf7-4ee08e76d1b1-kube-api-access-kx85x\") pod \"maas-default-gateway-openshift-default-fcb75c45f-srlvv\" (UID: \"98cb3e76-0b44-497d-aaf7-4ee08e76d1b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-srlvv"
Nov 02 15:56:29.054947 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:56:29.054909    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98cb3e76-0b44-497d-aaf7-4ee08e76d1b1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-srlvv\" (UID: \"98cb3e76-0b44-497d-aaf7-4ee08e76d1b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-srlvv"
Nov 02 15:56:29.055050 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:56:29.055026    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98cb3e76-0b44-497d-aaf7-4ee08e76d1b1-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-srlvv\" (UID: \"98cb3e76-0b44-497d-aaf7-4ee08e76d1b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-srlvv"
Nov 02 15:56:29.055115 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:56:29.055069    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98cb3e76-0b44-497d-aaf7-4ee08e76d1b1-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-srlvv\" (UID: \"98cb3e76-0b44-497d-aaf7-4ee08e76d1b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-srlvv"
Nov 02 15:56:29.055115 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:56:29.055109    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98cb3e76-0b44-497d-aaf7-4ee08e76d1b1-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-srlvv\" (UID: \"98cb3e76-0b44-497d-aaf7-4ee08e76d1b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-srlvv"
Nov 02 15:56:29.055257 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:56:29.055171    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98cb3e76-0b44-497d-aaf7-4ee08e76d1b1-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-srlvv\" (UID: \"98cb3e76-0b44-497d-aaf7-4ee08e76d1b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-srlvv"
Nov 02 15:56:29.055257 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:56:29.055201    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/98cb3e76-0b44-497d-aaf7-4ee08e76d1b1-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-srlvv\" (UID: \"98cb3e76-0b44-497d-aaf7-4ee08e76d1b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-srlvv"
Nov 02 15:56:29.055257 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:56:29.055241    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98cb3e76-0b44-497d-aaf7-4ee08e76d1b1-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-srlvv\" (UID: \"98cb3e76-0b44-497d-aaf7-4ee08e76d1b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-srlvv"
Nov 02 15:56:29.055423 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:56:29.055389    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98cb3e76-0b44-497d-aaf7-4ee08e76d1b1-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-srlvv\" (UID: \"98cb3e76-0b44-497d-aaf7-4ee08e76d1b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-srlvv"
Nov 02 15:56:29.055514 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:56:29.055489    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98cb3e76-0b44-497d-aaf7-4ee08e76d1b1-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-srlvv\" (UID: \"98cb3e76-0b44-497d-aaf7-4ee08e76d1b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-srlvv"
Nov 02 15:56:29.055557 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:56:29.055535    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98cb3e76-0b44-497d-aaf7-4ee08e76d1b1-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-srlvv\" (UID: \"98cb3e76-0b44-497d-aaf7-4ee08e76d1b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-srlvv"
Nov 02 15:56:29.055593 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:56:29.055575    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98cb3e76-0b44-497d-aaf7-4ee08e76d1b1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-srlvv\" (UID: \"98cb3e76-0b44-497d-aaf7-4ee08e76d1b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-srlvv"
Nov 02 15:56:29.057411 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:56:29.057384    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98cb3e76-0b44-497d-aaf7-4ee08e76d1b1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-srlvv\" (UID: \"98cb3e76-0b44-497d-aaf7-4ee08e76d1b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-srlvv"
Nov 02 15:56:29.057542 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:56:29.057467    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98cb3e76-0b44-497d-aaf7-4ee08e76d1b1-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-srlvv\" (UID: \"98cb3e76-0b44-497d-aaf7-4ee08e76d1b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-srlvv"
Nov 02 15:56:29.078141 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:56:29.078090    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/98cb3e76-0b44-497d-aaf7-4ee08e76d1b1-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-srlvv\" (UID: \"98cb3e76-0b44-497d-aaf7-4ee08e76d1b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-srlvv"
Nov 02 15:56:29.078306 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:56:29.078289    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kx85x\" (UniqueName: \"kubernetes.io/projected/98cb3e76-0b44-497d-aaf7-4ee08e76d1b1-kube-api-access-kx85x\") pod \"maas-default-gateway-openshift-default-fcb75c45f-srlvv\" (UID: \"98cb3e76-0b44-497d-aaf7-4ee08e76d1b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-srlvv"
Nov 02 15:56:29.196760 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:56:29.196682    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-srlvv"
Nov 02 15:56:29.510541 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:56:29.510436    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-srlvv"]
Nov 02 15:56:29.515361 ip-10-0-135-138 kubenswrapper[5304]: W1102 15:56:29.515333    5304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98cb3e76_0b44_497d_aaf7_4ee08e76d1b1.slice/crio-122e22a49f6829b0dcae257aea83ef35e124847274da43cd933b8135b1fdaa15 WatchSource:0}: Error finding container 122e22a49f6829b0dcae257aea83ef35e124847274da43cd933b8135b1fdaa15: Status 404 returned error can't find the container with id 122e22a49f6829b0dcae257aea83ef35e124847274da43cd933b8135b1fdaa15
Nov 02 15:56:29.517516 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:56:29.517481    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:56:29.517593 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:56:29.517552    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:56:29.517593 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:56:29.517587    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:56:30.457573 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:56:30.457533    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-srlvv" event={"ID":"98cb3e76-0b44-497d-aaf7-4ee08e76d1b1","Type":"ContainerStarted","Data":"35ad48df4625f027be005a047e611a319e93fbc7506dd4d76214ba46f6f434d5"}
Nov 02 15:56:30.458063 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:56:30.457611    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-srlvv" event={"ID":"98cb3e76-0b44-497d-aaf7-4ee08e76d1b1","Type":"ContainerStarted","Data":"122e22a49f6829b0dcae257aea83ef35e124847274da43cd933b8135b1fdaa15"}
Nov 02 15:56:30.529544 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:56:30.529495    5304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-srlvv" podStartSLOduration=2.529482749 podStartE2EDuration="2.529482749s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:56:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:56:30.529023995 +0000 UTC m=+3590.522848415" watchObservedRunningTime="2026-11-02 15:56:30.529482749 +0000 UTC m=+3590.523307155"
Nov 02 15:56:31.197609 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:56:31.197569    5304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-srlvv"
Nov 02 15:56:31.199153 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:56:31.199120    5304 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-srlvv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.222:15021/healthz/ready\": dial tcp 10.133.0.222:15021: connect: connection refused" start-of-body=
Nov 02 15:56:31.199232 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:56:31.199180    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-srlvv" podUID="98cb3e76-0b44-497d-aaf7-4ee08e76d1b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.222:15021/healthz/ready\": dial tcp 10.133.0.222:15021: connect: connection refused"
Nov 02 15:56:32.197136 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:56:32.197100    5304 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-srlvv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.222:15021/healthz/ready\": dial tcp 10.133.0.222:15021: connect: connection refused" start-of-body=
Nov 02 15:56:32.197547 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:56:32.197162    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-srlvv" podUID="98cb3e76-0b44-497d-aaf7-4ee08e76d1b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.222:15021/healthz/ready\": dial tcp 10.133.0.222:15021: connect: connection refused"
Nov 02 15:56:33.197739 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:56:33.197702    5304 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-srlvv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.222:15021/healthz/ready\": dial tcp 10.133.0.222:15021: connect: connection refused" start-of-body=
Nov 02 15:56:33.198185 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:56:33.197752    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-srlvv" podUID="98cb3e76-0b44-497d-aaf7-4ee08e76d1b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.222:15021/healthz/ready\": dial tcp 10.133.0.222:15021: connect: connection refused"
Nov 02 15:56:34.198064 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:56:34.198026    5304 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-srlvv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.222:15021/healthz/ready\": dial tcp 10.133.0.222:15021: connect: connection refused" start-of-body=
Nov 02 15:56:34.198506 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:56:34.198083    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-srlvv" podUID="98cb3e76-0b44-497d-aaf7-4ee08e76d1b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.222:15021/healthz/ready\": dial tcp 10.133.0.222:15021: connect: connection refused"
Nov 02 15:56:35.197757 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:56:35.197717    5304 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-srlvv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.222:15021/healthz/ready\": dial tcp 10.133.0.222:15021: connect: connection refused" start-of-body=
Nov 02 15:56:35.197976 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:56:35.197773    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-srlvv" podUID="98cb3e76-0b44-497d-aaf7-4ee08e76d1b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.222:15021/healthz/ready\": dial tcp 10.133.0.222:15021: connect: connection refused"
Nov 02 15:56:36.198141 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:56:36.198100    5304 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-srlvv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.222:15021/healthz/ready\": dial tcp 10.133.0.222:15021: connect: connection refused" start-of-body=
Nov 02 15:56:36.198561 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:56:36.198177    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-srlvv" podUID="98cb3e76-0b44-497d-aaf7-4ee08e76d1b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.222:15021/healthz/ready\": dial tcp 10.133.0.222:15021: connect: connection refused"
Nov 02 15:56:37.904654 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:56:37.904616    5304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-srlvv"
Nov 02 15:56:37.905191 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:56:37.905023    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-srlvv"
Nov 02 15:56:37.905985 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:56:37.905963    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-srlvv"
Nov 02 15:56:43.390019 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:56:43.389977    5304 scope.go:122] "RemoveContainer" containerID="9d0983157d56b071b0f8f59c7711c0c35e654feb059fa2e2b5150c445b1b25b5"
Nov 02 15:56:54.797143 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:56:54.797047    5304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k6fjk" containerName="extractor"
Nov 02 15:57:24.799081 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:57:24.799035    5304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k6fjk" containerName="extractor"
Nov 02 15:57:29.052136 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:57:29.052083    5304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2lc6p"]
Nov 02 15:57:29.053075 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:57:29.053051    5304 cpu_manager.go:414] "containerMap: removing container" podUID="1285186b-0b54-48b3-84c6-b640966bb312" containerName="istio-proxy"
Nov 02 15:57:29.053234 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:57:29.053076    5304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1285186b-0b54-48b3-84c6-b640966bb312" containerName="istio-proxy"
Nov 02 15:57:29.053234 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:57:29.053224    5304 memory_manager.go:365] "RemoveStaleState removing state" podUID="73b5a23d-1e75-4beb-920e-8953826899af" containerName="istio-proxy"
Nov 02 15:57:29.053362 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:57:29.053241    5304 memory_manager.go:365] "RemoveStaleState removing state" podUID="1285186b-0b54-48b3-84c6-b640966bb312" containerName="istio-proxy"
Nov 02 15:57:29.056043 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:57:29.056025    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2lc6p"
Nov 02 15:57:29.091473 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:57:29.091450    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2lc6p"]
Nov 02 15:57:29.096026 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:57:29.096002    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e24e2d68-b448-4c4f-99c6-4effdd9992e4-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-2lc6p\" (UID: \"e24e2d68-b448-4c4f-99c6-4effdd9992e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2lc6p"
Nov 02 15:57:29.096177 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:57:29.096040    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e24e2d68-b448-4c4f-99c6-4effdd9992e4-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-2lc6p\" (UID: \"e24e2d68-b448-4c4f-99c6-4effdd9992e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2lc6p"
Nov 02 15:57:29.096221 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:57:29.096181    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pt64s\" (UniqueName: \"kubernetes.io/projected/e24e2d68-b448-4c4f-99c6-4effdd9992e4-kube-api-access-pt64s\") pod \"maas-default-gateway-openshift-default-fcb75c45f-2lc6p\" (UID: \"e24e2d68-b448-4c4f-99c6-4effdd9992e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2lc6p"
Nov 02 15:57:29.096221 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:57:29.096209    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e24e2d68-b448-4c4f-99c6-4effdd9992e4-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-2lc6p\" (UID: \"e24e2d68-b448-4c4f-99c6-4effdd9992e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2lc6p"
Nov 02 15:57:29.096320 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:57:29.096238    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e24e2d68-b448-4c4f-99c6-4effdd9992e4-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-2lc6p\" (UID: \"e24e2d68-b448-4c4f-99c6-4effdd9992e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2lc6p"
Nov 02 15:57:29.096617 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:57:29.096394    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e24e2d68-b448-4c4f-99c6-4effdd9992e4-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-2lc6p\" (UID: \"e24e2d68-b448-4c4f-99c6-4effdd9992e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2lc6p"
Nov 02 15:57:29.096617 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:57:29.096434    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e24e2d68-b448-4c4f-99c6-4effdd9992e4-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-2lc6p\" (UID: \"e24e2d68-b448-4c4f-99c6-4effdd9992e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2lc6p"
Nov 02 15:57:29.096617 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:57:29.096470    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e24e2d68-b448-4c4f-99c6-4effdd9992e4-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-2lc6p\" (UID: \"e24e2d68-b448-4c4f-99c6-4effdd9992e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2lc6p"
Nov 02 15:57:29.096617 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:57:29.096516    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e24e2d68-b448-4c4f-99c6-4effdd9992e4-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-2lc6p\" (UID: \"e24e2d68-b448-4c4f-99c6-4effdd9992e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2lc6p"
Nov 02 15:57:29.105897 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:57:29.105849    5304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-j6lh6"]
Nov 02 15:57:29.109819 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:57:29.109797    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-j6lh6"
Nov 02 15:57:29.136558 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:57:29.136511    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-j6lh6"]
Nov 02 15:57:29.197204 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:57:29.197166    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a2f560c-e476-4c3a-b96d-1d3e7b313360-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-j6lh6\" (UID: \"9a2f560c-e476-4c3a-b96d-1d3e7b313360\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-j6lh6"
Nov 02 15:57:29.197402 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:57:29.197214    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pt64s\" (UniqueName: \"kubernetes.io/projected/e24e2d68-b448-4c4f-99c6-4effdd9992e4-kube-api-access-pt64s\") pod \"maas-default-gateway-openshift-default-fcb75c45f-2lc6p\" (UID: \"e24e2d68-b448-4c4f-99c6-4effdd9992e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2lc6p"
Nov 02 15:57:29.197402 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:57:29.197324    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e24e2d68-b448-4c4f-99c6-4effdd9992e4-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-2lc6p\" (UID: \"e24e2d68-b448-4c4f-99c6-4effdd9992e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2lc6p"
Nov 02 15:57:29.197512 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:57:29.197468    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a2f560c-e476-4c3a-b96d-1d3e7b313360-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-j6lh6\" (UID: \"9a2f560c-e476-4c3a-b96d-1d3e7b313360\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-j6lh6"
Nov 02 15:57:29.197512 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:57:29.197508    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e24e2d68-b448-4c4f-99c6-4effdd9992e4-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-2lc6p\" (UID: \"e24e2d68-b448-4c4f-99c6-4effdd9992e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2lc6p"
Nov 02 15:57:29.197628 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:57:29.197534    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qjfl7\" (UniqueName: \"kubernetes.io/projected/9a2f560c-e476-4c3a-b96d-1d3e7b313360-kube-api-access-qjfl7\") pod \"maas-default-gateway-openshift-default-fcb75c45f-j6lh6\" (UID: \"9a2f560c-e476-4c3a-b96d-1d3e7b313360\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-j6lh6"
Nov 02 15:57:29.197628 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:57:29.197558    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e24e2d68-b448-4c4f-99c6-4effdd9992e4-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-2lc6p\" (UID: \"e24e2d68-b448-4c4f-99c6-4effdd9992e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2lc6p"
Nov 02 15:57:29.197725 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:57:29.197630    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e24e2d68-b448-4c4f-99c6-4effdd9992e4-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-2lc6p\" (UID: \"e24e2d68-b448-4c4f-99c6-4effdd9992e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2lc6p"
Nov 02 15:57:29.197725 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:57:29.197694    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a2f560c-e476-4c3a-b96d-1d3e7b313360-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-j6lh6\" (UID: \"9a2f560c-e476-4c3a-b96d-1d3e7b313360\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-j6lh6"
Nov 02 15:57:29.197845 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:57:29.197729    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a2f560c-e476-4c3a-b96d-1d3e7b313360-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-j6lh6\" (UID: \"9a2f560c-e476-4c3a-b96d-1d3e7b313360\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-j6lh6"
Nov 02 15:57:29.197845 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:57:29.197801    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a2f560c-e476-4c3a-b96d-1d3e7b313360-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-j6lh6\" (UID: \"9a2f560c-e476-4c3a-b96d-1d3e7b313360\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-j6lh6"
Nov 02 15:57:29.197845 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:57:29.197837    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a2f560c-e476-4c3a-b96d-1d3e7b313360-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-j6lh6\" (UID: \"9a2f560c-e476-4c3a-b96d-1d3e7b313360\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-j6lh6"
Nov 02 15:57:29.197982 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:57:29.197898    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a2f560c-e476-4c3a-b96d-1d3e7b313360-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-j6lh6\" (UID: \"9a2f560c-e476-4c3a-b96d-1d3e7b313360\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-j6lh6"
Nov 02 15:57:29.197982 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:57:29.197918    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e24e2d68-b448-4c4f-99c6-4effdd9992e4-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-2lc6p\" (UID: \"e24e2d68-b448-4c4f-99c6-4effdd9992e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2lc6p"
Nov 02 15:57:29.197982 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:57:29.197923    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e24e2d68-b448-4c4f-99c6-4effdd9992e4-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-2lc6p\" (UID: \"e24e2d68-b448-4c4f-99c6-4effdd9992e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2lc6p"
Nov 02 15:57:29.197982 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:57:29.197964    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e24e2d68-b448-4c4f-99c6-4effdd9992e4-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-2lc6p\" (UID: \"e24e2d68-b448-4c4f-99c6-4effdd9992e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2lc6p"
Nov 02 15:57:29.198243 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:57:29.198090    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e24e2d68-b448-4c4f-99c6-4effdd9992e4-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-2lc6p\" (UID: \"e24e2d68-b448-4c4f-99c6-4effdd9992e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2lc6p"
Nov 02 15:57:29.198243 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:57:29.198116    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e24e2d68-b448-4c4f-99c6-4effdd9992e4-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-2lc6p\" (UID: \"e24e2d68-b448-4c4f-99c6-4effdd9992e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2lc6p"
Nov 02 15:57:29.198243 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:57:29.198213    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a2f560c-e476-4c3a-b96d-1d3e7b313360-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-j6lh6\" (UID: \"9a2f560c-e476-4c3a-b96d-1d3e7b313360\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-j6lh6"
Nov 02 15:57:29.198409 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:57:29.198314    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e24e2d68-b448-4c4f-99c6-4effdd9992e4-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-2lc6p\" (UID: \"e24e2d68-b448-4c4f-99c6-4effdd9992e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2lc6p"
Nov 02 15:57:29.198409 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:57:29.198348    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e24e2d68-b448-4c4f-99c6-4effdd9992e4-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-2lc6p\" (UID: \"e24e2d68-b448-4c4f-99c6-4effdd9992e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2lc6p"
Nov 02 15:57:29.198409 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:57:29.198376    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e24e2d68-b448-4c4f-99c6-4effdd9992e4-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-2lc6p\" (UID: \"e24e2d68-b448-4c4f-99c6-4effdd9992e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2lc6p"
Nov 02 15:57:29.198409 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:57:29.198407    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e24e2d68-b448-4c4f-99c6-4effdd9992e4-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-2lc6p\" (UID: \"e24e2d68-b448-4c4f-99c6-4effdd9992e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2lc6p"
Nov 02 15:57:29.200619 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:57:29.200592    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e24e2d68-b448-4c4f-99c6-4effdd9992e4-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-2lc6p\" (UID: \"e24e2d68-b448-4c4f-99c6-4effdd9992e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2lc6p"
Nov 02 15:57:29.200929 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:57:29.200903    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e24e2d68-b448-4c4f-99c6-4effdd9992e4-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-2lc6p\" (UID: \"e24e2d68-b448-4c4f-99c6-4effdd9992e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2lc6p"
Nov 02 15:57:29.220962 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:57:29.220936    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e24e2d68-b448-4c4f-99c6-4effdd9992e4-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-2lc6p\" (UID: \"e24e2d68-b448-4c4f-99c6-4effdd9992e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2lc6p"
Nov 02 15:57:29.221347 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:57:29.221325    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pt64s\" (UniqueName: \"kubernetes.io/projected/e24e2d68-b448-4c4f-99c6-4effdd9992e4-kube-api-access-pt64s\") pod \"maas-default-gateway-openshift-default-fcb75c45f-2lc6p\" (UID: \"e24e2d68-b448-4c4f-99c6-4effdd9992e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2lc6p"
Nov 02 15:57:29.299027 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:57:29.298991    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a2f560c-e476-4c3a-b96d-1d3e7b313360-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-j6lh6\" (UID: \"9a2f560c-e476-4c3a-b96d-1d3e7b313360\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-j6lh6"
Nov 02 15:57:29.299027 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:57:29.299031    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a2f560c-e476-4c3a-b96d-1d3e7b313360-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-j6lh6\" (UID: \"9a2f560c-e476-4c3a-b96d-1d3e7b313360\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-j6lh6"
Nov 02 15:57:29.299406 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:57:29.299057    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a2f560c-e476-4c3a-b96d-1d3e7b313360-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-j6lh6\" (UID: \"9a2f560c-e476-4c3a-b96d-1d3e7b313360\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-j6lh6"
Nov 02 15:57:29.299406 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:57:29.299161    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a2f560c-e476-4c3a-b96d-1d3e7b313360-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-j6lh6\" (UID: \"9a2f560c-e476-4c3a-b96d-1d3e7b313360\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-j6lh6"
Nov 02 15:57:29.299406 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:57:29.299190    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a2f560c-e476-4c3a-b96d-1d3e7b313360-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-j6lh6\" (UID: \"9a2f560c-e476-4c3a-b96d-1d3e7b313360\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-j6lh6"
Nov 02 15:57:29.299406 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:57:29.299243    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a2f560c-e476-4c3a-b96d-1d3e7b313360-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-j6lh6\" (UID: \"9a2f560c-e476-4c3a-b96d-1d3e7b313360\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-j6lh6"
Nov 02 15:57:29.299406 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:57:29.299279    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qjfl7\" (UniqueName: \"kubernetes.io/projected/9a2f560c-e476-4c3a-b96d-1d3e7b313360-kube-api-access-qjfl7\") pod \"maas-default-gateway-openshift-default-fcb75c45f-j6lh6\" (UID: \"9a2f560c-e476-4c3a-b96d-1d3e7b313360\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-j6lh6"
Nov 02 15:57:29.299406 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:57:29.299309    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a2f560c-e476-4c3a-b96d-1d3e7b313360-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-j6lh6\" (UID: \"9a2f560c-e476-4c3a-b96d-1d3e7b313360\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-j6lh6"
Nov 02 15:57:29.299406 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:57:29.299399    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a2f560c-e476-4c3a-b96d-1d3e7b313360-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-j6lh6\" (UID: \"9a2f560c-e476-4c3a-b96d-1d3e7b313360\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-j6lh6"
Nov 02 15:57:29.299770 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:57:29.299484    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a2f560c-e476-4c3a-b96d-1d3e7b313360-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-j6lh6\" (UID: \"9a2f560c-e476-4c3a-b96d-1d3e7b313360\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-j6lh6"
Nov 02 15:57:29.299770 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:57:29.299654    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a2f560c-e476-4c3a-b96d-1d3e7b313360-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-j6lh6\" (UID: \"9a2f560c-e476-4c3a-b96d-1d3e7b313360\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-j6lh6"
Nov 02 15:57:29.299770 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:57:29.299710    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a2f560c-e476-4c3a-b96d-1d3e7b313360-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-j6lh6\" (UID: \"9a2f560c-e476-4c3a-b96d-1d3e7b313360\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-j6lh6"
Nov 02 15:57:29.299891 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:57:29.299852    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a2f560c-e476-4c3a-b96d-1d3e7b313360-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-j6lh6\" (UID: \"9a2f560c-e476-4c3a-b96d-1d3e7b313360\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-j6lh6"
Nov 02 15:57:29.300078 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:57:29.300062    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a2f560c-e476-4c3a-b96d-1d3e7b313360-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-j6lh6\" (UID: \"9a2f560c-e476-4c3a-b96d-1d3e7b313360\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-j6lh6"
Nov 02 15:57:29.301549 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:57:29.301529    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a2f560c-e476-4c3a-b96d-1d3e7b313360-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-j6lh6\" (UID: \"9a2f560c-e476-4c3a-b96d-1d3e7b313360\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-j6lh6"
Nov 02 15:57:29.301751 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:57:29.301733    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a2f560c-e476-4c3a-b96d-1d3e7b313360-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-j6lh6\" (UID: \"9a2f560c-e476-4c3a-b96d-1d3e7b313360\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-j6lh6"
Nov 02 15:57:29.323108 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:57:29.323039    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a2f560c-e476-4c3a-b96d-1d3e7b313360-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-j6lh6\" (UID: \"9a2f560c-e476-4c3a-b96d-1d3e7b313360\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-j6lh6"
Nov 02 15:57:29.323248 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:57:29.323121    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qjfl7\" (UniqueName: \"kubernetes.io/projected/9a2f560c-e476-4c3a-b96d-1d3e7b313360-kube-api-access-qjfl7\") pod \"maas-default-gateway-openshift-default-fcb75c45f-j6lh6\" (UID: \"9a2f560c-e476-4c3a-b96d-1d3e7b313360\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-j6lh6"
Nov 02 15:57:29.369019 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:57:29.368988    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2lc6p"
Nov 02 15:57:29.431754 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:57:29.431427    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-j6lh6"
Nov 02 15:57:29.685981 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:57:29.685897    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2lc6p"]
Nov 02 15:57:29.691389 ip-10-0-135-138 kubenswrapper[5304]: W1102 15:57:29.691360    5304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode24e2d68_b448_4c4f_99c6_4effdd9992e4.slice/crio-350a8eec9f413e2cc24cf61fb5850a11d4ed8ddef3a3db10e07f98edb0d1f80f WatchSource:0}: Error finding container 350a8eec9f413e2cc24cf61fb5850a11d4ed8ddef3a3db10e07f98edb0d1f80f: Status 404 returned error can't find the container with id 350a8eec9f413e2cc24cf61fb5850a11d4ed8ddef3a3db10e07f98edb0d1f80f
Nov 02 15:57:29.693444 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:57:29.693411    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:57:29.693529 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:57:29.693482    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:57:29.693529 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:57:29.693514    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:57:29.711968 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:57:29.711936    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2lc6p" event={"ID":"e24e2d68-b448-4c4f-99c6-4effdd9992e4","Type":"ContainerStarted","Data":"350a8eec9f413e2cc24cf61fb5850a11d4ed8ddef3a3db10e07f98edb0d1f80f"}
Nov 02 15:57:29.747526 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:57:29.747501    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-j6lh6"]
Nov 02 15:57:29.751350 ip-10-0-135-138 kubenswrapper[5304]: W1102 15:57:29.751324    5304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a2f560c_e476_4c3a_b96d_1d3e7b313360.slice/crio-f658fab31bb0b80c91df5fddf1999f554ac0216ef8d29270ee134a5e42178253 WatchSource:0}: Error finding container f658fab31bb0b80c91df5fddf1999f554ac0216ef8d29270ee134a5e42178253: Status 404 returned error can't find the container with id f658fab31bb0b80c91df5fddf1999f554ac0216ef8d29270ee134a5e42178253
Nov 02 15:57:29.753535 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:57:29.753504    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:57:29.753636 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:57:29.753565    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:57:29.753636 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:57:29.753591    5304 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Nov 02 15:57:30.718622 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:57:30.718585    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2lc6p" event={"ID":"e24e2d68-b448-4c4f-99c6-4effdd9992e4","Type":"ContainerStarted","Data":"8569c8d582fb5faff4a0b065fcef9b4e0e916424f865c43b32f4494b6d9d48e2"}
Nov 02 15:57:30.720598 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:57:30.720575    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-j6lh6" event={"ID":"9a2f560c-e476-4c3a-b96d-1d3e7b313360","Type":"ContainerStarted","Data":"ae75edda82cf49659a71a5935b98d5d780773885668a10aae3ff57f80656c270"}
Nov 02 15:57:30.720696 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:57:30.720615    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-j6lh6" event={"ID":"9a2f560c-e476-4c3a-b96d-1d3e7b313360","Type":"ContainerStarted","Data":"f658fab31bb0b80c91df5fddf1999f554ac0216ef8d29270ee134a5e42178253"}
Nov 02 15:57:30.791145 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:57:30.791076    5304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2lc6p" podStartSLOduration=1.791061206 podStartE2EDuration="1.791061206s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:57:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:57:30.789828826 +0000 UTC m=+3650.783653268" watchObservedRunningTime="2026-11-02 15:57:30.791061206 +0000 UTC m=+3650.784885615"
Nov 02 15:57:30.848160 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:57:30.846385    5304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-j6lh6" podStartSLOduration=1.846350316 podStartE2EDuration="1.846350316s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:57:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:57:30.845095501 +0000 UTC m=+3650.838919912" watchObservedRunningTime="2026-11-02 15:57:30.846350316 +0000 UTC m=+3650.840174731"
Nov 02 15:57:31.370149 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:57:31.369541    5304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2lc6p"
Nov 02 15:57:31.371860 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:57:31.371830    5304 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-2lc6p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.223:15021/healthz/ready\": dial tcp 10.133.0.223:15021: connect: connection refused" start-of-body=
Nov 02 15:57:31.372073 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:57:31.372049    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2lc6p" podUID="e24e2d68-b448-4c4f-99c6-4effdd9992e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.223:15021/healthz/ready\": dial tcp 10.133.0.223:15021: connect: connection refused"
Nov 02 15:57:31.446541 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:57:31.434505    5304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-j6lh6"
Nov 02 15:57:31.447283 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:57:31.447248    5304 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-j6lh6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.224:15021/healthz/ready\": dial tcp 10.133.0.224:15021: connect: connection refused" start-of-body=
Nov 02 15:57:31.448444 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:57:31.448410    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-j6lh6" podUID="9a2f560c-e476-4c3a-b96d-1d3e7b313360" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.224:15021/healthz/ready\": dial tcp 10.133.0.224:15021: connect: connection refused"
Nov 02 15:57:32.373151 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:57:32.371546    5304 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-2lc6p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.223:15021/healthz/ready\": dial tcp 10.133.0.223:15021: connect: connection refused" start-of-body=
Nov 02 15:57:32.373151 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:57:32.371609    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2lc6p" podUID="e24e2d68-b448-4c4f-99c6-4effdd9992e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.223:15021/healthz/ready\": dial tcp 10.133.0.223:15021: connect: connection refused"
Nov 02 15:57:32.455154 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:57:32.451575    5304 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-j6lh6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.224:15021/healthz/ready\": dial tcp 10.133.0.224:15021: connect: connection refused" start-of-body=
Nov 02 15:57:32.455154 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:57:32.451635    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-j6lh6" podUID="9a2f560c-e476-4c3a-b96d-1d3e7b313360" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.224:15021/healthz/ready\": dial tcp 10.133.0.224:15021: connect: connection refused"
Nov 02 15:57:33.369695 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:57:33.369650    5304 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-2lc6p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.223:15021/healthz/ready\": dial tcp 10.133.0.223:15021: connect: connection refused" start-of-body=
Nov 02 15:57:33.370214 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:57:33.370165    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2lc6p" podUID="e24e2d68-b448-4c4f-99c6-4effdd9992e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.223:15021/healthz/ready\": dial tcp 10.133.0.223:15021: connect: connection refused"
Nov 02 15:57:33.445573 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:57:33.445531    5304 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-j6lh6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.224:15021/healthz/ready\": dial tcp 10.133.0.224:15021: connect: connection refused" start-of-body=
Nov 02 15:57:33.447153 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:57:33.447097    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-j6lh6" podUID="9a2f560c-e476-4c3a-b96d-1d3e7b313360" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.224:15021/healthz/ready\": dial tcp 10.133.0.224:15021: connect: connection refused"
Nov 02 15:57:34.369620 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:57:34.369578    5304 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-2lc6p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.223:15021/healthz/ready\": dial tcp 10.133.0.223:15021: connect: connection refused" start-of-body=
Nov 02 15:57:34.369944 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:57:34.369912    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2lc6p" podUID="e24e2d68-b448-4c4f-99c6-4effdd9992e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.223:15021/healthz/ready\": dial tcp 10.133.0.223:15021: connect: connection refused"
Nov 02 15:57:34.434586 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:57:34.434542    5304 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-j6lh6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.224:15021/healthz/ready\": dial tcp 10.133.0.224:15021: connect: connection refused" start-of-body=
Nov 02 15:57:34.434920 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:57:34.434886    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-j6lh6" podUID="9a2f560c-e476-4c3a-b96d-1d3e7b313360" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.224:15021/healthz/ready\": dial tcp 10.133.0.224:15021: connect: connection refused"
Nov 02 15:57:35.369585 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:57:35.369544    5304 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-2lc6p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.223:15021/healthz/ready\": dial tcp 10.133.0.223:15021: connect: connection refused" start-of-body=
Nov 02 15:57:35.370249 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:57:35.370212    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2lc6p" podUID="e24e2d68-b448-4c4f-99c6-4effdd9992e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.223:15021/healthz/ready\": dial tcp 10.133.0.223:15021: connect: connection refused"
Nov 02 15:57:35.438154 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:57:35.434506    5304 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-j6lh6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.224:15021/healthz/ready\": dial tcp 10.133.0.224:15021: connect: connection refused" start-of-body=
Nov 02 15:57:35.438154 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:57:35.434564    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-j6lh6" podUID="9a2f560c-e476-4c3a-b96d-1d3e7b313360" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.224:15021/healthz/ready\": dial tcp 10.133.0.224:15021: connect: connection refused"
Nov 02 15:57:36.370427 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:57:36.370380    5304 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-2lc6p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.223:15021/healthz/ready\": dial tcp 10.133.0.223:15021: connect: connection refused" start-of-body=
Nov 02 15:57:36.370846 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:57:36.370448    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2lc6p" podUID="e24e2d68-b448-4c4f-99c6-4effdd9992e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.223:15021/healthz/ready\": dial tcp 10.133.0.223:15021: connect: connection refused"
Nov 02 15:57:36.432427 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:57:36.432383    5304 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-j6lh6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.224:15021/healthz/ready\": dial tcp 10.133.0.224:15021: connect: connection refused" start-of-body=
Nov 02 15:57:36.432633 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:57:36.432444    5304 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-j6lh6" podUID="9a2f560c-e476-4c3a-b96d-1d3e7b313360" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.224:15021/healthz/ready\": dial tcp 10.133.0.224:15021: connect: connection refused"
Nov 02 15:57:38.048890 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:57:38.048084    5304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2lc6p"
Nov 02 15:57:38.048890 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:57:38.048712    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2lc6p"
Nov 02 15:57:38.049654 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:57:38.049634    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2lc6p"
Nov 02 15:57:38.117485 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:57:38.117449    5304 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-j6lh6"
Nov 02 15:57:38.117836 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:57:38.117816    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-j6lh6"
Nov 02 15:57:38.118740 ip-10-0-135-138 kubenswrapper[5304]: I1102 15:57:38.118720    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-j6lh6"
Nov 02 15:57:54.884749 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:57:54.884707    5304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k6fjk" containerName="extractor"
Nov 02 15:58:24.796565 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:58:24.796477    5304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k6fjk" containerName="extractor"
Nov 02 15:58:54.797797 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:58:54.797756    5304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k6fjk" containerName="extractor"
Nov 02 15:59:24.797643 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:59:24.797600    5304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k6fjk" containerName="extractor"
Nov 02 15:59:54.797932 ip-10-0-135-138 kubenswrapper[5304]: E1102 15:59:54.797842    5304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k6fjk" containerName="extractor"
Nov 02 16:00:24.798226 ip-10-0-135-138 kubenswrapper[5304]: E1102 16:00:24.798187    5304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k6fjk" containerName="extractor"
Nov 02 16:00:54.798004 ip-10-0-135-138 kubenswrapper[5304]: E1102 16:00:54.797965    5304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k6fjk" containerName="extractor"
Nov 02 16:01:24.797350 ip-10-0-135-138 kubenswrapper[5304]: E1102 16:01:24.797272    5304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k6fjk" containerName="extractor"
Nov 02 16:01:54.797692 ip-10-0-135-138 kubenswrapper[5304]: E1102 16:01:54.797650    5304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k6fjk" containerName="extractor"
Nov 02 16:02:24.797258 ip-10-0-135-138 kubenswrapper[5304]: E1102 16:02:24.797219    5304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k6fjk" containerName="extractor"
Nov 02 16:02:54.799388 ip-10-0-135-138 kubenswrapper[5304]: E1102 16:02:54.799305    5304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k6fjk" containerName="extractor"
Nov 02 16:03:24.798477 ip-10-0-135-138 kubenswrapper[5304]: E1102 16:03:24.798438    5304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k6fjk" containerName="extractor"
Nov 02 16:03:54.798827 ip-10-0-135-138 kubenswrapper[5304]: E1102 16:03:54.798786    5304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k6fjk" containerName="extractor"
Nov 02 16:04:24.797273 ip-10-0-135-138 kubenswrapper[5304]: E1102 16:04:24.797192    5304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k6fjk" containerName="extractor"
Nov 02 16:04:54.800498 ip-10-0-135-138 kubenswrapper[5304]: E1102 16:04:54.800451    5304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k6fjk" containerName="extractor"
Nov 02 16:05:24.795813 ip-10-0-135-138 kubenswrapper[5304]: E1102 16:05:24.795774    5304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k6fjk" containerName="extractor"
Nov 02 16:05:54.797733 ip-10-0-135-138 kubenswrapper[5304]: E1102 16:05:54.797651    5304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k6fjk" containerName="extractor"
Nov 02 16:06:24.798355 ip-10-0-135-138 kubenswrapper[5304]: E1102 16:06:24.798320    5304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k6fjk" containerName="extractor"
Nov 02 16:06:54.797734 ip-10-0-135-138 kubenswrapper[5304]: E1102 16:06:54.797694    5304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k6fjk" containerName="extractor"
Nov 02 16:07:24.798799 ip-10-0-135-138 kubenswrapper[5304]: E1102 16:07:24.798713    5304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k6fjk" containerName="extractor"
Nov 02 16:07:54.798673 ip-10-0-135-138 kubenswrapper[5304]: E1102 16:07:54.798638    5304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k6fjk" containerName="extractor"
Nov 02 16:08:24.798813 ip-10-0-135-138 kubenswrapper[5304]: E1102 16:08:24.798766    5304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k6fjk" containerName="extractor"
Nov 02 16:08:54.798964 ip-10-0-135-138 kubenswrapper[5304]: E1102 16:08:54.798884    5304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k6fjk" containerName="extractor"
Nov 02 16:09:24.798782 ip-10-0-135-138 kubenswrapper[5304]: E1102 16:09:24.798741    5304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k6fjk" containerName="extractor"
Nov 02 16:09:54.797977 ip-10-0-135-138 kubenswrapper[5304]: E1102 16:09:54.797937    5304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k6fjk" containerName="extractor"
Nov 02 16:10:01.194264 ip-10-0-135-138 kubenswrapper[5304]: I1102 16:10:01.194206    5304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-5484769d88-cbstn"]
Nov 02 16:10:01.197847 ip-10-0-135-138 kubenswrapper[5304]: I1102 16:10:01.197823    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5484769d88-cbstn"
Nov 02 16:10:01.225597 ip-10-0-135-138 kubenswrapper[5304]: I1102 16:10:01.225567    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5484769d88-cbstn"]
Nov 02 16:10:01.324471 ip-10-0-135-138 kubenswrapper[5304]: I1102 16:10:01.324433    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb1a05be-2f3b-4f5a-a1b7-69d822c1d658-openshift-service-ca\") pod \"maas-api-5484769d88-cbstn\" (UID: \"cb1a05be-2f3b-4f5a-a1b7-69d822c1d658\") " pod="odh-ai-gateway-infra/maas-api-5484769d88-cbstn"
Nov 02 16:10:01.324661 ip-10-0-135-138 kubenswrapper[5304]: I1102 16:10:01.324483    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cb1a05be-2f3b-4f5a-a1b7-69d822c1d658-metrics-certs\") pod \"maas-api-5484769d88-cbstn\" (UID: \"cb1a05be-2f3b-4f5a-a1b7-69d822c1d658\") " pod="odh-ai-gateway-infra/maas-api-5484769d88-cbstn"
Nov 02 16:10:01.324661 ip-10-0-135-138 kubenswrapper[5304]: I1102 16:10:01.324570    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cb1a05be-2f3b-4f5a-a1b7-69d822c1d658-maas-api-tls\") pod \"maas-api-5484769d88-cbstn\" (UID: \"cb1a05be-2f3b-4f5a-a1b7-69d822c1d658\") " pod="odh-ai-gateway-infra/maas-api-5484769d88-cbstn"
Nov 02 16:10:01.324661 ip-10-0-135-138 kubenswrapper[5304]: I1102 16:10:01.324646    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z29xm\" (UniqueName: \"kubernetes.io/projected/cb1a05be-2f3b-4f5a-a1b7-69d822c1d658-kube-api-access-z29xm\") pod \"maas-api-5484769d88-cbstn\" (UID: \"cb1a05be-2f3b-4f5a-a1b7-69d822c1d658\") " pod="odh-ai-gateway-infra/maas-api-5484769d88-cbstn"
Nov 02 16:10:01.425588 ip-10-0-135-138 kubenswrapper[5304]: I1102 16:10:01.425550    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cb1a05be-2f3b-4f5a-a1b7-69d822c1d658-maas-api-tls\") pod \"maas-api-5484769d88-cbstn\" (UID: \"cb1a05be-2f3b-4f5a-a1b7-69d822c1d658\") " pod="odh-ai-gateway-infra/maas-api-5484769d88-cbstn"
Nov 02 16:10:01.425794 ip-10-0-135-138 kubenswrapper[5304]: I1102 16:10:01.425624    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z29xm\" (UniqueName: \"kubernetes.io/projected/cb1a05be-2f3b-4f5a-a1b7-69d822c1d658-kube-api-access-z29xm\") pod \"maas-api-5484769d88-cbstn\" (UID: \"cb1a05be-2f3b-4f5a-a1b7-69d822c1d658\") " pod="odh-ai-gateway-infra/maas-api-5484769d88-cbstn"
Nov 02 16:10:01.425794 ip-10-0-135-138 kubenswrapper[5304]: I1102 16:10:01.425674    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb1a05be-2f3b-4f5a-a1b7-69d822c1d658-openshift-service-ca\") pod \"maas-api-5484769d88-cbstn\" (UID: \"cb1a05be-2f3b-4f5a-a1b7-69d822c1d658\") " pod="odh-ai-gateway-infra/maas-api-5484769d88-cbstn"
Nov 02 16:10:01.425794 ip-10-0-135-138 kubenswrapper[5304]: I1102 16:10:01.425702    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cb1a05be-2f3b-4f5a-a1b7-69d822c1d658-metrics-certs\") pod \"maas-api-5484769d88-cbstn\" (UID: \"cb1a05be-2f3b-4f5a-a1b7-69d822c1d658\") " pod="odh-ai-gateway-infra/maas-api-5484769d88-cbstn"
Nov 02 16:10:01.426392 ip-10-0-135-138 kubenswrapper[5304]: I1102 16:10:01.426366    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb1a05be-2f3b-4f5a-a1b7-69d822c1d658-openshift-service-ca\") pod \"maas-api-5484769d88-cbstn\" (UID: \"cb1a05be-2f3b-4f5a-a1b7-69d822c1d658\") " pod="odh-ai-gateway-infra/maas-api-5484769d88-cbstn"
Nov 02 16:10:01.428221 ip-10-0-135-138 kubenswrapper[5304]: I1102 16:10:01.428193    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cb1a05be-2f3b-4f5a-a1b7-69d822c1d658-maas-api-tls\") pod \"maas-api-5484769d88-cbstn\" (UID: \"cb1a05be-2f3b-4f5a-a1b7-69d822c1d658\") " pod="odh-ai-gateway-infra/maas-api-5484769d88-cbstn"
Nov 02 16:10:01.428326 ip-10-0-135-138 kubenswrapper[5304]: I1102 16:10:01.428200    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cb1a05be-2f3b-4f5a-a1b7-69d822c1d658-metrics-certs\") pod \"maas-api-5484769d88-cbstn\" (UID: \"cb1a05be-2f3b-4f5a-a1b7-69d822c1d658\") " pod="odh-ai-gateway-infra/maas-api-5484769d88-cbstn"
Nov 02 16:10:01.448674 ip-10-0-135-138 kubenswrapper[5304]: I1102 16:10:01.448617    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z29xm\" (UniqueName: \"kubernetes.io/projected/cb1a05be-2f3b-4f5a-a1b7-69d822c1d658-kube-api-access-z29xm\") pod \"maas-api-5484769d88-cbstn\" (UID: \"cb1a05be-2f3b-4f5a-a1b7-69d822c1d658\") " pod="odh-ai-gateway-infra/maas-api-5484769d88-cbstn"
Nov 02 16:10:01.510555 ip-10-0-135-138 kubenswrapper[5304]: I1102 16:10:01.510518    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5484769d88-cbstn"
Nov 02 16:10:01.812946 ip-10-0-135-138 kubenswrapper[5304]: I1102 16:10:01.812843    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5484769d88-cbstn"]
Nov 02 16:10:01.817777 ip-10-0-135-138 kubenswrapper[5304]: W1102 16:10:01.817742    5304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb1a05be_2f3b_4f5a_a1b7_69d822c1d658.slice/crio-6e7ddcbe48250f6004d3d7b3f71be7cb8e134253d341c85a16aadbe154a60004 WatchSource:0}: Error finding container 6e7ddcbe48250f6004d3d7b3f71be7cb8e134253d341c85a16aadbe154a60004: Status 404 returned error can't find the container with id 6e7ddcbe48250f6004d3d7b3f71be7cb8e134253d341c85a16aadbe154a60004
Nov 02 16:10:01.819174 ip-10-0-135-138 kubenswrapper[5304]: I1102 16:10:01.819155    5304 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 02 16:10:02.384461 ip-10-0-135-138 kubenswrapper[5304]: I1102 16:10:02.384429    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5484769d88-cbstn" event={"ID":"cb1a05be-2f3b-4f5a-a1b7-69d822c1d658","Type":"ContainerStarted","Data":"39eb8b52d9853d2c81c893c2aa8a0ac1f0df30cf00548f9555aed66c88d6c5b1"}
Nov 02 16:10:02.384896 ip-10-0-135-138 kubenswrapper[5304]: I1102 16:10:02.384477    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5484769d88-cbstn" event={"ID":"cb1a05be-2f3b-4f5a-a1b7-69d822c1d658","Type":"ContainerStarted","Data":"6e7ddcbe48250f6004d3d7b3f71be7cb8e134253d341c85a16aadbe154a60004"}
Nov 02 16:10:02.384896 ip-10-0-135-138 kubenswrapper[5304]: I1102 16:10:02.384573    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-5484769d88-cbstn"
Nov 02 16:10:02.430528 ip-10-0-135-138 kubenswrapper[5304]: I1102 16:10:02.430471    5304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-5484769d88-cbstn" podStartSLOduration=1.0552988189999999 podStartE2EDuration="1.4304565s" totalImagesPullingTime="375.157681ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 16:10:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 16:10:02.429932792 +0000 UTC m=+4402.423757200" watchObservedRunningTime="2026-11-02 16:10:02.4304565 +0000 UTC m=+4402.424280903"
Nov 02 16:10:08.394228 ip-10-0-135-138 kubenswrapper[5304]: I1102 16:10:08.394193    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-5484769d88-cbstn"
Nov 02 16:10:08.474446 ip-10-0-135-138 kubenswrapper[5304]: I1102 16:10:08.474404    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-59cd689d68-cznfp"]
Nov 02 16:10:08.474725 ip-10-0-135-138 kubenswrapper[5304]: I1102 16:10:08.474653    5304 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-59cd689d68-cznfp" podUID="ff2f1cbd-70c5-48d0-b133-0e4ea14e8dec" containerName="maas-api" containerID="cri-o://9e8be530eaadf1113ff4ac06481154ce80ce59a6fd181dc92c5a6ec9c3b5a103" gracePeriod=30
Nov 02 16:10:09.727965 ip-10-0-135-138 kubenswrapper[5304]: I1102 16:10:09.727933    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-59cd689d68-cznfp"
Nov 02 16:10:09.811763 ip-10-0-135-138 kubenswrapper[5304]: I1102 16:10:09.811724    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff2f1cbd-70c5-48d0-b133-0e4ea14e8dec-kube-api-access-9rwsk\" (UniqueName: \"kubernetes.io/projected/ff2f1cbd-70c5-48d0-b133-0e4ea14e8dec-kube-api-access-9rwsk\") pod \"ff2f1cbd-70c5-48d0-b133-0e4ea14e8dec\" (UID: \"ff2f1cbd-70c5-48d0-b133-0e4ea14e8dec\") "
Nov 02 16:10:09.811970 ip-10-0-135-138 kubenswrapper[5304]: I1102 16:10:09.811778    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ff2f1cbd-70c5-48d0-b133-0e4ea14e8dec-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff2f1cbd-70c5-48d0-b133-0e4ea14e8dec-openshift-service-ca\") pod \"ff2f1cbd-70c5-48d0-b133-0e4ea14e8dec\" (UID: \"ff2f1cbd-70c5-48d0-b133-0e4ea14e8dec\") "
Nov 02 16:10:09.811970 ip-10-0-135-138 kubenswrapper[5304]: I1102 16:10:09.811799    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ff2f1cbd-70c5-48d0-b133-0e4ea14e8dec-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ff2f1cbd-70c5-48d0-b133-0e4ea14e8dec-maas-api-tls\") pod \"ff2f1cbd-70c5-48d0-b133-0e4ea14e8dec\" (UID: \"ff2f1cbd-70c5-48d0-b133-0e4ea14e8dec\") "
Nov 02 16:10:09.811970 ip-10-0-135-138 kubenswrapper[5304]: I1102 16:10:09.811835    5304 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ff2f1cbd-70c5-48d0-b133-0e4ea14e8dec-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ff2f1cbd-70c5-48d0-b133-0e4ea14e8dec-metrics-certs\") pod \"ff2f1cbd-70c5-48d0-b133-0e4ea14e8dec\" (UID: \"ff2f1cbd-70c5-48d0-b133-0e4ea14e8dec\") "
Nov 02 16:10:09.812247 ip-10-0-135-138 kubenswrapper[5304]: I1102 16:10:09.812215    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff2f1cbd-70c5-48d0-b133-0e4ea14e8dec-openshift-service-ca" pod "ff2f1cbd-70c5-48d0-b133-0e4ea14e8dec" (UID: "ff2f1cbd-70c5-48d0-b133-0e4ea14e8dec"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 16:10:09.814196 ip-10-0-135-138 kubenswrapper[5304]: I1102 16:10:09.814095    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff2f1cbd-70c5-48d0-b133-0e4ea14e8dec-metrics-certs" pod "ff2f1cbd-70c5-48d0-b133-0e4ea14e8dec" (UID: "ff2f1cbd-70c5-48d0-b133-0e4ea14e8dec"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 16:10:09.814196 ip-10-0-135-138 kubenswrapper[5304]: I1102 16:10:09.814186    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff2f1cbd-70c5-48d0-b133-0e4ea14e8dec-maas-api-tls" pod "ff2f1cbd-70c5-48d0-b133-0e4ea14e8dec" (UID: "ff2f1cbd-70c5-48d0-b133-0e4ea14e8dec"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 16:10:09.814375 ip-10-0-135-138 kubenswrapper[5304]: I1102 16:10:09.814279    5304 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff2f1cbd-70c5-48d0-b133-0e4ea14e8dec-kube-api-access-9rwsk" pod "ff2f1cbd-70c5-48d0-b133-0e4ea14e8dec" (UID: "ff2f1cbd-70c5-48d0-b133-0e4ea14e8dec"). InnerVolumeSpecName "kube-api-access-9rwsk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 16:10:09.917148 ip-10-0-135-138 kubenswrapper[5304]: I1102 16:10:09.913675    5304 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9rwsk\" (UniqueName: \"kubernetes.io/projected/ff2f1cbd-70c5-48d0-b133-0e4ea14e8dec-kube-api-access-9rwsk\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 16:10:09.917148 ip-10-0-135-138 kubenswrapper[5304]: I1102 16:10:09.913716    5304 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff2f1cbd-70c5-48d0-b133-0e4ea14e8dec-openshift-service-ca\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 16:10:09.917148 ip-10-0-135-138 kubenswrapper[5304]: I1102 16:10:09.913732    5304 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ff2f1cbd-70c5-48d0-b133-0e4ea14e8dec-maas-api-tls\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 16:10:09.917148 ip-10-0-135-138 kubenswrapper[5304]: I1102 16:10:09.913745    5304 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ff2f1cbd-70c5-48d0-b133-0e4ea14e8dec-metrics-certs\") on node \"ip-10-0-135-138.ec2.internal\" DevicePath \"\""
Nov 02 16:10:10.422996 ip-10-0-135-138 kubenswrapper[5304]: I1102 16:10:10.422897    5304 generic.go:356] "Generic (PLEG): container finished" podID="ff2f1cbd-70c5-48d0-b133-0e4ea14e8dec" containerID="9e8be530eaadf1113ff4ac06481154ce80ce59a6fd181dc92c5a6ec9c3b5a103" exitCode=0
Nov 02 16:10:10.422996 ip-10-0-135-138 kubenswrapper[5304]: I1102 16:10:10.422954    5304 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-59cd689d68-cznfp"
Nov 02 16:10:10.422996 ip-10-0-135-138 kubenswrapper[5304]: I1102 16:10:10.422987    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-59cd689d68-cznfp" event={"ID":"ff2f1cbd-70c5-48d0-b133-0e4ea14e8dec","Type":"ContainerDied","Data":"9e8be530eaadf1113ff4ac06481154ce80ce59a6fd181dc92c5a6ec9c3b5a103"}
Nov 02 16:10:10.423362 ip-10-0-135-138 kubenswrapper[5304]: I1102 16:10:10.423041    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-59cd689d68-cznfp" event={"ID":"ff2f1cbd-70c5-48d0-b133-0e4ea14e8dec","Type":"ContainerDied","Data":"df7745087b8641938f66c42b877358300c3070a2943d7eb1ec2126a67e8395d6"}
Nov 02 16:10:10.423362 ip-10-0-135-138 kubenswrapper[5304]: I1102 16:10:10.423063    5304 scope.go:122] "RemoveContainer" containerID="9e8be530eaadf1113ff4ac06481154ce80ce59a6fd181dc92c5a6ec9c3b5a103"
Nov 02 16:10:10.432188 ip-10-0-135-138 kubenswrapper[5304]: I1102 16:10:10.432170    5304 scope.go:122] "RemoveContainer" containerID="9e8be530eaadf1113ff4ac06481154ce80ce59a6fd181dc92c5a6ec9c3b5a103"
Nov 02 16:10:10.432480 ip-10-0-135-138 kubenswrapper[5304]: E1102 16:10:10.432460    5304 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e8be530eaadf1113ff4ac06481154ce80ce59a6fd181dc92c5a6ec9c3b5a103\": container with ID starting with 9e8be530eaadf1113ff4ac06481154ce80ce59a6fd181dc92c5a6ec9c3b5a103 not found: ID does not exist" containerID="9e8be530eaadf1113ff4ac06481154ce80ce59a6fd181dc92c5a6ec9c3b5a103"
Nov 02 16:10:10.432534 ip-10-0-135-138 kubenswrapper[5304]: I1102 16:10:10.432491    5304 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e8be530eaadf1113ff4ac06481154ce80ce59a6fd181dc92c5a6ec9c3b5a103"} err="failed to get container status \"9e8be530eaadf1113ff4ac06481154ce80ce59a6fd181dc92c5a6ec9c3b5a103\": rpc error: code = NotFound desc = could not find container \"9e8be530eaadf1113ff4ac06481154ce80ce59a6fd181dc92c5a6ec9c3b5a103\": container with ID starting with 9e8be530eaadf1113ff4ac06481154ce80ce59a6fd181dc92c5a6ec9c3b5a103 not found: ID does not exist"
Nov 02 16:10:10.473055 ip-10-0-135-138 kubenswrapper[5304]: I1102 16:10:10.472994    5304 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-59cd689d68-cznfp"]
Nov 02 16:10:10.485078 ip-10-0-135-138 kubenswrapper[5304]: I1102 16:10:10.485046    5304 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-59cd689d68-cznfp"]
Nov 02 16:10:12.116664 ip-10-0-135-138 kubenswrapper[5304]: I1102 16:10:12.116630    5304 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ff2f1cbd-70c5-48d0-b133-0e4ea14e8dec" path="/var/lib/kubelet/pods/ff2f1cbd-70c5-48d0-b133-0e4ea14e8dec/volumes"
Nov 02 16:10:24.798244 ip-10-0-135-138 kubenswrapper[5304]: E1102 16:10:24.798166    5304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k6fjk" containerName="extractor"
Nov 02 16:10:54.796855 ip-10-0-135-138 kubenswrapper[5304]: E1102 16:10:54.796818    5304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k6fjk" containerName="extractor"
Nov 02 16:11:24.797882 ip-10-0-135-138 kubenswrapper[5304]: E1102 16:11:24.797846    5304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k6fjk" containerName="extractor"
Nov 02 16:11:54.799347 ip-10-0-135-138 kubenswrapper[5304]: E1102 16:11:54.799267    5304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k6fjk" containerName="extractor"
Nov 02 16:12:24.798840 ip-10-0-135-138 kubenswrapper[5304]: E1102 16:12:24.798804    5304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k6fjk" containerName="extractor"
Nov 02 16:12:54.799861 ip-10-0-135-138 kubenswrapper[5304]: E1102 16:12:54.799818    5304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k6fjk" containerName="extractor"
Nov 02 16:13:24.799883 ip-10-0-135-138 kubenswrapper[5304]: E1102 16:13:24.799788    5304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k6fjk" containerName="extractor"
Nov 02 16:13:47.177666 ip-10-0-135-138 kubenswrapper[5304]: I1102 16:13:47.177625    5304 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-84d5ffc6f9-rvpgh_2dd4123d-04a7-4eee-b14e-8c79b45a310d/authorino/0.log"
Nov 02 16:13:54.798778 ip-10-0-135-138 kubenswrapper[5304]: E1102 16:13:54.798737    5304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k6fjk" containerName="extractor"
Nov 02 16:14:24.798798 ip-10-0-135-138 kubenswrapper[5304]: E1102 16:14:24.798761    5304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k6fjk" containerName="extractor"
Nov 02 16:14:39.356741 ip-10-0-135-138 kubenswrapper[5304]: I1102 16:14:39.356704    5304 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-6d8bf9c9ff-r77kz_0f933e27-7304-44d5-acc5-8089829b1b90/manager/0.log"
Nov 02 16:14:40.807316 ip-10-0-135-138 kubenswrapper[5304]: I1102 16:14:40.807287    5304 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-7lk7m_3fc11777-186c-445a-8b71-2aefdf642e75/manager/0.log"
Nov 02 16:14:41.821483 ip-10-0-135-138 kubenswrapper[5304]: I1102 16:14:41.821449    5304 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-5484769d88-cbstn_cb1a05be-2f3b-4f5a-a1b7-69d822c1d658/maas-api/0.log"
Nov 02 16:14:54.581221 ip-10-0-135-138 kubenswrapper[5304]: I1102 16:14:54.581149    5304 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-84d5ffc6f9-rvpgh_2dd4123d-04a7-4eee-b14e-8c79b45a310d/authorino/0.log"
Nov 02 16:14:54.798475 ip-10-0-135-138 kubenswrapper[5304]: E1102 16:14:54.798370    5304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k6fjk" containerName="extractor"
Nov 02 16:14:55.253908 ip-10-0-135-138 kubenswrapper[5304]: I1102 16:14:55.253879    5304 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-n9gf2_6f7db1ab-d7ad-4aa8-b04e-147b63a46ad6/kuadrant-console-plugin/0.log"
Nov 02 16:14:57.009481 ip-10-0-135-138 kubenswrapper[5304]: I1102 16:14:57.009444    5304 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-zkvr7_f967e0cd-2c0f-42ad-9163-2786089f3ca9/discovery/0.log"
Nov 02 16:14:57.233277 ip-10-0-135-138 kubenswrapper[5304]: I1102 16:14:57.233237    5304 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-77f469495f-6cphj_a427d6de-97f8-4bbd-99d8-6327efcf1794/kube-auth-proxy/0.log"
Nov 02 16:14:57.618315 ip-10-0-135-138 kubenswrapper[5304]: I1102 16:14:57.618276    5304 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-fcb75c45f-2lc6p_e24e2d68-b448-4c4f-99c6-4effdd9992e4/istio-proxy/0.log"
Nov 02 16:14:58.251301 ip-10-0-135-138 kubenswrapper[5304]: I1102 16:14:58.251265    5304 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-fcb75c45f-j6lh6_9a2f560c-e476-4c3a-b96d-1d3e7b313360/istio-proxy/0.log"
Nov 02 16:14:59.262941 ip-10-0-135-138 kubenswrapper[5304]: I1102 16:14:59.262906    5304 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-fcb75c45f-srlvv_98cb3e76-0b44-497d-aaf7-4ee08e76d1b1/istio-proxy/0.log"
Nov 02 16:15:00.085318 ip-10-0-135-138 kubenswrapper[5304]: I1102 16:15:00.085283    5304 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-wxjl7_2f905a4e-b982-4d66-a7bc-171cda1c78dd/payload-processing/0.log"
Nov 02 16:15:01.127266 ip-10-0-135-138 kubenswrapper[5304]: I1102 16:15:01.127238    5304 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-cbd4f5586-7t7f8_b76c8a20-92cc-41ac-b1be-2a779ec1a5ed/main/0.log"
Nov 02 16:15:01.325780 ip-10-0-135-138 kubenswrapper[5304]: I1102 16:15:01.325744    5304 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-7b998b79d9-rmdsn_8cca3fa3-d2b1-4abb-bbb5-a38dfaa3c6b3/main/0.log"
Nov 02 16:15:01.732408 ip-10-0-135-138 kubenswrapper[5304]: I1102 16:15:01.732377    5304 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-69b588897d-56lvt_569cd1bd-5016-4c33-98d7-d9b76c93de43/main/0.log"
Nov 02 16:15:02.108870 ip-10-0-135-138 kubenswrapper[5304]: I1102 16:15:02.108835    5304 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-t8nst_6eaab677-88a0-49d8-9c89-5deec4c81a4d/main/0.log"
Nov 02 16:15:02.303047 ip-10-0-135-138 kubenswrapper[5304]: I1102 16:15:02.303005    5304 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-dzs48_ed68913c-1ebd-4bb6-b3d0-58a9a19e2192/main/0.log"
Nov 02 16:15:02.496173 ip-10-0-135-138 kubenswrapper[5304]: I1102 16:15:02.496069    5304 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-nkbrk_1f68d837-fff2-418b-84d7-c2d7eb21a57e/main/0.log"
Nov 02 16:15:03.444660 ip-10-0-135-138 kubenswrapper[5304]: I1102 16:15:03.444609    5304 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-84d5ffc6f9-rvpgh_2dd4123d-04a7-4eee-b14e-8c79b45a310d/authorino/0.log"
Nov 02 16:15:14.177049 ip-10-0-135-138 kubenswrapper[5304]: I1102 16:15:14.177015    5304 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-6d8bf9c9ff-r77kz_0f933e27-7304-44d5-acc5-8089829b1b90/manager/0.log"
Nov 02 16:15:15.451501 ip-10-0-135-138 kubenswrapper[5304]: I1102 16:15:15.451461    5304 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-7lk7m_3fc11777-186c-445a-8b71-2aefdf642e75/manager/0.log"
Nov 02 16:15:16.243968 ip-10-0-135-138 kubenswrapper[5304]: I1102 16:15:16.243932    5304 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-5484769d88-cbstn_cb1a05be-2f3b-4f5a-a1b7-69d822c1d658/maas-api/0.log"
Nov 02 16:15:18.609408 ip-10-0-135-138 kubenswrapper[5304]: I1102 16:15:18.609367    5304 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-84d5ffc6f9-rvpgh_2dd4123d-04a7-4eee-b14e-8c79b45a310d/authorino/0.log"
Nov 02 16:15:19.239917 ip-10-0-135-138 kubenswrapper[5304]: I1102 16:15:19.239886    5304 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-n9gf2_6f7db1ab-d7ad-4aa8-b04e-147b63a46ad6/kuadrant-console-plugin/0.log"
Nov 02 16:15:20.592894 ip-10-0-135-138 kubenswrapper[5304]: I1102 16:15:20.592860    5304 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-zkvr7_f967e0cd-2c0f-42ad-9163-2786089f3ca9/discovery/0.log"
Nov 02 16:15:20.818593 ip-10-0-135-138 kubenswrapper[5304]: I1102 16:15:20.818552    5304 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-77f469495f-6cphj_a427d6de-97f8-4bbd-99d8-6327efcf1794/kube-auth-proxy/0.log"
Nov 02 16:15:21.218224 ip-10-0-135-138 kubenswrapper[5304]: I1102 16:15:21.218194    5304 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-fcb75c45f-2lc6p_e24e2d68-b448-4c4f-99c6-4effdd9992e4/istio-proxy/0.log"
Nov 02 16:15:21.838172 ip-10-0-135-138 kubenswrapper[5304]: I1102 16:15:21.838142    5304 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-fcb75c45f-j6lh6_9a2f560c-e476-4c3a-b96d-1d3e7b313360/istio-proxy/0.log"
Nov 02 16:15:22.836541 ip-10-0-135-138 kubenswrapper[5304]: I1102 16:15:22.836489    5304 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-fcb75c45f-srlvv_98cb3e76-0b44-497d-aaf7-4ee08e76d1b1/istio-proxy/0.log"
Nov 02 16:15:23.612254 ip-10-0-135-138 kubenswrapper[5304]: I1102 16:15:23.612220    5304 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-wxjl7_2f905a4e-b982-4d66-a7bc-171cda1c78dd/payload-processing/0.log"
Nov 02 16:15:24.374016 ip-10-0-135-138 kubenswrapper[5304]: I1102 16:15:24.373983    5304 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-cbd4f5586-7t7f8_b76c8a20-92cc-41ac-b1be-2a779ec1a5ed/main/0.log"
Nov 02 16:15:24.579392 ip-10-0-135-138 kubenswrapper[5304]: I1102 16:15:24.579363    5304 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-7b998b79d9-rmdsn_8cca3fa3-d2b1-4abb-bbb5-a38dfaa3c6b3/main/0.log"
Nov 02 16:15:24.798099 ip-10-0-135-138 kubenswrapper[5304]: E1102 16:15:24.798009    5304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k6fjk" containerName="extractor"
Nov 02 16:15:24.953390 ip-10-0-135-138 kubenswrapper[5304]: I1102 16:15:24.953359    5304 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-69b588897d-56lvt_569cd1bd-5016-4c33-98d7-d9b76c93de43/main/0.log"
Nov 02 16:15:25.361992 ip-10-0-135-138 kubenswrapper[5304]: I1102 16:15:25.361954    5304 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-t8nst_6eaab677-88a0-49d8-9c89-5deec4c81a4d/main/0.log"
Nov 02 16:15:25.546489 ip-10-0-135-138 kubenswrapper[5304]: I1102 16:15:25.546453    5304 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-dzs48_ed68913c-1ebd-4bb6-b3d0-58a9a19e2192/main/0.log"
Nov 02 16:15:25.743691 ip-10-0-135-138 kubenswrapper[5304]: I1102 16:15:25.743607    5304 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-nkbrk_1f68d837-fff2-418b-84d7-c2d7eb21a57e/main/0.log"
Nov 02 16:15:54.797985 ip-10-0-135-138 kubenswrapper[5304]: E1102 16:15:54.797946    5304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k6fjk" containerName="extractor"
Nov 02 16:16:11.363264 ip-10-0-135-138 kubenswrapper[5304]: I1102 16:16:11.363229    5304 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-wxjl7_2f905a4e-b982-4d66-a7bc-171cda1c78dd/payload-processing/0.log"
Nov 02 16:16:11.557866 ip-10-0-135-138 kubenswrapper[5304]: I1102 16:16:11.557831    5304 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-wxjl7_2f905a4e-b982-4d66-a7bc-171cda1c78dd/payload-processing/0.log"
Nov 02 16:16:24.798251 ip-10-0-135-138 kubenswrapper[5304]: E1102 16:16:24.798159    5304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k6fjk" containerName="extractor"
Nov 02 16:16:54.798655 ip-10-0-135-138 kubenswrapper[5304]: E1102 16:16:54.798618    5304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k6fjk" containerName="extractor"
Nov 02 16:17:24.798347 ip-10-0-135-138 kubenswrapper[5304]: E1102 16:17:24.798300    5304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k6fjk" containerName="extractor"
Nov 02 16:17:26.510041 ip-10-0-135-138 kubenswrapper[5304]: I1102 16:17:26.510009    5304 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-wxjl7_2f905a4e-b982-4d66-a7bc-171cda1c78dd/payload-processing/0.log"
Nov 02 16:17:26.708771 ip-10-0-135-138 kubenswrapper[5304]: I1102 16:17:26.708737    5304 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-wxjl7_2f905a4e-b982-4d66-a7bc-171cda1c78dd/payload-processing/0.log"
Nov 02 16:17:31.677239 ip-10-0-135-138 kubenswrapper[5304]: I1102 16:17:31.677206    5304 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-vl6zb_332f4a2d-a241-4c71-8ebb-a99618c0d3dc/global-pull-secret-syncer/0.log"
Nov 02 16:17:32.359455 ip-10-0-135-138 kubenswrapper[5304]: I1102 16:17:32.359419    5304 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-d5t6w_68b781b2-be96-4c5c-8daa-2238fd0a9623/konnectivity-agent/0.log"
Nov 02 16:17:32.815078 ip-10-0-135-138 kubenswrapper[5304]: I1102 16:17:32.815000    5304 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-135-138.ec2.internal_9a8741d4cd8a99bb77c79c43a5144068/haproxy/0.log"
Nov 02 16:17:49.747040 ip-10-0-135-138 kubenswrapper[5304]: I1102 16:17:49.746995    5304 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-xl725/perf-node-gather-daemonset-dpbff"]
Nov 02 16:17:49.747674 ip-10-0-135-138 kubenswrapper[5304]: I1102 16:17:49.747454    5304 cpu_manager.go:414] "containerMap: removing container" podUID="ff2f1cbd-70c5-48d0-b133-0e4ea14e8dec" containerName="maas-api"
Nov 02 16:17:49.747674 ip-10-0-135-138 kubenswrapper[5304]: I1102 16:17:49.747468    5304 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff2f1cbd-70c5-48d0-b133-0e4ea14e8dec" containerName="maas-api"
Nov 02 16:17:49.747674 ip-10-0-135-138 kubenswrapper[5304]: I1102 16:17:49.747549    5304 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff2f1cbd-70c5-48d0-b133-0e4ea14e8dec" containerName="maas-api"
Nov 02 16:17:49.750482 ip-10-0-135-138 kubenswrapper[5304]: I1102 16:17:49.750465    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-xl725/perf-node-gather-daemonset-dpbff"
Nov 02 16:17:49.768632 ip-10-0-135-138 kubenswrapper[5304]: I1102 16:17:49.768597    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-xl725/perf-node-gather-daemonset-dpbff"]
Nov 02 16:17:49.829667 ip-10-0-135-138 kubenswrapper[5304]: I1102 16:17:49.829629    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bfdaf1e1-1216-4876-9939-7721a2d91640-sys\") pod \"perf-node-gather-daemonset-dpbff\" (UID: \"bfdaf1e1-1216-4876-9939-7721a2d91640\") " pod="openshift-must-gather-xl725/perf-node-gather-daemonset-dpbff"
Nov 02 16:17:49.829858 ip-10-0-135-138 kubenswrapper[5304]: I1102 16:17:49.829685    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pw9zd\" (UniqueName: \"kubernetes.io/projected/bfdaf1e1-1216-4876-9939-7721a2d91640-kube-api-access-pw9zd\") pod \"perf-node-gather-daemonset-dpbff\" (UID: \"bfdaf1e1-1216-4876-9939-7721a2d91640\") " pod="openshift-must-gather-xl725/perf-node-gather-daemonset-dpbff"
Nov 02 16:17:49.829858 ip-10-0-135-138 kubenswrapper[5304]: I1102 16:17:49.829767    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/bfdaf1e1-1216-4876-9939-7721a2d91640-lib-modules\") pod \"perf-node-gather-daemonset-dpbff\" (UID: \"bfdaf1e1-1216-4876-9939-7721a2d91640\") " pod="openshift-must-gather-xl725/perf-node-gather-daemonset-dpbff"
Nov 02 16:17:49.829942 ip-10-0-135-138 kubenswrapper[5304]: I1102 16:17:49.829851    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/bfdaf1e1-1216-4876-9939-7721a2d91640-proc\") pod \"perf-node-gather-daemonset-dpbff\" (UID: \"bfdaf1e1-1216-4876-9939-7721a2d91640\") " pod="openshift-must-gather-xl725/perf-node-gather-daemonset-dpbff"
Nov 02 16:17:49.829942 ip-10-0-135-138 kubenswrapper[5304]: I1102 16:17:49.829924    5304 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/bfdaf1e1-1216-4876-9939-7721a2d91640-podres\") pod \"perf-node-gather-daemonset-dpbff\" (UID: \"bfdaf1e1-1216-4876-9939-7721a2d91640\") " pod="openshift-must-gather-xl725/perf-node-gather-daemonset-dpbff"
Nov 02 16:17:49.859713 ip-10-0-135-138 kubenswrapper[5304]: I1102 16:17:49.859681    5304 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-xl725\"/\"default-dockercfg-vkz4g\""
Nov 02 16:17:49.860214 ip-10-0-135-138 kubenswrapper[5304]: I1102 16:17:49.860201    5304 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-xl725\"/\"kube-root-ca.crt\""
Nov 02 16:17:49.860326 ip-10-0-135-138 kubenswrapper[5304]: I1102 16:17:49.860308    5304 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-xl725\"/\"openshift-service-ca.crt\""
Nov 02 16:17:49.930915 ip-10-0-135-138 kubenswrapper[5304]: I1102 16:17:49.930874    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/bfdaf1e1-1216-4876-9939-7721a2d91640-podres\") pod \"perf-node-gather-daemonset-dpbff\" (UID: \"bfdaf1e1-1216-4876-9939-7721a2d91640\") " pod="openshift-must-gather-xl725/perf-node-gather-daemonset-dpbff"
Nov 02 16:17:49.931101 ip-10-0-135-138 kubenswrapper[5304]: I1102 16:17:49.930933    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bfdaf1e1-1216-4876-9939-7721a2d91640-sys\") pod \"perf-node-gather-daemonset-dpbff\" (UID: \"bfdaf1e1-1216-4876-9939-7721a2d91640\") " pod="openshift-must-gather-xl725/perf-node-gather-daemonset-dpbff"
Nov 02 16:17:49.931101 ip-10-0-135-138 kubenswrapper[5304]: I1102 16:17:49.930978    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pw9zd\" (UniqueName: \"kubernetes.io/projected/bfdaf1e1-1216-4876-9939-7721a2d91640-kube-api-access-pw9zd\") pod \"perf-node-gather-daemonset-dpbff\" (UID: \"bfdaf1e1-1216-4876-9939-7721a2d91640\") " pod="openshift-must-gather-xl725/perf-node-gather-daemonset-dpbff"
Nov 02 16:17:49.931101 ip-10-0-135-138 kubenswrapper[5304]: I1102 16:17:49.931007    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/bfdaf1e1-1216-4876-9939-7721a2d91640-lib-modules\") pod \"perf-node-gather-daemonset-dpbff\" (UID: \"bfdaf1e1-1216-4876-9939-7721a2d91640\") " pod="openshift-must-gather-xl725/perf-node-gather-daemonset-dpbff"
Nov 02 16:17:49.931101 ip-10-0-135-138 kubenswrapper[5304]: I1102 16:17:49.931027    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/bfdaf1e1-1216-4876-9939-7721a2d91640-podres\") pod \"perf-node-gather-daemonset-dpbff\" (UID: \"bfdaf1e1-1216-4876-9939-7721a2d91640\") " pod="openshift-must-gather-xl725/perf-node-gather-daemonset-dpbff"
Nov 02 16:17:49.931101 ip-10-0-135-138 kubenswrapper[5304]: I1102 16:17:49.931078    5304 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/bfdaf1e1-1216-4876-9939-7721a2d91640-proc\") pod \"perf-node-gather-daemonset-dpbff\" (UID: \"bfdaf1e1-1216-4876-9939-7721a2d91640\") " pod="openshift-must-gather-xl725/perf-node-gather-daemonset-dpbff"
Nov 02 16:17:49.931349 ip-10-0-135-138 kubenswrapper[5304]: I1102 16:17:49.931110    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/bfdaf1e1-1216-4876-9939-7721a2d91640-lib-modules\") pod \"perf-node-gather-daemonset-dpbff\" (UID: \"bfdaf1e1-1216-4876-9939-7721a2d91640\") " pod="openshift-must-gather-xl725/perf-node-gather-daemonset-dpbff"
Nov 02 16:17:49.931349 ip-10-0-135-138 kubenswrapper[5304]: I1102 16:17:49.931121    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/bfdaf1e1-1216-4876-9939-7721a2d91640-proc\") pod \"perf-node-gather-daemonset-dpbff\" (UID: \"bfdaf1e1-1216-4876-9939-7721a2d91640\") " pod="openshift-must-gather-xl725/perf-node-gather-daemonset-dpbff"
Nov 02 16:17:49.931349 ip-10-0-135-138 kubenswrapper[5304]: I1102 16:17:49.931075    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bfdaf1e1-1216-4876-9939-7721a2d91640-sys\") pod \"perf-node-gather-daemonset-dpbff\" (UID: \"bfdaf1e1-1216-4876-9939-7721a2d91640\") " pod="openshift-must-gather-xl725/perf-node-gather-daemonset-dpbff"
Nov 02 16:17:49.954692 ip-10-0-135-138 kubenswrapper[5304]: I1102 16:17:49.954666    5304 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pw9zd\" (UniqueName: \"kubernetes.io/projected/bfdaf1e1-1216-4876-9939-7721a2d91640-kube-api-access-pw9zd\") pod \"perf-node-gather-daemonset-dpbff\" (UID: \"bfdaf1e1-1216-4876-9939-7721a2d91640\") " pod="openshift-must-gather-xl725/perf-node-gather-daemonset-dpbff"
Nov 02 16:17:50.063364 ip-10-0-135-138 kubenswrapper[5304]: I1102 16:17:50.063279    5304 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-xl725/perf-node-gather-daemonset-dpbff"
Nov 02 16:17:50.364793 ip-10-0-135-138 kubenswrapper[5304]: I1102 16:17:50.364759    5304 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-xl725/perf-node-gather-daemonset-dpbff"]
Nov 02 16:17:50.369303 ip-10-0-135-138 kubenswrapper[5304]: W1102 16:17:50.369274    5304 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podbfdaf1e1_1216_4876_9939_7721a2d91640.slice/crio-d6e71a3649d1165814111c01c876052c96bf5dee0b5cbae4163ab101069fbabb WatchSource:0}: Error finding container d6e71a3649d1165814111c01c876052c96bf5dee0b5cbae4163ab101069fbabb: Status 404 returned error can't find the container with id d6e71a3649d1165814111c01c876052c96bf5dee0b5cbae4163ab101069fbabb
Nov 02 16:17:50.444447 ip-10-0-135-138 kubenswrapper[5304]: I1102 16:17:50.444417    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-xl725/perf-node-gather-daemonset-dpbff" event={"ID":"bfdaf1e1-1216-4876-9939-7721a2d91640","Type":"ContainerStarted","Data":"d6e71a3649d1165814111c01c876052c96bf5dee0b5cbae4163ab101069fbabb"}
Nov 02 16:17:51.450607 ip-10-0-135-138 kubenswrapper[5304]: I1102 16:17:51.450520    5304 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-xl725/perf-node-gather-daemonset-dpbff" event={"ID":"bfdaf1e1-1216-4876-9939-7721a2d91640","Type":"ContainerStarted","Data":"26a2a5fb661574409eb8c3370c596b5b63d28ea3a5e6de5c509562a9283a1e65"}
Nov 02 16:17:51.450607 ip-10-0-135-138 kubenswrapper[5304]: I1102 16:17:51.450586    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-xl725/perf-node-gather-daemonset-dpbff"
Nov 02 16:17:51.497428 ip-10-0-135-138 kubenswrapper[5304]: I1102 16:17:51.497381    5304 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-xl725/perf-node-gather-daemonset-dpbff" podStartSLOduration=2.497366666 podStartE2EDuration="2.497366666s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 16:17:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 16:17:51.497070577 +0000 UTC m=+4871.490894986" watchObservedRunningTime="2026-11-02 16:17:51.497366666 +0000 UTC m=+4871.491191090"
Nov 02 16:17:54.800427 ip-10-0-135-138 kubenswrapper[5304]: E1102 16:17:54.800378    5304 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k6fjk" containerName="extractor"
Nov 02 16:17:57.464629 ip-10-0-135-138 kubenswrapper[5304]: I1102 16:17:57.464601    5304 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-xl725/perf-node-gather-daemonset-dpbff"
Nov 02 16:18:16.504094 ip-10-0-135-138 kubenswrapper[5304]: I1102 16:18:16.504058    5304 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-zkvr7_f967e0cd-2c0f-42ad-9163-2786089f3ca9/discovery/0.log"
Nov 02 16:18:16.537901 ip-10-0-135-138 kubenswrapper[5304]: I1102 16:18:16.537866    5304 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-77f469495f-6cphj_a427d6de-97f8-4bbd-99d8-6327efcf1794/kube-auth-proxy/0.log"
Nov 02 16:18:16.667416 ip-10-0-135-138 kubenswrapper[5304]: I1102 16:18:16.667387    5304 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-fcb75c45f-2lc6p_e24e2d68-b448-4c4f-99c6-4effdd9992e4/istio-proxy/0.log"
Nov 02 16:18:16.857538 ip-10-0-135-138 kubenswrapper[5304]: I1102 16:18:16.857504    5304 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-fcb75c45f-j6lh6_9a2f560c-e476-4c3a-b96d-1d3e7b313360/istio-proxy/0.log"
Nov 02 16:18:17.180857 ip-10-0-135-138 kubenswrapper[5304]: I1102 16:18:17.180772    5304 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-fcb75c45f-srlvv_98cb3e76-0b44-497d-aaf7-4ee08e76d1b1/istio-proxy/0.log"
Nov 02 16:18:17.435312 ip-10-0-135-138 kubenswrapper[5304]: I1102 16:18:17.435219    5304 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-wxjl7_2f905a4e-b982-4d66-a7bc-171cda1c78dd/payload-processing/0.log"